com.northconcepts.datapipeline.parquet.output

Class ParquetPipelineOutput

java.lang.Object

com.northconcepts.datapipeline.foundations.core.Bean

com.northconcepts.datapipeline.foundations.core.FoundationObject

com.northconcepts.datapipeline.foundations.pipeline.PipelineObject

com.northconcepts.datapipeline.foundations.pipeline.PipelineOutput

com.northconcepts.datapipeline.foundations.pipeline.output.FileSinkPipelineOutput

com.northconcepts.datapipeline.parquet.output.ParquetPipelineOutput

All Implemented Interfaces:

DataExceptionContributor, JsonSerializable, RecordSerializable, XmlSerializable, DataWriterFactory, JavaCodeGenerator, Serializable

public class ParquetPipelineOutput
extends FileSinkPipelineOutput

See Also:

Serialized Form

Field Summary

Fields inherited from class com.northconcepts.datapipeline.foundations.core.FoundationObject

internalId, internalName, log, TIMESTAMP_FORMAT

Fields inherited from interface com.northconcepts.datapipeline.core.RecordSerializable

SERIALIZED_CLASS_NAME, TYPE

Fields inherited from interface com.northconcepts.datapipeline.core.XmlSerializable

XML_SERIALIZED_CLASS_NAME

Constructor Summary

Constructors

Constructor and Description
ParquetPipelineOutput()

Method Summary

Modifier and Type	Method and Description
DataWriter	createDataWriter()
ParquetPipelineOutput	fromRecord(Record source) Loads this instance's state from a record and returns this (for fluid API call chaining).
ParquetPipelineOutput	fromXmlElement(Element pipelineOutputElement)
void	generateJavaCode(JavaCodeBuilder code)
String	getCacheFolder()
CompressionCodecName	getCompressionCodecName() Indicates the compression used for writing (default UNCOMPRESSED).
Map<String,String>	getConfiguration() Returns the Parquet configuration parameters.
int	getDefaultBigDecimalScale() Returns the default scale used when writing BigDecimal values (default 5).
int	getDefaultBigNumberPrecision() Returns the default precision used when writing BigDecimal & BigInteger values (default 25).
FileSink	getFileSink()
String	getName()
int	getRecordsPerCacheFile()
RoundingMode	getRoundingMode() Indicates the rounding algorithm used for all BigDecimal values (default is RoundingMode.HALF_UP).
MessageType	getSchema() Returns the schema used to write the file.
boolean	isDefaultAdjustedToUTC() Indicates if all datetime fields should be marked as AdjustedToUTC.
ParquetPipelineOutput	setCacheFolder(String cacheFolder)
ParquetPipelineOutput	setCompressionCodecName(CompressionCodecName compressionCodecName) Indicates the compression used for writing (default UNCOMPRESSED).
ParquetPipelineOutput	setConfiguration(Map<String,String> configuration) Sets the Parquet configuration parameters.
ParquetPipelineOutput	setConfiguration(String key, String value)
ParquetPipelineOutput	setDefaultAdjustedToUTC(boolean defaultAdjustedToUTC) Indicates if all datetime fields should be marked as AdjustedToUTC.
ParquetPipelineOutput	setDefaultBigDecimalScale(int defaultBigDecimalScale) Sets the default scale used when writing BigDecimal values (default 5).
ParquetPipelineOutput	setDefaultBigNumberPrecision(int defaultBigNumberPrecision) Sets the default precision used when writing BigDecimal & BigInteger values (default 25).
ParquetPipelineOutput	setFileSink(FileSink fileSink)
ParquetPipelineOutput	setMaxRecordsAnalyzed(Long maxRecordsAnalyzed) Indicates how many records should be analyzed and cached to generate the Parquet schema if no schema was explicitly set on this writer (default is 1000).
ParquetPipelineOutput	setRecordsPerCacheFile(int recordsPerCacheFile)
ParquetPipelineOutput	setRoundingMode(RoundingMode roundingMode) Indicates the rounding algorithm used for all BigDecimal values (default is RoundingMode.HALF_UP).
ParquetPipelineOutput	setSchema(MessageType schema) Sets the schema used to write the file.
Record	toRecord()
Element	toXmlElement(Document document)

Methods inherited from class com.northconcepts.datapipeline.foundations.pipeline.output.FileSinkPipelineOutput

generateSaveCode

Methods inherited from class com.northconcepts.datapipeline.foundations.pipeline.PipelineOutput

getNestedPipelineOutput, getRootPipelineOutput

Methods inherited from class com.northconcepts.datapipeline.foundations.core.FoundationObject

addExceptionProperties, assertValid, assertValid, clone, exception, exception, exception, getInternalId, getInternalName, resetInternalId

Methods inherited from class com.northconcepts.datapipeline.foundations.core.Bean

toString

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface com.northconcepts.datapipeline.core.RecordSerializable

fromArrayValue, fromArrayValue, fromArrayValue, fromJson, fromJson, newInstanceFromRecord, newInstanceFromRecord, toArrayValue, toJson, toJson, toJson, toRecord

Methods inherited from interface com.northconcepts.datapipeline.core.JsonSerializable

newInstanceFromJson

Methods inherited from interface com.northconcepts.datapipeline.core.XmlSerializable

forEachGrandChildElement, fromXml, fromXml, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getChildElement, getChildElements, newInstanceFromXml, newInstanceFromXml, setAttribute, toXml, toXml, toXml, toXml, toXml, writeXml, writeXml

Methods inherited from interface com.northconcepts.datapipeline.core.DataExceptionContributor

addExceptionProperties, contributeExceptionProperties, contributeExceptionProperties

Constructor Detail

ParquetPipelineOutput

public ParquetPipelineOutput()

Method Detail

getName

public String getName()

Overrides:

getName in class FileSinkPipelineOutput

createDataWriter

public DataWriter createDataWriter()

getFileSink

public FileSink getFileSink()

Specified by:

getFileSink in class FileSinkPipelineOutput

setFileSink

public ParquetPipelineOutput setFileSink(FileSink fileSink)

Specified by:

setFileSink in class FileSinkPipelineOutput

getCompressionCodecName

public CompressionCodecName getCompressionCodecName()

Indicates the compression used for writing (default UNCOMPRESSED).

getConfiguration

public Map<String,String> getConfiguration()

Returns the Parquet configuration parameters.

setConfiguration

public ParquetPipelineOutput setConfiguration(Map<String,String> configuration)

Sets the Parquet configuration parameters.

setConfiguration

public ParquetPipelineOutput setConfiguration(String key,
                                              String value)

setCompressionCodecName

public ParquetPipelineOutput setCompressionCodecName(CompressionCodecName compressionCodecName)

Indicates the compression used for writing (default UNCOMPRESSED).

isDefaultAdjustedToUTC

public boolean isDefaultAdjustedToUTC()

Indicates if all datetime fields should be marked as AdjustedToUTC.

setDefaultAdjustedToUTC

public ParquetPipelineOutput setDefaultAdjustedToUTC(boolean defaultAdjustedToUTC)

Indicates if all datetime fields should be marked as AdjustedToUTC.

getRoundingMode

public RoundingMode getRoundingMode()

Indicates the rounding algorithm used for all BigDecimal values (default is RoundingMode.HALF_UP).

setRoundingMode

public ParquetPipelineOutput setRoundingMode(RoundingMode roundingMode)

Indicates the rounding algorithm used for all BigDecimal values (default is RoundingMode.HALF_UP).

getSchema

public MessageType getSchema()

Returns the schema used to write the file.

setSchema

public ParquetPipelineOutput setSchema(MessageType schema)

Sets the schema used to write the file.

getDefaultBigDecimalScale

public int getDefaultBigDecimalScale()

Returns the default scale used when writing BigDecimal values (default 5).

setDefaultBigDecimalScale

public ParquetPipelineOutput setDefaultBigDecimalScale(int defaultBigDecimalScale)

Sets the default scale used when writing BigDecimal values (default 5).

getDefaultBigNumberPrecision

public int getDefaultBigNumberPrecision()

Returns the default precision used when writing BigDecimal & BigInteger values (default 25).

setDefaultBigNumberPrecision

public ParquetPipelineOutput setDefaultBigNumberPrecision(int defaultBigNumberPrecision)

Sets the default precision used when writing BigDecimal & BigInteger values (default 25).

setMaxRecordsAnalyzed

public ParquetPipelineOutput setMaxRecordsAnalyzed(Long maxRecordsAnalyzed)

Indicates how many records should be analyzed and cached to generate the Parquet schema if no schema was explicitly set on this writer (default is 1000). This value will not be used if a schema was set on this writer.

Passing in null will cause all records to be read and cached to determine the schema.

The value will be set to 1 if a value less than 1 is passed in.

Note: Using null or a high record count can significantly slow down processing and cause an OutOfMemoryError.

getCacheFolder

public String getCacheFolder()

setCacheFolder

public ParquetPipelineOutput setCacheFolder(String cacheFolder)

getRecordsPerCacheFile

public int getRecordsPerCacheFile()

setRecordsPerCacheFile

public ParquetPipelineOutput setRecordsPerCacheFile(int recordsPerCacheFile)

generateJavaCode

public void generateJavaCode(JavaCodeBuilder code)

toRecord

public Record toRecord()

Specified by:

toRecord in interface RecordSerializable

Overrides:

toRecord in class Bean

fromRecord

public ParquetPipelineOutput fromRecord(Record source)

Description copied from interface: RecordSerializable

Loads this instance's state from a record and returns this (for fluid API call chaining). For fluid API call chaining, the overridden method should change the declared return type to its class.

Specified by:

fromRecord in interface RecordSerializable

Overrides:

fromRecord in class PipelineOutput

Returns:

this instance.

toXmlElement

public Element toXmlElement(Document document)

Specified by:

toXmlElement in interface XmlSerializable

Overrides:

toXmlElement in class PipelineOutput

fromXmlElement

public ParquetPipelineOutput fromXmlElement(Element pipelineOutputElement)