com.northconcepts.datapipeline.parquet

Class ParquetDataWriter

java.lang.Object

com.northconcepts.datapipeline.core.DataObject

com.northconcepts.datapipeline.core.Endpoint

com.northconcepts.datapipeline.core.DataEndpoint

com.northconcepts.datapipeline.core.DataWriter

com.northconcepts.datapipeline.internal.lang.IntegrationWriter

com.northconcepts.datapipeline.parquet.ParquetDataWriter

public class ParquetDataWriter
extends IntegrationWriter

Writes records to Apache Parquet columnar files. See Apache Parquet columnar storage.

Nested Class Summary

Nested classes/interfaces inherited from class com.northconcepts.datapipeline.core.DataEndpoint

DataEndpoint.State

Field Summary

Fields inherited from class com.northconcepts.datapipeline.core.DataEndpoint

lastRecord, PRODUCT, PRODUCT_VERSION, VENDOR, XML_INPUT_FACTORY_KEY

Fields inherited from class com.northconcepts.datapipeline.core.Endpoint

BUFFER_SIZE, captureElapsedTime, DEFAULT_READ_BUFFER_SIZE

Fields inherited from class com.northconcepts.datapipeline.core.DataObject

id, log, name, TIMESTAMP_FORMAT

Constructor Summary

Constructors

Constructor and Description
ParquetDataWriter(File file) Write parquet data to a file.
ParquetDataWriter(OutputFile outputFile) Write parquet data to an OutputFile.

Method Summary

Modifier and Type	Method and Description
DataException	addExceptionProperties(DataException exception) Adds this endpoint's current state to a DataException.
void	close() Indicates that this endpoint has finished reading or writing.
CompressionCodecName	getCompressionCodecName() Indicates the compression used for writing (default UNCOMPRESSED).
Configuration	getConfiguration() Returns the Parquet configuration parameters.
int	getDefaultBigDecimalScale() Returns the default scale used when writing BigDecimal values (default 5).
RoundingMode	getRoundingMode() Indicates the rounding algorithm used for all BigDecimal values (default is RoundingMode.HALF_UP).
MessageType	getSchema() Returns the schema used to write the file.
boolean	isDefaulAdjustToUTC() Deprecated. see isDefaultAdjustedToUTC()
boolean	isDefaultAdjustedToUTC() Indicates if all datetime fields should be marked as AdjustedToUTC.
void	open() Makes this endpoint ready for reading or writing.
ParquetDataWriter	setCompressionCodecName(CompressionCodecName compressionCodecName) Indicates the compression used for writing (default UNCOMPRESSED).
ParquetDataWriter	setConfiguration(Configuration configuration) Sets the Parquet configuration parameters.
ParquetDataWriter	setDefaultAdjustedToUTC(boolean defaultAdjustedToUTC) Indicates if all datetime fields should be marked as AdjustedToUTC.
ParquetDataWriter	setDefaultAdjustToUTC(boolean defaultAdjustToUTC) Deprecated. see setDefaultAdjustedToUTC(boolean)
ParquetDataWriter	setDefaultBigDecimalScale(int defaultBigDecimalScale) Sets the default scale used when writing BigDecimal values (default 5).
ParquetDataWriter	setRoundingMode(RoundingMode roundingMode) Indicates the rounding algorithm used for all BigDecimal values (default is RoundingMode.HALF_UP).
ParquetDataWriter	setSchema(MessageType schema) Sets the schema used to write the file.
protected void	writeImpl(Record record) Overridden by subclasses to write the specified record to this DataWriter.

Methods inherited from class com.northconcepts.datapipeline.core.DataWriter

available, getNestedEndpoint, getNestedWriter, getRootEndpoint, getRootWriter, write

Methods inherited from class com.northconcepts.datapipeline.core.DataEndpoint

decrementRecordCount, enableJmx, getLastRecord, getRecordCount, getRecordCountAsBigInteger, getRecordCountAsString, incrementRecordCount, isRecordCountBigInteger, resetRecordCount, toString

Methods inherited from class com.northconcepts.datapipeline.core.Endpoint

addElapsedtime, assertClosed, assertNotOpened, assertOpened, finalize, getClosedOn, getDescription, getElapsedTime, getElapsedTimeAsString, getOpenedOn, getOpenElapsedTime, getOpenElapsedTimeAsString, getSelfTime, getSelfTimeAsString, getState, isCaptureElapsedTime, isClosed, isOpen, setCaptureElapsedTime, setDescription

Methods inherited from class com.northconcepts.datapipeline.core.DataObject

exception, exception, exception, getId, getName, resetID

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

ParquetDataWriter

public ParquetDataWriter(File file)

Write parquet data to a file.

ParquetDataWriter

public ParquetDataWriter(OutputFile outputFile)

Write parquet data to an OutputFile.

Parameters:

outputFile - - OutputFile with FileSystem.

Method Detail

open

public void open()
          throws DataException

Description copied from class: DataEndpoint

Makes this endpoint ready for reading or writing.

Overrides:

open in class IntegrationWriter

Throws:

DataException

writeImpl

protected void writeImpl(Record record)
                  throws Throwable

Description copied from class: DataWriter

Overridden by subclasses to write the specified record to this DataWriter.

Specified by:

writeImpl in class DataWriter

Throws:

Throwable

close

public void close()
           throws DataException

Description copied from class: DataEndpoint

Indicates that this endpoint has finished reading or writing.

Overrides:

close in class DataEndpoint

Throws:

DataException

getSchema

public MessageType getSchema()

Returns the schema used to write the file.

setSchema

public ParquetDataWriter setSchema(MessageType schema)

Sets the schema used to write the file.

getDefaultBigDecimalScale

public int getDefaultBigDecimalScale()

Returns the default scale used when writing BigDecimal values (default 5).

setDefaultBigDecimalScale

public ParquetDataWriter setDefaultBigDecimalScale(int defaultBigDecimalScale)

Sets the default scale used when writing BigDecimal values (default 5).

addExceptionProperties

public DataException addExceptionProperties(DataException exception)

Description copied from class: Endpoint

Adds this endpoint's current state to a DataException. Since this method is called whenever an exception is thrown, subclasses should override it to add their specific information.

Overrides:

addExceptionProperties in class DataWriter

getCompressionCodecName

public CompressionCodecName getCompressionCodecName()

Indicates the compression used for writing (default UNCOMPRESSED).

setCompressionCodecName

public ParquetDataWriter setCompressionCodecName(CompressionCodecName compressionCodecName)

Indicates the compression used for writing (default UNCOMPRESSED).

isDefaulAdjustToUTC

@Deprecated
public boolean isDefaulAdjustToUTC()

Deprecated. see isDefaultAdjustedToUTC()

setDefaultAdjustToUTC

@Deprecated
public ParquetDataWriter setDefaultAdjustToUTC(boolean defaultAdjustToUTC)

Deprecated. see setDefaultAdjustedToUTC(boolean)

isDefaultAdjustedToUTC

public boolean isDefaultAdjustedToUTC()

Indicates if all datetime fields should be marked as AdjustedToUTC.

setDefaultAdjustedToUTC

public ParquetDataWriter setDefaultAdjustedToUTC(boolean defaultAdjustedToUTC)

Indicates if all datetime fields should be marked as AdjustedToUTC.

getRoundingMode

public RoundingMode getRoundingMode()

Indicates the rounding algorithm used for all BigDecimal values (default is RoundingMode.HALF_UP).

setRoundingMode

public ParquetDataWriter setRoundingMode(RoundingMode roundingMode)

Indicates the rounding algorithm used for all BigDecimal values (default is RoundingMode.HALF_UP).

getConfiguration

public Configuration getConfiguration()

Returns the Parquet configuration parameters.

setConfiguration

public ParquetDataWriter setConfiguration(Configuration configuration)

Sets the Parquet configuration parameters.