com.northconcepts.datapipeline.parquet

Class ParquetDataReader

java.lang.Object

com.northconcepts.datapipeline.core.DataObject

com.northconcepts.datapipeline.core.Endpoint

com.northconcepts.datapipeline.core.DataEndpoint

com.northconcepts.datapipeline.core.DataReader

com.northconcepts.datapipeline.internal.lang.IntegrationReader

com.northconcepts.datapipeline.parquet.ParquetDataReader

public class ParquetDataReader
extends IntegrationReader

Read records from Apache Parquet columnar files. See Apache Parquet columnar storage.

Nested Class Summary

Nested classes/interfaces inherited from class com.northconcepts.datapipeline.core.DataEndpoint

DataEndpoint.State

Field Summary

Fields inherited from class com.northconcepts.datapipeline.core.DataReader

fieldLineage, recordLineage

Fields inherited from class com.northconcepts.datapipeline.core.DataEndpoint

lastRecord, PRODUCT, PRODUCT_VERSION, VENDOR, XML_INPUT_FACTORY_KEY

Fields inherited from class com.northconcepts.datapipeline.core.Endpoint

BUFFER_SIZE, captureElapsedTime, DEFAULT_READ_BUFFER_SIZE

Fields inherited from class com.northconcepts.datapipeline.core.DataObject

id, log, name, TIMESTAMP_FORMAT

Constructor Summary

Constructors

Constructor and Description
ParquetDataReader(InputFile inputFile) Reads parquet data from an InputFile.

Method Summary

Modifier and Type	Method and Description
DataException	addExceptionProperties(DataException exception) Adds this endpoint's current state to a DataException.
protected Record	addLineage(Record record)
void	close() Indicates that this endpoint has finished reading or writing.
Configuration	getConfiguration() Returns the Parquet configuration parameters.
MetadataFilter	getFilter() Returns the filter settings.
MessageType	getModifiedSchema() Indicates the modified schema used to read the file.
MessageType	getSchema() Indicates the schema used to read the file.
protected Date	int96ToTimestamp(byte[] int96Bytes)
boolean	isDebug() Indicates if debugging is enabled to print log statements.
boolean	isLineageSupported()
boolean	isMakeOptionalFieldsRequired() Used whether to check for optional columns that can be made required using ColumnChunkMetaData#getStatistics() during open().
boolean	isMakeRequiredFieldsOptional() When true, check for required columns that can be made optional using ColumnChunkMetaData#getStatistics().
boolean	isRemoveFieldsWithoutColumnMetadata() When true, remove fields in the schema does not have a corresponding ColumnChunkMetaData in the file.
boolean	isRemoveFieldsWithoutValues() When true, remove fields in the schema whose value count is <= 0 See getFieldsWithoutColumnMetadata() for details.
void	open() Makes this endpoint ready for reading or writing.
protected void	readGroupValue(Group group, int depth, int fieldIndex, ValueNodeContainer valueContainer)
protected Record	readImpl() Overridden by subclasses to read the next record from this DataReader.
protected void	readPrimitiveValue(Group group, int fieldIndex, ValueNodeContainer valueContainer, Type fieldType)
protected Record	readRecord(Group group, int depth)
ParquetDataReader	setConfiguration(Configuration configuration) Sets the Parquet configuration parameters.
ParquetDataReader	setDebug(boolean debug) Indicates if debugging is enabled to print log statements.
ParquetDataReader	setFilter(MetadataFilter filter) Sets the filter settings.
ParquetDataReader	setMakeOptionalFieldsRequired(boolean makeOptionalFieldsRequired) Used whether to check for optional columns that can be made required using ColumnChunkMetaData#getStatistics() during open().
ParquetDataReader	setMakeRequiredFieldsOptional(boolean makeRequiredFieldsOptional) When true, check for required columns that can be made optional using ColumnChunkMetaData#getStatistics().
ParquetDataReader	setRemoveFieldsWithoutColumnMetadata(boolean removeFieldsWithoutColumnMetadata) Used whether to remove fields in the schema does not have a corresponding ColumnChunkMetaData in the file during open().
ParquetDataReader	setRemoveFieldsWithoutValues(boolean removeFieldsWithoutValues) Used whether to remove fields in the schema whose value count is <= 0 during open().
ParquetDataReader	setSchema(MessageType schema) Indicates the schema used to read the file.

Methods inherited from class com.northconcepts.datapipeline.core.DataReader

available, getBufferSize, getNestedEndpoint, getNestedReader, getRootEndpoint, getRootReader, isExhausted, isSaveLineage, peek, pop, push, read, setSaveLineage, skip

Methods inherited from class com.northconcepts.datapipeline.core.DataEndpoint

decrementRecordCount, enableJmx, getLastRecord, getRecordCount, getRecordCountAsBigInteger, getRecordCountAsString, incrementRecordCount, isRecordCountBigInteger, resetRecordCount, toString

Methods inherited from class com.northconcepts.datapipeline.core.Endpoint

addElapsedtime, assertClosed, assertNotOpened, assertOpened, finalize, getClosedOn, getDescription, getElapsedTime, getElapsedTimeAsString, getOpenedOn, getOpenElapsedTime, getOpenElapsedTimeAsString, getSelfTime, getSelfTimeAsString, getState, isCaptureElapsedTime, isClosed, isOpen, setCaptureElapsedTime, setDescription

Methods inherited from class com.northconcepts.datapipeline.core.DataObject

exception, exception, exception, getId, getName, resetID

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

ParquetDataReader

public ParquetDataReader(InputFile inputFile)

Reads parquet data from an InputFile.

Method Detail

isDebug

public boolean isDebug()

Indicates if debugging is enabled to print log statements.

setDebug

public ParquetDataReader setDebug(boolean debug)

Indicates if debugging is enabled to print log statements.

getConfiguration

public Configuration getConfiguration()

Returns the Parquet configuration parameters.

setConfiguration

public ParquetDataReader setConfiguration(Configuration configuration)

Sets the Parquet configuration parameters.

getFilter

public MetadataFilter getFilter()

Returns the filter settings.

setFilter

public ParquetDataReader setFilter(MetadataFilter filter)

Sets the filter settings.

getSchema

public MessageType getSchema()

Indicates the schema used to read the file.

getModifiedSchema

public MessageType getModifiedSchema()

Indicates the modified schema used to read the file.

setSchema

public ParquetDataReader setSchema(MessageType schema)

Indicates the schema used to read the file.

isMakeRequiredFieldsOptional

public boolean isMakeRequiredFieldsOptional()

When true, check for required columns that can be made optional using ColumnChunkMetaData#getStatistics(). See getRequiredColumnsWithNullValues() for details.

setMakeRequiredFieldsOptional

public ParquetDataReader setMakeRequiredFieldsOptional(boolean makeRequiredFieldsOptional)

When true, check for required columns that can be made optional using ColumnChunkMetaData#getStatistics(). See getRequiredColumnsWithNullValues() for details.

isMakeOptionalFieldsRequired

public boolean isMakeOptionalFieldsRequired()

Used whether to check for optional columns that can be made required using ColumnChunkMetaData#getStatistics() during open(). See getOptionalColumnsWithoutNullValues() ()} for details.

setMakeOptionalFieldsRequired

public ParquetDataReader setMakeOptionalFieldsRequired(boolean makeOptionalFieldsRequired)

Used whether to check for optional columns that can be made required using ColumnChunkMetaData#getStatistics() during open(). See getOptionalColumnsWithoutNullValues() ()} for details.

isRemoveFieldsWithoutColumnMetadata

public boolean isRemoveFieldsWithoutColumnMetadata()

When true, remove fields in the schema does not have a corresponding ColumnChunkMetaData in the file. See getFieldsWithoutColumnMetadata() for details.

setRemoveFieldsWithoutColumnMetadata

public ParquetDataReader setRemoveFieldsWithoutColumnMetadata(boolean removeFieldsWithoutColumnMetadata)

Used whether to remove fields in the schema does not have a corresponding ColumnChunkMetaData in the file during open(). See getFieldsWithoutColumnMetadata() for details.

isRemoveFieldsWithoutValues

public boolean isRemoveFieldsWithoutValues()

When true, remove fields in the schema whose value count is <= 0 See getFieldsWithoutColumnMetadata() for details.

setRemoveFieldsWithoutValues

public ParquetDataReader setRemoveFieldsWithoutValues(boolean removeFieldsWithoutValues)

Used whether to remove fields in the schema whose value count is <= 0 during open(). See getFieldsWithoutColumnMetadata() for details.

open

public void open()
          throws DataException

Description copied from class: DataEndpoint

Makes this endpoint ready for reading or writing.

Overrides:

open in class IntegrationReader

Throws:

DataException

close

public void close()
           throws DataException

Description copied from class: DataEndpoint

Indicates that this endpoint has finished reading or writing.

Overrides:

close in class DataEndpoint

Throws:

DataException

readImpl

protected Record readImpl()
                   throws Throwable

Description copied from class: DataReader

Overridden by subclasses to read the next record from this DataReader. The default implementation of DataReader.read() now insures that this method will not be called again after it returns a null.

If no record is available, null will be returned.

Specified by:

readImpl in class DataReader

Throws:

Throwable

readRecord

protected Record readRecord(Group group,
                            int depth)

readGroupValue

protected void readGroupValue(Group group,
                              int depth,
                              int fieldIndex,
                              ValueNodeContainer valueContainer)

readPrimitiveValue

protected void readPrimitiveValue(Group group,
                                  int fieldIndex,
                                  ValueNodeContainer valueContainer,
                                  Type fieldType)

int96ToTimestamp

protected Date int96ToTimestamp(byte[] int96Bytes)

isLineageSupported

public boolean isLineageSupported()

Overrides:

isLineageSupported in class DataReader

addLineage

protected Record addLineage(Record record)

Overrides:

addLineage in class DataReader

addExceptionProperties

public DataException addExceptionProperties(DataException exception)

Description copied from class: Endpoint

Adds this endpoint's current state to a DataException. Since this method is called whenever an exception is thrown, subclasses should override it to add their specific information.

Overrides:

addExceptionProperties in class DataReader