com.northconcepts.datapipeline.orc

Class OrcDataReader

java.lang.Object

com.northconcepts.datapipeline.core.DataObject

com.northconcepts.datapipeline.core.Endpoint

com.northconcepts.datapipeline.core.DataEndpoint

com.northconcepts.datapipeline.core.DataReader

com.northconcepts.datapipeline.internal.lang.IntegrationReader

com.northconcepts.datapipeline.orc.OrcDataReader

public class OrcDataReader
extends IntegrationReader

Read records from Apache ORC columnar files. See Apache ORC columnar storage.

Nested Class Summary

Nested classes/interfaces inherited from class com.northconcepts.datapipeline.core.DataEndpoint

DataEndpoint.State

Field Summary

Fields inherited from class com.northconcepts.datapipeline.core.DataReader

fieldLineage, recordLineage

Fields inherited from class com.northconcepts.datapipeline.core.DataEndpoint

lastRecord, PRODUCT, PRODUCT_VERSION, VENDOR, XML_INPUT_FACTORY_KEY

Fields inherited from class com.northconcepts.datapipeline.core.Endpoint

BUFFER_SIZE, captureElapsedTime, DEFAULT_READ_BUFFER_SIZE

Fields inherited from class com.northconcepts.datapipeline.core.DataObject

id, log, name, TIMESTAMP_FORMAT

Constructor Summary

Constructors

Constructor and Description
OrcDataReader(File file) Reads ORC data from a file.
OrcDataReader(Path path) Read ORC data from a Path.

Method Summary

Modifier and Type	Method and Description
DataException	addExceptionProperties(DataException exception) Adds this endpoint's current state to a DataException.
protected Record	addLineage(Record record)
void	close() Indicates that this endpoint has finished reading or writing.
int	getBatchSize() Indicates the maximum number of records to buffer when reading ORC data (default 1024).
FieldList	getColumns() Indicates the columns to read from ORC file.
Configuration	getConfig() Returns the Orc configuration parameters.
Path	getPath() Returns the Path of the ORC file being read.
TypeDescription	getSchema() Indicates the schema used to read the file.
boolean	isLineageSupported()
void	open() Makes this endpoint ready for reading or writing.
protected Record	readImpl() Overridden by subclasses to read the next record from this DataReader.
OrcDataReader	setBatchSize(int batchSize) Indicates the maximum number of records to buffer when reading ORC data (default 1024).
OrcDataReader	setColumns(FieldList columns) Indicates the columns to read from ORC file.
OrcDataReader	setConfig(Configuration config) Sets the Orc configuration parameters.
OrcDataReader	setSchema(String schema) Indicates the schema used to read the file.
OrcDataReader	setSchema(TypeDescription schema) Indicates the schema used to read the file.

Methods inherited from class com.northconcepts.datapipeline.core.DataReader

available, getBufferSize, getNestedEndpoint, getNestedReader, getRootEndpoint, getRootReader, isExhausted, isSaveLineage, peek, pop, push, read, setSaveLineage, skip

Methods inherited from class com.northconcepts.datapipeline.core.DataEndpoint

decrementRecordCount, enableJmx, getLastRecord, getRecordCount, getRecordCountAsBigInteger, getRecordCountAsString, incrementRecordCount, isRecordCountBigInteger, resetRecordCount, toString

Methods inherited from class com.northconcepts.datapipeline.core.Endpoint

addElapsedtime, assertClosed, assertNotOpened, assertOpened, finalize, getClosedOn, getDescription, getElapsedTime, getElapsedTimeAsString, getOpenedOn, getOpenElapsedTime, getOpenElapsedTimeAsString, getSelfTime, getSelfTimeAsString, getState, isCaptureElapsedTime, isClosed, isOpen, setCaptureElapsedTime, setDescription

Methods inherited from class com.northconcepts.datapipeline.core.DataObject

exception, exception, exception, getId, getName, resetID

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

OrcDataReader

public OrcDataReader(File file)

Reads ORC data from a file.

OrcDataReader

public OrcDataReader(Path path)

Read ORC data from a Path.

Method Detail

open

public void open()
          throws DataException

Description copied from class: DataEndpoint

Makes this endpoint ready for reading or writing.

Overrides:

open in class IntegrationReader

Throws:

DataException

close

public void close()
           throws DataException

Description copied from class: DataEndpoint

Indicates that this endpoint has finished reading or writing.

Overrides:

close in class DataEndpoint

Throws:

DataException

readImpl

protected Record readImpl()
                   throws Throwable

Description copied from class: DataReader

Overridden by subclasses to read the next record from this DataReader. The default implementation of DataReader.read() now insures that this method will not be called again after it returns a null.

If no record is available, null will be returned.

Specified by:

readImpl in class DataReader

Throws:

Throwable

getBatchSize

public int getBatchSize()

Indicates the maximum number of records to buffer when reading ORC data (default 1024).

setBatchSize

public OrcDataReader setBatchSize(int batchSize)

Indicates the maximum number of records to buffer when reading ORC data (default 1024).

getColumns

public FieldList getColumns()

Indicates the columns to read from ORC file.

setColumns

public OrcDataReader setColumns(FieldList columns)

Indicates the columns to read from ORC file.

getSchema

public TypeDescription getSchema()

Indicates the schema used to read the file.

setSchema

public OrcDataReader setSchema(String schema)

Indicates the schema used to read the file.

setSchema

public OrcDataReader setSchema(TypeDescription schema)

Indicates the schema used to read the file.

getConfig

public Configuration getConfig()

Returns the Orc configuration parameters.

setConfig

public OrcDataReader setConfig(Configuration config)

Sets the Orc configuration parameters.

isLineageSupported

public boolean isLineageSupported()

Overrides:

isLineageSupported in class DataReader

addLineage

protected Record addLineage(Record record)

Overrides:

addLineage in class DataReader

addExceptionProperties

public DataException addExceptionProperties(DataException exception)

Description copied from class: Endpoint

Adds this endpoint's current state to a DataException. Since this method is called whenever an exception is thrown, subclasses should override it to add their specific information.

Overrides:

addExceptionProperties in class DataReader

getPath

public Path getPath()

Returns the Path of the ORC file being read.