com.northconcepts.datapipeline.core

Class DataReader

java.lang.Object

com.northconcepts.datapipeline.core.DataObject

com.northconcepts.datapipeline.core.Endpoint

com.northconcepts.datapipeline.core.DataEndpoint

com.northconcepts.datapipeline.core.DataReader

Direct Known Subclasses:

AbstractReader, AsyncMultiReader, DataReaderIterator, Dataset.ColumnsDataReader, DatasetReader, EventBusReader, FileReader, FileSystemInfoReader, IntegrationReader, JdbcReader, JsonRecordReader, MemoryReader, NullReader, PipedReader, ProxyReader, SequenceReader, SimpleJsonReader, SimpleXmlReader, SplitReader, XmlReader, XmlRecordReader

public abstract class DataReader
extends DataEndpoint

Abstract super-class for reading records. The only method that a subclass must implement is readImpl(), however, most subclasses will also override DataEndpoint.open() and DataEndpoint.close().

Nested Class Summary

Nested classes/interfaces inherited from class com.northconcepts.datapipeline.core.DataEndpoint

DataEndpoint.State

Field Summary

Fields

Modifier and Type	Field and Description
protected FieldLineage	fieldLineage
protected RecordLineage	recordLineage

Fields inherited from class com.northconcepts.datapipeline.core.DataEndpoint

lastRecord, PRODUCT, PRODUCT_VERSION, VENDOR, XML_INPUT_FACTORY_KEY

Fields inherited from class com.northconcepts.datapipeline.core.Endpoint

BUFFER_SIZE, captureElapsedTime, DEFAULT_READ_BUFFER_SIZE

Fields inherited from class com.northconcepts.datapipeline.core.DataObject

id, log, name, TIMESTAMP_FORMAT

Constructor Summary

Constructors

Constructor and Description
DataReader()

Method Summary

Modifier and Type	Method and Description
DataException	addExceptionProperties(DataException exception) Adds this endpoint's current state to a DataException.
protected Record	addLineage(Record record)
int	available() Returns the number of records that can probably be read without blocking.
int	getBufferSize() Returns the number of records in this DataReaders buffer.
DataReader	getNestedEndpoint() Returns the Endpoint held inside this one or null if there isn't one.
DataReader	getNestedReader() Returns the DataReader held inside this one or null if there isn't one.
DataReader	getRootEndpoint() Returns the deepest, nested Endpoint held inside this one, otherwise this instance is returned if there aren't any nested Endpoints.
DataReader	getRootReader() Returns the deepest, nested DataReader held inside this one, otherwise this instance is returned if there aren't any nested DataReaders.
boolean	isExhausted() Returns true if this stream has already returned a null indicating that no further reads are possible.
boolean	isLineageSupported()
boolean	isSaveLineage()
Record	peek(int index) Looks ahead to return the next record at the specified index without reading it or null if one does not exist.
protected Record	pop() Removes and returns the next record in this DataReaders buffer or null if it is empty.
void	push(Record record) Adds a record to this DataReaders buffer.
Record	read() Reads the next record from this DataReader and increases the record-count by 1.
protected abstract Record	readImpl() Overridden by subclasses to read the next record from this DataReader.
DataReader	setSaveLineage(boolean saveLineage)
int	skip(int count) Skips over the specified number of records.

Methods inherited from class com.northconcepts.datapipeline.core.DataEndpoint

close, decrementRecordCount, enableJmx, getLastRecord, getRecordCount, getRecordCountAsBigInteger, getRecordCountAsString, incrementRecordCount, isRecordCountBigInteger, open, resetRecordCount, toString

Methods inherited from class com.northconcepts.datapipeline.core.Endpoint

addElapsedtime, assertClosed, assertNotOpened, assertOpened, finalize, getClosedOn, getDescription, getElapsedTime, getElapsedTimeAsString, getOpenedOn, getOpenElapsedTime, getOpenElapsedTimeAsString, getSelfTime, getSelfTimeAsString, getState, isCaptureElapsedTime, isClosed, isOpen, setCaptureElapsedTime, setDescription

Methods inherited from class com.northconcepts.datapipeline.core.DataObject

exception, exception, exception, getId, getName, resetID

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

recordLineage

protected final RecordLineage recordLineage

fieldLineage

protected final FieldLineage fieldLineage

Constructor Detail

DataReader

public DataReader()

Method Detail

getNestedEndpoint

public DataReader getNestedEndpoint()

Description copied from class: Endpoint

Returns the Endpoint held inside this one or null if there isn't one.

Overrides:

getNestedEndpoint in class DataEndpoint

getRootEndpoint

public DataReader getRootEndpoint()

Description copied from class: Endpoint

Returns the deepest, nested Endpoint held inside this one, otherwise this instance is returned if there aren't any nested Endpoints.

Overrides:

getRootEndpoint in class DataEndpoint

getNestedReader

public DataReader getNestedReader()

Returns the DataReader held inside this one or null if there isn't one.

getRootReader

public DataReader getRootReader()

Returns the deepest, nested DataReader held inside this one, otherwise this instance is returned if there aren't any nested DataReaders.

available

public int available()
              throws DataException

Returns the number of records that can probably be read without blocking.

Throws:

DataException

getBufferSize

public final int getBufferSize()

Returns the number of records in this DataReaders buffer.

isExhausted

public boolean isExhausted()

Returns true if this stream has already returned a null indicating that no further reads are possible.

push

public void push(Record record)

Adds a record to this DataReaders buffer. Records in the buffer will be returned by read() before attempting to read from the underlying implementation.

See Also:

read()

pop

protected Record pop()

Removes and returns the next record in this DataReaders buffer or null if it is empty.

See Also:

push(Record)

peek

public final Record peek(int index)

Looks ahead to return the next record at the specified index without reading it or null if one does not exist. The buffer of look ahead records are added to this DataReader using push(Record).

Parameters:

index - specifies how far to look ahead (0 refers to the next record, 1--to the one after that, etc.)

Returns:

the record at the look ahead position or null if one does not exist

readImpl

protected abstract Record readImpl()
                            throws Throwable

Overridden by subclasses to read the next record from this DataReader. The default implementation of read() now insures that this method will not be called again after it returns a null.

If no record is available, null will be returned.

Throws:

Throwable

read

public Record read()
            throws DataException

Reads the next record from this DataReader and increases the record-count by 1. This method will first read any pushed (push(Record)) records before reading from the underlying source.

If no record is available, null will be returned. This method blocks until a record is available, the end of the stream is reached, or an exception is thrown.

Any exception raised while reading will be converted to a DataException using DataObject.exception(Throwable).

Subclasses generally do not need to override this method, instead they should implement readImpl().

Throws:

DataException

See Also:

push(Record), peek(int), read()

addLineage

protected Record addLineage(Record record)

skip

public int skip(int count)
         throws DataException

Skips over the specified number of records. Since there may be fewer records than the specified number, this method returns the actual number skipped.

Parameters:

count - the number of records to skip

Returns:

The actual number of records skipped

Throws:

DataException

isLineageSupported

public boolean isLineageSupported()

isSaveLineage

public boolean isSaveLineage()

setSaveLineage

public DataReader setSaveLineage(boolean saveLineage)

addExceptionProperties

public DataException addExceptionProperties(DataException exception)

Description copied from class: Endpoint

Adds this endpoint's current state to a DataException. Since this method is called whenever an exception is thrown, subclasses should override it to add their specific information.

Overrides:

addExceptionProperties in class DataEndpoint