com.northconcepts.datapipeline.core

Class RemoveDuplicatesReader

java.lang.Object

com.northconcepts.datapipeline.core.DataObject

com.northconcepts.datapipeline.core.Endpoint

com.northconcepts.datapipeline.core.DataEndpoint

com.northconcepts.datapipeline.core.DataReader

com.northconcepts.datapipeline.core.ProxyReader

com.northconcepts.datapipeline.core.RemoveDuplicatesReader

public class RemoveDuplicatesReader
extends ProxyReader

A proxy that removes duplicate records. Duplicates can be determined by checking all fields (RemoveDuplicatesReader(DataReader)) or subset of fields (RemoveDuplicatesReader(DataReader, FieldList)).

Nested Class Summary

Nested classes/interfaces inherited from class com.northconcepts.datapipeline.core.DataEndpoint

DataEndpoint.State

Field Summary

Fields inherited from class com.northconcepts.datapipeline.core.DataReader

fieldLineage, recordLineage

Fields inherited from class com.northconcepts.datapipeline.core.DataEndpoint

lastRecord, PRODUCT, PRODUCT_VERSION, VENDOR, XML_INPUT_FACTORY_KEY

Fields inherited from class com.northconcepts.datapipeline.core.Endpoint

BUFFER_SIZE, captureElapsedTime, DEFAULT_READ_BUFFER_SIZE

Fields inherited from class com.northconcepts.datapipeline.core.DataObject

id, log, name, TIMESTAMP_FORMAT

Constructor Summary

Constructors

Constructor and Description
RemoveDuplicatesReader(DataReader targetDataReader) Remove duplicate records matching all fields.
RemoveDuplicatesReader(DataReader targetDataReader, FieldList fields) Removes duplicate records based on the matching FieldList.
RemoveDuplicatesReader(DataReader targetDataReader, String... fields) Remove duplicate records based on their field names.

Method Summary

Modifier and Type	Method and Description
DataException	addExceptionProperties(DataException exception) Adds this endpoint's current state to a DataException.
void	close() Indicates that this endpoint has finished reading or writing.
DataWriter	getDiscardWriter() Returns the discard sink for discarded records or null if one was not assigned.
long	getDuplicateRecordCount() Returns the number of duplicate records that have been read.
long	getUniqueRecordCount() Returns the number of unique records that have been read.
protected Record	interceptRecord(Record record)
protected void	onDuplicate(Record record) Called for each duplicate record.
protected void	onUnique(Record record) Called for each unique record.
void	open() Makes this endpoint ready for reading or writing.
RemoveDuplicatesReader	setDiscardWriter(DataWriter writer) Assign a discard sink for duplicate records.

Methods inherited from class com.northconcepts.datapipeline.core.ProxyReader

available, getNestedReader, map, map, readImpl, setNestedDataReader, setNestedDataReader

Methods inherited from class com.northconcepts.datapipeline.core.DataReader

addLineage, getBufferSize, getNestedEndpoint, getReader, getRootEndpoint, getRootReader, isExhausted, isLineageSupported, isSaveLineage, peek, pop, push, read, setSaveLineage, skip

Methods inherited from class com.northconcepts.datapipeline.core.DataEndpoint

decrementRecordCount, enableJmx, getLastRecord, getRecordCount, getRecordCountAsBigInteger, getRecordCountAsString, incrementRecordCount, isRecordCountBigInteger, resetRecordCount, toString

Methods inherited from class com.northconcepts.datapipeline.core.Endpoint

addElapsedtime, assertClosed, assertNotOpened, assertOpened, finalize, getClosedOn, getDescription, getElapsedTime, getElapsedTimeAsString, getOpenedOn, getOpenElapsedTime, getOpenElapsedTimeAsString, getSelfTime, getSelfTimeAsString, getState, isCaptureElapsedTime, isClosed, isOpen, setCaptureElapsedTime, setDescription

Methods inherited from class com.northconcepts.datapipeline.core.DataObject

exception, exception, exception, getId, getName, resetID

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

RemoveDuplicatesReader

public RemoveDuplicatesReader(DataReader targetDataReader,
                              FieldList fields)

Removes duplicate records based on the matching FieldList.

Parameters:

targetDataReader - source DataReader

fields - the FieldList to be match

RemoveDuplicatesReader

public RemoveDuplicatesReader(DataReader targetDataReader,
                              String... fields)

Remove duplicate records based on their field names.

Parameters:

targetDataReader - source DataReader

fields - the field names to be checked for duplicates

RemoveDuplicatesReader

public RemoveDuplicatesReader(DataReader targetDataReader)

Remove duplicate records matching all fields.

Parameters:

targetDataReader - source DataReader

Method Detail

getDiscardWriter

public DataWriter getDiscardWriter()

Returns the discard sink for discarded records or null if one was not assigned.

setDiscardWriter

public RemoveDuplicatesReader setDiscardWriter(DataWriter writer)

Assign a discard sink for duplicate records.

Parameters:

writer - the discard sink

getUniqueRecordCount

public long getUniqueRecordCount()

Returns the number of unique records that have been read.

getDuplicateRecordCount

public long getDuplicateRecordCount()

Returns the number of duplicate records that have been read.

open

public void open()
          throws DataException

Description copied from class: DataEndpoint

Makes this endpoint ready for reading or writing.

Overrides:

open in class ProxyReader

Throws:

DataException

close

public void close()
           throws DataException

Description copied from class: DataEndpoint

Indicates that this endpoint has finished reading or writing.

Overrides:

close in class ProxyReader

Throws:

DataException

interceptRecord

protected Record interceptRecord(Record record)
                          throws Throwable

Overrides:

interceptRecord in class ProxyReader

Throws:

Throwable

onUnique

protected void onUnique(Record record)

Called for each unique record.

onDuplicate

protected void onDuplicate(Record record)

Called for each duplicate record.

addExceptionProperties

public DataException addExceptionProperties(DataException exception)

Description copied from class: Endpoint

Adds this endpoint's current state to a DataException. Since this method is called whenever an exception is thrown, subclasses should override it to add their specific information.

Overrides:

addExceptionProperties in class ProxyReader