com.northconcepts.datapipeline.group

Class GroupByReader

java.lang.Object

com.northconcepts.datapipeline.core.DataObject

com.northconcepts.datapipeline.core.Endpoint

com.northconcepts.datapipeline.core.DataEndpoint

com.northconcepts.datapipeline.core.DataReader

com.northconcepts.datapipeline.core.ProxyReader

com.northconcepts.datapipeline.group.GroupByReader

public class GroupByReader
extends ProxyReader

A proxy that divides records into groups and applies summary operations to each group; similar to "group by" in SQL, but applied to streaming data.

Nested Class Summary

Nested classes/interfaces inherited from class com.northconcepts.datapipeline.core.DataEndpoint

DataEndpoint.State

Field Summary

Fields inherited from class com.northconcepts.datapipeline.core.DataReader

fieldLineage, recordLineage

Fields inherited from class com.northconcepts.datapipeline.core.DataEndpoint

lastRecord, PRODUCT, PRODUCT_VERSION, VENDOR, XML_INPUT_FACTORY_KEY

Fields inherited from class com.northconcepts.datapipeline.core.Endpoint

BUFFER_SIZE, captureElapsedTime, DEFAULT_READ_BUFFER_SIZE

Fields inherited from class com.northconcepts.datapipeline.core.DataObject

id, log, name, TIMESTAMP_FORMAT

Constructor Summary

Constructors

Constructor and Description
GroupByReader(DataReader reader, FieldList groupByFields)
GroupByReader(DataReader reader, int queueSize, FieldList groupByFields)
GroupByReader(DataReader reader, int queueSize, String... groupByFields)
GroupByReader(DataReader reader, String... groupByFields)

Method Summary

Modifier and Type	Method and Description
GroupByReader	add(GroupOperation<?> operation)
DataException	addExceptionProperties(DataException exception) Adds this endpoint's current state to a DataException.
GroupByReader	avg(FieldPath sourceFieldName, FieldPath targetFieldName)
GroupByReader	avg(FieldPath sourceFieldName, FieldPath targetFieldName, int scale, RoundingMode roundingMode)
GroupByReader	avg(String sourceFieldName, String targetFieldName)
GroupByReader	avg(String sourceFieldName, String targetFieldName, int scale, RoundingMode roundingMode)
void	close() Indicates that this endpoint has finished reading or writing.
GroupByReader	collect(FieldPath sourceFieldName, boolean excludeNulls, boolean distinctValues, boolean flattenArrayValues) Collects the individual values into an array, using the source field name as the target field name.
GroupByReader	collect(FieldPath sourceFieldName, FieldPath targetFieldName, boolean distinctValues, boolean flattenArrayValues) Collects the individual values into an array, excluding nulls by default.
GroupByReader	collect(FieldPath sourceFieldName, FieldPath targetFieldName, boolean distinctValues, boolean flattenArrayValues, boolean excludeNulls) Collects the individual values into an array.
GroupByReader	collect(String sourceFieldName, boolean excludeNulls, boolean distinctValues, boolean flattenArrayValues) Collects the individual values into an array, using the source field name as the target field name.
GroupByReader	collect(String sourceFieldName, String targetFieldName, boolean distinctValues, boolean flattenArrayValues) Collects the individual values into an array, excluding nulls by default.
GroupByReader	collect(String sourceFieldName, String targetFieldName, boolean distinctValues, boolean flattenArrayValues, boolean excludeNulls) Collects the individual values into an array.
GroupByReader	count(FieldPath targetFieldName)
GroupByReader	count(FieldPath targetFieldName, boolean excludeNulls)
GroupByReader	count(String targetFieldName)
GroupByReader	count(String targetFieldName, boolean excludeNulls)
GroupByReader	first(FieldPath sourceFieldName, boolean excludeNulls)
GroupByReader	first(FieldPath sourceFieldName, FieldPath targetFieldName)
GroupByReader	first(FieldPath sourceFieldName, FieldPath targetFieldName, boolean excludeNulls)
GroupByReader	first(String sourceFieldName, boolean excludeNulls)
GroupByReader	first(String sourceFieldName, String targetFieldName)
GroupByReader	first(String sourceFieldName, String targetFieldName, boolean excludeNulls)
List<Window>	getClosedWindows()
CloseWindowStrategy	getCloseWindowStrategy()
CreateWindowStrategy	getCreateWindowStrategy()
FieldList	getGroupByFields()
List<Window>	getOpenedWindows()
List<GroupOperation<?>>	getOperations()
long	getWindowsClosed()
long	getWindowsCreated()
boolean	isDebug()
boolean	isExcludeNulls()
GroupByReader	last(FieldPath sourceFieldName, boolean excludeNulls)
GroupByReader	last(FieldPath sourceFieldName, FieldPath targetFieldName)
GroupByReader	last(FieldPath sourceFieldName, FieldPath targetFieldName, boolean excludeNulls)
GroupByReader	last(String sourceFieldName, boolean excludeNulls)
GroupByReader	last(String sourceFieldName, String targetFieldName)
GroupByReader	last(String sourceFieldName, String targetFieldName, boolean excludeNulls)
GroupByReader	max(FieldPath sourceFieldName)
GroupByReader	max(FieldPath sourceFieldName, boolean excludeNulls)
GroupByReader	max(FieldPath sourceFieldName, FieldPath targetFieldName)
GroupByReader	max(FieldPath sourceFieldName, FieldPath targetFieldName, boolean excludeNulls)
GroupByReader	max(String sourceFieldName)
GroupByReader	max(String sourceFieldName, boolean excludeNulls)
GroupByReader	max(String sourceFieldName, String targetFieldName)
GroupByReader	max(String sourceFieldName, String targetFieldName, boolean excludeNulls)
GroupByReader	min(FieldPath sourceFieldName)
GroupByReader	min(FieldPath sourceFieldName, boolean excludeNulls)
GroupByReader	min(FieldPath sourceFieldName, FieldPath targetFieldName)
GroupByReader	min(FieldPath sourceFieldName, FieldPath targetFieldName, boolean excludeNulls)
GroupByReader	min(String sourceFieldName)
GroupByReader	min(String sourceFieldName, boolean excludeNulls)
GroupByReader	min(String sourceFieldName, String targetFieldName)
GroupByReader	min(String sourceFieldName, String targetFieldName, boolean excludeNulls)
void	open() Makes this endpoint ready for reading or writing.
protected Record	readImpl() Overridden by subclasses to read the next record from this DataReader.
GroupByReader	setCloseWindowStrategy(CloseWindowStrategy closeWindowStrategy)
GroupByReader	setCreateWindowStrategy(CreateWindowStrategy createWindowStrategy)
GroupByReader	setDebug(boolean debug)
GroupByReader	setExcludeNulls(boolean excludeNulls) Indicates if null groups should be excluded from results (defaults to false).
GroupByReader	sum(FieldPath sourceFieldName)
GroupByReader	sum(FieldPath sourceFieldName, FieldPath targetFieldName)
GroupByReader	sum(String sourceFieldName)
GroupByReader	sum(String sourceFieldName, String targetFieldName)
String	toString()

Methods inherited from class com.northconcepts.datapipeline.core.ProxyReader

available, getNestedReader, interceptRecord, map, map, setNestedDataReader, setNestedDataReader

Methods inherited from class com.northconcepts.datapipeline.core.DataReader

addLineage, getBufferSize, getNestedEndpoint, getReader, getRootEndpoint, getRootReader, isExhausted, isLineageSupported, isSaveLineage, peek, pop, push, read, setSaveLineage, skip

Methods inherited from class com.northconcepts.datapipeline.core.DataEndpoint

decrementRecordCount, enableJmx, getLastRecord, getRecordCount, getRecordCountAsBigInteger, getRecordCountAsString, incrementRecordCount, isRecordCountBigInteger, resetRecordCount

Methods inherited from class com.northconcepts.datapipeline.core.Endpoint

addElapsedtime, assertClosed, assertNotOpened, assertOpened, finalize, getClosedOn, getDescription, getElapsedTime, getElapsedTimeAsString, getOpenedOn, getOpenElapsedTime, getOpenElapsedTimeAsString, getSelfTime, getSelfTimeAsString, getState, isCaptureElapsedTime, isClosed, isOpen, setCaptureElapsedTime, setDescription

Methods inherited from class com.northconcepts.datapipeline.core.DataObject

exception, exception, exception, getId, getName, resetID

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

GroupByReader

public GroupByReader(DataReader reader,
                     String... groupByFields)

GroupByReader

public GroupByReader(DataReader reader,
                     int queueSize,
                     String... groupByFields)

GroupByReader

public GroupByReader(DataReader reader,
                     FieldList groupByFields)

GroupByReader

public GroupByReader(DataReader reader,
                     int queueSize,
                     FieldList groupByFields)

Method Detail

getGroupByFields

public FieldList getGroupByFields()

isExcludeNulls

public boolean isExcludeNulls()

setExcludeNulls

public GroupByReader setExcludeNulls(boolean excludeNulls)

Indicates if null groups should be excluded from results (defaults to false). For example, when grouping on country and city, any cases where both country and city are null will be skipped if set to true.

isDebug

public boolean isDebug()

setDebug

public GroupByReader setDebug(boolean debug)

getOperations

public List<GroupOperation<?>> getOperations()

add

public GroupByReader add(GroupOperation<?> operation)

count

public GroupByReader count(FieldPath targetFieldName)

count

public GroupByReader count(String targetFieldName)

count

public GroupByReader count(FieldPath targetFieldName,
                           boolean excludeNulls)

count

public GroupByReader count(String targetFieldName,
                           boolean excludeNulls)

sum

public GroupByReader sum(FieldPath sourceFieldName,
                         FieldPath targetFieldName)

sum

public GroupByReader sum(FieldPath sourceFieldName)

sum

public GroupByReader sum(String sourceFieldName,
                         String targetFieldName)

sum

public GroupByReader sum(String sourceFieldName)

avg

public GroupByReader avg(FieldPath sourceFieldName,
                         FieldPath targetFieldName)

avg

public GroupByReader avg(String sourceFieldName,
                         String targetFieldName)

avg

public GroupByReader avg(FieldPath sourceFieldName,
                         FieldPath targetFieldName,
                         int scale,
                         RoundingMode roundingMode)

avg

public GroupByReader avg(String sourceFieldName,
                         String targetFieldName,
                         int scale,
                         RoundingMode roundingMode)

first

public GroupByReader first(FieldPath sourceFieldName,
                           FieldPath targetFieldName,
                           boolean excludeNulls)

first

public GroupByReader first(String sourceFieldName,
                           String targetFieldName,
                           boolean excludeNulls)

first

public GroupByReader first(FieldPath sourceFieldName,
                           FieldPath targetFieldName)

first

public GroupByReader first(String sourceFieldName,
                           String targetFieldName)

first

public GroupByReader first(FieldPath sourceFieldName,
                           boolean excludeNulls)

first

public GroupByReader first(String sourceFieldName,
                           boolean excludeNulls)

last

public GroupByReader last(FieldPath sourceFieldName,
                          FieldPath targetFieldName,
                          boolean excludeNulls)

last

public GroupByReader last(String sourceFieldName,
                          String targetFieldName,
                          boolean excludeNulls)

last

public GroupByReader last(FieldPath sourceFieldName,
                          FieldPath targetFieldName)

last

public GroupByReader last(String sourceFieldName,
                          String targetFieldName)

last

public GroupByReader last(FieldPath sourceFieldName,
                          boolean excludeNulls)

last

public GroupByReader last(String sourceFieldName,
                          boolean excludeNulls)

max

public GroupByReader max(FieldPath sourceFieldName,
                         FieldPath targetFieldName,
                         boolean excludeNulls)

max

public GroupByReader max(String sourceFieldName,
                         String targetFieldName,
                         boolean excludeNulls)

max

public GroupByReader max(FieldPath sourceFieldName,
                         FieldPath targetFieldName)

max

public GroupByReader max(String sourceFieldName,
                         String targetFieldName)

max

public GroupByReader max(FieldPath sourceFieldName,
                         boolean excludeNulls)

max

public GroupByReader max(String sourceFieldName,
                         boolean excludeNulls)

max

public GroupByReader max(FieldPath sourceFieldName)

max

public GroupByReader max(String sourceFieldName)

min

public GroupByReader min(FieldPath sourceFieldName,
                         FieldPath targetFieldName,
                         boolean excludeNulls)

min

public GroupByReader min(String sourceFieldName,
                         String targetFieldName,
                         boolean excludeNulls)

min

public GroupByReader min(FieldPath sourceFieldName,
                         FieldPath targetFieldName)

min

public GroupByReader min(String sourceFieldName,
                         String targetFieldName)

min

public GroupByReader min(FieldPath sourceFieldName,
                         boolean excludeNulls)

min

public GroupByReader min(String sourceFieldName,
                         boolean excludeNulls)

min

public GroupByReader min(FieldPath sourceFieldName)

min

public GroupByReader min(String sourceFieldName)

collect

public GroupByReader collect(FieldPath sourceFieldName,
                             FieldPath targetFieldName,
                             boolean distinctValues,
                             boolean flattenArrayValues,
                             boolean excludeNulls)

Collects the individual values into an array.

Parameters:

sourceFieldName - the path of the field to collect values from

targetFieldName - the path of the field where the collected array will be placed

distinctValues - true to collect only distinct values, false to allow duplicates

flattenArrayValues - true to flatten array values (add elements individually instead of as nested arrays)

excludeNulls - true to exclude null values from the collection

Returns:

this GroupByReader instance for method chaining

See Also:

GroupCollect

collect

public GroupByReader collect(String sourceFieldName,
                             String targetFieldName,
                             boolean distinctValues,
                             boolean flattenArrayValues,
                             boolean excludeNulls)

Collects the individual values into an array.

Parameters:

sourceFieldName - the name of the field to collect values from

targetFieldName - the name of the field where the collected array will be placed

distinctValues - true to collect only distinct values, false to allow duplicates

flattenArrayValues - true to flatten array values (add elements individually instead of as nested arrays)

excludeNulls - true to exclude null values from the collection

Returns:

this GroupByReader instance for method chaining

See Also:

GroupCollect

collect

public GroupByReader collect(FieldPath sourceFieldName,
                             FieldPath targetFieldName,
                             boolean distinctValues,
                             boolean flattenArrayValues)

Collects the individual values into an array, excluding nulls by default.

Parameters:

sourceFieldName - the path of the field to collect values from

targetFieldName - the path of the field where the collected array will be placed

distinctValues - true to collect only distinct values, false to allow duplicates

flattenArrayValues - true to flatten array values (add elements individually instead of as nested arrays)

Returns:

this GroupByReader instance for method chaining

See Also:

GroupCollect

collect

public GroupByReader collect(String sourceFieldName,
                             String targetFieldName,
                             boolean distinctValues,
                             boolean flattenArrayValues)

Collects the individual values into an array, excluding nulls by default.

Parameters:

sourceFieldName - the name of the field to collect values from

targetFieldName - the name of the field where the collected array will be placed

distinctValues - true to collect only distinct values, false to allow duplicates

flattenArrayValues - true to flatten array values (add elements individually instead of as nested arrays)

Returns:

this GroupByReader instance for method chaining

See Also:

GroupCollect

collect

public GroupByReader collect(FieldPath sourceFieldName,
                             boolean excludeNulls,
                             boolean distinctValues,
                             boolean flattenArrayValues)

Collects the individual values into an array, using the source field name as the target field name.

Parameters:

sourceFieldName - the path of the field to collect values from and place the result in

excludeNulls - true to exclude null values from the collection

distinctValues - true to collect only distinct values, false to allow duplicates

flattenArrayValues - true to flatten array values (add elements individually instead of as nested arrays)

Returns:

this GroupByReader instance for method chaining

See Also:

GroupCollect

collect

public GroupByReader collect(String sourceFieldName,
                             boolean excludeNulls,
                             boolean distinctValues,
                             boolean flattenArrayValues)

Collects the individual values into an array, using the source field name as the target field name.

Parameters:

sourceFieldName - the name of the field to collect values from and place the result in

excludeNulls - true to exclude null values from the collection

distinctValues - true to collect only distinct values, false to allow duplicates

flattenArrayValues - true to flatten array values (add elements individually instead of as nested arrays)

Returns:

this GroupByReader instance for method chaining

See Also:

GroupCollect

getCloseWindowStrategy

public CloseWindowStrategy getCloseWindowStrategy()

setCloseWindowStrategy

public GroupByReader setCloseWindowStrategy(CloseWindowStrategy closeWindowStrategy)

getCreateWindowStrategy

public CreateWindowStrategy getCreateWindowStrategy()

setCreateWindowStrategy

public GroupByReader setCreateWindowStrategy(CreateWindowStrategy createWindowStrategy)

getOpenedWindows

public List<Window> getOpenedWindows()

getClosedWindows

public List<Window> getClosedWindows()

getWindowsCreated

public long getWindowsCreated()

getWindowsClosed

public long getWindowsClosed()

open

public void open()

Description copied from class: DataEndpoint

Makes this endpoint ready for reading or writing.

Overrides:

open in class ProxyReader

close

public void close()
           throws DataException

Description copied from class: DataEndpoint

Indicates that this endpoint has finished reading or writing.

Overrides:

close in class ProxyReader

Throws:

DataException

readImpl

protected Record readImpl()
                   throws Throwable

Description copied from class: DataReader

Overridden by subclasses to read the next record from this DataReader. The default implementation of DataReader.read() now insures that this method will not be called again after it returns a null.

If no record is available, null will be returned.

Overrides:

readImpl in class ProxyReader

Throws:

Throwable

addExceptionProperties

public DataException addExceptionProperties(DataException exception)

Description copied from class: Endpoint

Adds this endpoint's current state to a DataException. Since this method is called whenever an exception is thrown, subclasses should override it to add their specific information.

Overrides:

addExceptionProperties in class ProxyReader

toString

public String toString()

Overrides:

toString in class DataEndpoint