1. Home
  2. Documentation
  3. Transformations

Transformations

Transformation Description Docs Examples

Aggregate

 Perform calculations (count, max, etc.) using all records. Aggregate Javadocs
Data Aggregation User Guide 		Aggregate Examples

Assign Field

 Create or update fields with specific values. SetField Javadocs Assign Field Examples

Async

 Read or write using additional threads. AsyncReader Javadocs
AsyncWriter Javadocs
Multi-threaded Processing User Guide 		Threading Examples

Calculated Field

 Assign field values using the built-in expression language or programmatically. SetCalculatedField Javadocs Calculate Field Examples

Copy Field

 Duplicate fields within records. CopyField Javadocs Copy a Field Examples

Data Mapping

 Transform data from source to target using the expression language and functions. Data Mapping Javadocs
Data Mapping User Guide
Data mapping demo tool 		Data Mapping Examples

Decision Table

 Replace complicated conditional logic with simple matrices. Decision Table Javadocs
Decision Table User Guide 		Decision Table Examples

Decision Tree

 Express complicated business logic as simple flow charts. Decision Tree Javadocs
Decision Tree User Guide 		Decision Tree Examples

Declarative Pipelines

 Create jobs declaratively with inputs, outputs, actions, and datasets. Declarative Pipelines Javadocs
Declarative Pipeline User Guide
Declarative Pipelines Demo Tool 		Declarative Pipelines Examples

DeMux

 Split or duplicate a data source using one of the defined policies. DeMux Javadocs
Reading One-to-Many User Guide 		Demux Examples

Detailed exception reporting

 Get vital state info along with the stack trace when exceptions occur. DataException Javadocs Exceptions Examples

Expression language

 Use the runtime expression language to save on code changes. Expression Language User Guide EL Examples

Exclude Fields

 Remove fields from records using a black-list approach. RemoveFields Javadocs Exclude Fields Examples

Filter

 Select records using the built-in expression language or programmatic rules. Filter Javadocs Filter Examples

JDBC Batch updates

 Chunk database writes to improve performance. JDBC Javadocs JDBC Examples

JDBC Multi-Writer

 Write to a single database using multiple connections. JdbcMultiWriter Javadocs JDBC Multi-Writer Examples

Job management

 Use automatic reader-writer transfer or create hooks for greater control and visibility. Job Javadocs
Managing Running Jobs User Guide 		Job Management Examples

Lookups/Joins

 Enrich data with information from secondary sources. Lookup Javadocs Lookups/Joins Examples

Meter

 Record stats about your data transfer. Meter Javadocs Meter Examples

Multi-Writer

 Write to multiple targets simultaneously. MultiWriter Javadocs
Writing One-to-Many User Guide 		Multi-Writer Examples

Out-of-band data

 Attach temporary, transient data to any field or record Node Javadocs Out-of-band Data Examples

Remove Duplicates (Dedup)

 Delete records where one or more fields contain repeating values. RemoveDuplicatesReader Javadocs Remove Duplicates (Dedup) Examples

Rename Field

 Change field names to match targets. RenameField Javadocs Rename Field Examples

Retrying

 Retry read or write when a failure occurs. RetryingReader/Writer Javadocs Retrying Examples

Schema Transformer

 Validate and convert records using a target schema. SchemaTransformer Javadocs SchemaTransformer Examples

Select/Include Fields

 Select and arrange fields in records using a white-list approach. SelectFields Javadocs Select/Include Fields Examples

Sequence

 Combine multiple data sources into one serially. AsyncMultiReader Javadocs
Reading Many-to-One User Guide 		AsynMultiReader Examples

Sort

 Sort huge datasets with the external, disk-based sorting. Smaller sets can be sorted in-memory. SortingReader Javadocs Sort Examples

SplitWriter

 Converts a single source DataReader into many downstream sources using one of the provided strategies. SplitWriter Javadocs SplitWriter Examples

Streaming data

 Start writing data as soon as the first record is read. TwitterFilterStreamReader V1 Javadocs
TwitterFilterStreamReader V2 Javadocs 		Streaming Data Examples

Throttle

 Limit your data transfer to a specified number of bytes or records per second. Throttle Javadocs Throttle Examples

Transform

 Manipulate data using predefined or user-defined transformations. Transform Javadocs Transform Examples

Validate

 Ensure records match your criteria by using the built-in expression language or programmatic rules. Validate Javadocs Validate Examples

Mobile Analytics