Transformations

Transformation Description Docs Examples

Aggregate

Perform calculations (count, max, etc.) using all records.

Assign Field

Create or update fields with specific values.

Async

Read or write using additional threads.

Calculated Field

Assign field values using the built-in expression language or programmatically.

Copy Field

Duplicate fields within records.

Data Mapping

Transform data from source to target using the expression language and functions.

Decision Table

Replace complicated conditional logic with simple matrices.

Decision Tree

Express complicated business logic as simple flow charts.

Declarative Pipelines

Create jobs declaratively with inputs, outputs, actions, and datasets.

DeMux

Split or duplicate a data source using one of the defined policies.

Detailed exception reporting

Get vital state info along with the stack trace when exceptions occur.

Expression language

Use the runtime expression language to save on code changes.

Exclude Fields

Remove fields from records using a black-list approach.

Filter

Select records using the built-in expression language or programmatic rules.

JDBC Batch updates

Chunk database writes to improve performance.

JDBC Multi-Writer

Write to a single database using multiple connections.

Job management

Use automatic reader-writer transfer or create hooks for greater control and visibility.

Lookups/Joins

Enrich data with information from secondary sources.

Meter

Record stats about your data transfer.

Multi-Writer

Write to multiple targets simultaneously.

Out-of-band data

Attach temporary, transient data to any field or record

Remove Duplicates (Dedup)

Delete records where one or more fields contain repeating values.

Rename Field

Change field names to match targets.

Retrying

Retry read or write when a failure occurs.

Schema Transformer

Validate and convert records using a target schema.

Select/Include Fields

Select and arrange fields in records using a white-list approach.

Sequence

Combine multiple data sources into one serially.

Sort

Sort huge datasets with the external, disk-based sorting. Smaller sets can be sorted in-memory.

SplitWriter

Converts a single source DataReader into many downstream sources using one of the provided strategies.

Streaming data

Start writing data as soon as the first record is read.

Throttle

Limit your data transfer to a specified number of bytes or records per second.

Transform

Manipulate data using predefined or user-defined transformations.

Validate

Ensure records match your criteria by using the built-in expression language or programmatic rules.

Mobile Analytics