Data Mapping

Overview

Data mapping is a quick way to transform data from source to target using expressions and functions. It's a high-level tool that can fulfill most mapping needs without relying on the lower level readers and writers in DataPipeline core. The online data mapping demo tool uses this feature behind the scenes.

Simple Data Mapping

This example shows how to take two simple source fields (fname and lname) and map them to four new target fields (first_name, last_name, name, name_length) using the DataPipeline Expression Language.

Source Fields

The source/input fields are always referenced as source.XXX in the mapping expressions.

addFieldMapping("first_name", "source.fname")

Target Fields

Target fields are the first argument to the field mapping function. Once a mapping is added, its target field can then be referenced in the source part of subsequent mappings using target.XXX.

addFieldMapping("name", "source.fname + ' ' + target.last_name")

Resulting Records

The result of a mapping is a DataMappingResult containing a target Record with the target fields (without the "target." prefix).

Example: MapDataFromSourceToTarget.java

Data Mapping in a DataPipeline Job

While data mappings can be used as a standalone feature in your applications, it can also be plugged into an existing DataPipeline job to perform the mapping part of the work flow. 

This example reads a CSV file (jewelry.csv), uses a DataMappingReader to apply the source-target field data mapping, and writes to System.out.

Since one of the source fields contains a space, the field variable is enclosed in ${}.

addFieldMapping(new FieldMapping("Cost", "${source.Variant Price}"))

Example: MapDataFromSourceToTargetInAPipeline.java

Conditional Mapping

Sometimes a mapping should only be performed if a condition is met. This example shows how to apply a data mapping when a condition specified using the using the DataPipeline Expression Language is true.

Example: ConditionallyMapDataFromSourceToTarget.java

Conditional Field Mapping

Like overall conditional mappings, individual field mappings can also be limited to cases where a condition is met. This example shows how to apply a condition to a single field's mapping.

Example: ConditionallyMapDataFieldFromSourceToTarget.java

Mapping with Lookup

Data mappings can use the same lookups from the core DataPipeline framework as part of any field mapping or conditional expression. This example uses a lookup to map currency symbols to their common display names.

It uses the built-in lookup(int fieldIndex, Lookup lookupSource, Object ... lookupKeys) expression language function that takes a field index, but can also use the overloaded lookup(String fieldName, Lookup lookupSource, Object ... lookupKeys) version that takes a field name.

Example: MapDataFromSourceToTargetWithLookup.java

Mapping with Schema-Based Validation

Data mapping provides a way to validate the output of a mapping against a schema. This includes structural validation as well as rule-based validation. You can read about schema-based validations for more details. This example defines a schema with a customer entity and validates the data mapping target data against it.

Example: MapDataWithSchemaBasedValidation.java

Mapping with Rule-Based Validation

Data mapping's schema-based validation can also include arbitrary validation rules. Validation rules can be used on their own or combined with structural validations. You can read more about rule-based validation. This example adds an expression and field count validation and applies them to the data mapping's target data.

Example: MapDataWithRuleBasedValidation.java

Data Mapping in a DataPipeline Job with Validation

This is a complete example combining field and rule-based validations with a data mapping in a DataPipeline job.

Example: MapDataFromSourceToTargetInAPipelineWithValidation.java

Save a Data Mapping to/from XML

This example saves a programmatically built data mapping to XML and loads it back. This allows you to store data mappings as program resources on disk or in a database. It also allows you to modify them programmatically and save them back to disk.

Example: SaveAndLoadDataMappingToXML.java

The generated XML for the above data mapping would be the following.

Expression Language Built-In Variables