Conditionally map Data from Source to Target

This example shows how you can use Data Pipeline to perform data mapping operations from source to target with some conditions. It allows users to define conditional rules that determine how data should be mapped based on specific criteria, enabling flexible and dynamic data transformation.

The example can be used to selectively map data based on certain conditions, allowing users to filter and segment data during the mapping process. This can be useful for creating subsets of data based on specific criteria, such as mapping only sales data for a particular region or time period.

Java Code Listing

package com.northconcepts.datapipeline.foundations.examples.datamapping;

import com.northconcepts.datapipeline.core.Record;
import com.northconcepts.datapipeline.foundations.datamapping.DataMapping;
import com.northconcepts.datapipeline.foundations.datamapping.DataMappingResult;
import com.northconcepts.datapipeline.internal.expression.DefaultExpressionContext;

public class ConditionallyMapDataFromSourceToTarget {

    public static void main(String[] args) {
        DefaultExpressionContext input = new DefaultExpressionContext();
        input.setValue("fname", "John");
        input.setValue("lname", "Smith");

        DataMapping mapping = new DataMapping()
                .addCondition("length(source.fname) > 3") // don't map anything if fname is not greater than 3 characters
                .addFieldMapping("first_name", "source.fname")
                .addFieldMapping("length", "length(source.fname)");
        
        DataMappingResult result = mapping.map(input);
        Record target = result.getTarget();

        System.out.println("target = " + target);
        
        result.logExceptions();
    }

}

Code Walkthrough

1. DefaultExpressionContext instance is used to hold the source/input data with the key-value pattern. Values are entered with setValue() method.
2. DataMapping is created to enable the field mapping rules. 
3. Checking for a condition is applied with addCondition() method. The expression in the example checks whether the length of source.fname field is greater than 3. Only if this condition is met, the next steps are implemented.
4. addFieldMapping() method accepts two String arguments:
   1. targetFieldName - the name of the field in the target record.
   2. sourceExpression - a custom expression to hold the formatting logic for a target field.
5. Fields in the source and target records can be accessed with the dot (.) operator: source.fname.
6. Different methods for formatting strings can be used in sourceExpression. In the given example, length() is called.
7. DataMappingResult instance is created to apply mapping for the input.
8. The target record is obtained from result object and printed in the console with System.out.println().
9. The exceptions from the result, if exist, are also logged.

Console output

target = Record (MODIFIED) {
    0:[first_name]:STRING=[John]:String
    1:[length]:LONG=[4]:Long
}