Map Data from Source to Target with Lookup

This example shows how you can use Data Pipeline to apply lookup operations when mapping data from a source to a target. It allows users to reference and retrieve additional data from external sources or reference tables based on specified keys or conditions, enhancing the mapping process with dynamic and contextual information.

Users can leverage the example to enrich mapped data by performing lookups to external sources such as databases, APIs, or reference tables. This enables the retrieval of additional information, such as customer details, product attributes, or financial data, and enhances the mapped data with more comprehensive and relevant insights.

Java Code Listing

package com.northconcepts.datapipeline.foundations.examples.datamapping;

import com.northconcepts.datapipeline.core.FieldList;
import com.northconcepts.datapipeline.core.Record;
import com.northconcepts.datapipeline.foundations.datamapping.DataMapping;
import com.northconcepts.datapipeline.foundations.datamapping.DataMappingResult;
import com.northconcepts.datapipeline.internal.expression.DefaultExpressionContext;
import com.northconcepts.datapipeline.transform.lookup.BasicLookup;
import com.northconcepts.datapipeline.transform.lookup.Lookup;

public class MapDataFromSourceToTargetWithLookup {

    public static void main(String[] args) {
        DefaultExpressionContext input = new DefaultExpressionContext();
        input.setValue("fname", "John");
        input.setValue("lname", "Smith");
        input.setValue("currency_code", "mxn");

        Lookup currencyLookup = new BasicLookup(new FieldList("Currency Name"))
                .add("CAD", "Canadian Dollar")
                .add("USD", "American Dollar")
                .add("EUR", "Euro")
                .add("GBP", "British Pound")
                .add("MXN", "Mexican Peso")
                ;
        
        DataMapping mapping = new DataMapping()
                .setValue("currencyLookup", currencyLookup)
                .addFieldMapping("first_name", "source.fname")
                .addFieldMapping("currency", "lookup(0, currencyLookup, toUpperCase(source.currency_code))")
                ;
        
        DataMappingResult result = mapping.map(input);
        Record target = result.getTarget();
        System.out.println("target = " + target);
        
        result.logExceptions();

    }

}

Code Walkthrough

1. DefaultExpressionContext instance is used to hold the source/input data with the key-value pattern. Values are entered with setValue() method.
2. BasicLookup is created to specify additional information about currency codes. A new field Currency Name is added with five rows.
3. DataMapping is created to enable the field mapping rules. 
4. The lookup declared in step 2 is initialized as a variable called "currencyLookup" in order to be used in the source expression.
5. addFieldMapping() method accepts two String arguments:
   1. targetFieldName - the name of the field in the target record.
   2. sourceExpression - a custom expression to hold the formatting logic for a target field.
6. Fields in the source and target records can be accessed with the dot (.) operator: source.fname.
7. DataMappingResult instance is created to apply mapping for the input.
8. The target record is obtained from result object and printed in the console with System.out.println().
9. The exceptions from the result, if exist, are also logged.

Console output

target = Record (MODIFIED) {
    0:[first_name]:STRING=[John]:String
    1:[currency]:STRING=[Mexican Peso]:String
}