Automatically Map Data And Exclude Fields Declaratively

This example shows you how to automatically map data from source-to-target using Data Pipeline. It is also similar to Map Data Using Automatic Mapping And Exclude Fields the only difference being this example loads the DataMapping from XML rather than using java code.

To find similar examples see the Declarative Examples.

This demo code uses DataMappingReader to read records from an input CSV file.

If there are many fields in the source, then rather than defining all fields in DataMapping, use the autoMapping property to automatically map all the incoming source fields to their target. i.e <data-mapping autoMapping="true">.

When this flag is set, your custom mappings can use the automatically mapped fields in their source expressions.

DataMapping now also allows you to add a set of excluded fields to be removed after your custom mapping is performed, but before the target data is returned. With this you can rename a field by: 1) mapping fields automatically with the new flag, 2) rename a field by adding a new explicit field mapping, and 3) remove the old automatically mapped field by adding it to the excluded fields.

DataMapping XML file

<data-mapping autoMapping="true">
  <field-mappings>
    <field-mapping fieldName="Full_Name" sourceExpression="source.LastName + ' ' + source.FirstName"/>
  </field-mappings>
  <excluded-fields>
    <excluded-field>AccountCreated</excluded-field>
    <excluded-field>Rating</excluded-field>
  </excluded-fields>
</data-mapping>

Input CSV file

Id,AccountNo,LastName,FirstName,Balance,CreditLimit,AccountCreated,Rating
1,101,Reeves,Keanu,12,1000.23,17-01-1998,A
2,102,Butler,Gerard,123456,1234567890.98,06-08-2003,B
3,103,Hewitt,Jennifer,4294967295,9876543210.776655,25-05-1985,B
4,104,Pinkett-Smith,Jada,184467440737095,-1.23457E+18,05-12-2006,A
5,105,Murray,Bill,9223372036854,-112233.99,05-02-2007,C
6,106,Murray,Bill,1,-778899.12345,05-02-2007,D
7,107,John,Doe,9223372036854,900000.98765,25-05-1985,B
8,108,Jane,Doe,789456123123456,8000000000.887765,17-01-1998,B
9,109,James,Bond,7894561230123,10000.99887,05-12-2006,A

Java Code listing

package com.northconcepts.datapipeline.foundations.examples.datamapping;

import com.northconcepts.datapipeline.core.DataReader;
import com.northconcepts.datapipeline.core.StreamWriter;
import com.northconcepts.datapipeline.csv.CSVReader;
import com.northconcepts.datapipeline.foundations.datamapping.DataMapping;
import com.northconcepts.datapipeline.foundations.datamapping.DataMappingReader;
import com.northconcepts.datapipeline.job.Job;

import java.io.File;
import java.io.FileInputStream;

public class AutomaticallyMapDataAndExcludeFieldsDeclaratively {

    public static void main(String[] args) throws Throwable {
    
        //Load DataMapping from this XML File.
        DataMapping mappingFromXml = new DataMapping()
                .fromXml(new FileInputStream("example/data/input/datamapping/customer-details-data-mapping.xml"));

        DataReader reader = new CSVReader(new File("example/data/input/bank_account.csv"))
                .setAllowMultiLineText(true)
                .setFieldNamesInFirstRow(true);

        reader = new DataMappingReader(reader, mappingFromXml);

        Job.run(reader, new StreamWriter(System.out));
    }

}

Code Walkthrough

1. DataMapping is used to enable auto mapping of fields from source to target.
2. <data-mapping autoMapping="true"> sets the autoMapping flag on DataMapping to enable auto mapping.
3. <field-mapping ..."/> adds custom fields to DataMapping.
4. <excluded-field>fieldName</excluded-field> excludes the specified fields after auto mapping and custom mapping performed.
5. Job.run() is used to run the job while StreamWriter prints the output to the console.

Console output

-----------------------------------------------
0 - Record (MODIFIED) {
    0:[Id]:STRING=[1]:String
    1:[AccountNo]:STRING=[101]:String
    2:[LastName]:STRING=[Reeves]:String
    3:[FirstName]:STRING=[Keanu]:String
    4:[Balance]:STRING=[12]:String
    5:[CreditLimit]:STRING=[1000.23]:String
    6:[Full_Name]:STRING=[Reeves Keanu]:String
}

-----------------------------------------------
1 - Record (MODIFIED) {
    0:[Id]:STRING=[2]:String
    1:[AccountNo]:STRING=[102]:String
    2:[LastName]:STRING=[Butler]:String
    3:[FirstName]:STRING=[Gerard]:String
    4:[Balance]:STRING=[123456]:String
    5:[CreditLimit]:STRING=[1234567890.98]:String
    6:[Full_Name]:STRING=[Butler Gerard]:String
}

-----------------------------------------------
2 - Record (MODIFIED) {
    0:[Id]:STRING=[3]:String
    1:[AccountNo]:STRING=[103]:String
    2:[LastName]:STRING=[Hewitt]:String
    3:[FirstName]:STRING=[Jennifer]:String
    4:[Balance]:STRING=[4294967295]:String
    5:[CreditLimit]:STRING=[9876543210.776655]:String
    6:[Full_Name]:STRING=[Hewitt Jennifer]:String
}

-----------------------------------------------
3 - Record (MODIFIED) {
    0:[Id]:STRING=[4]:String
    1:[AccountNo]:STRING=[104]:String
    2:[LastName]:STRING=[Pinkett-Smith]:String
    3:[FirstName]:STRING=[Jada]:String
    4:[Balance]:STRING=[184467440737095]:String
    5:[CreditLimit]:STRING=[-1.23457E+18]:String
    6:[Full_Name]:STRING=[Pinkett-Smith Jada]:String
}

-----------------------------------------------
4 - Record (MODIFIED) {
    0:[Id]:STRING=[5]:String
    1:[AccountNo]:STRING=[105]:String
    2:[LastName]:STRING=[Murray]:String
    3:[FirstName]:STRING=[Bill]:String
    4:[Balance]:STRING=[9223372036854]:String
    5:[CreditLimit]:STRING=[-112233.99]:String
    6:[Full_Name]:STRING=[Murray Bill]:String
}

-----------------------------------------------
5 - Record (MODIFIED) {
    0:[Id]:STRING=[6]:String
    1:[AccountNo]:STRING=[106]:String
    2:[LastName]:STRING=[Murray]:String
    3:[FirstName]:STRING=[Bill]:String
    4:[Balance]:STRING=[1]:String
    5:[CreditLimit]:STRING=[-778899.12345]:String
    6:[Full_Name]:STRING=[Murray Bill]:String
}

-----------------------------------------------
6 - Record (MODIFIED) {
    0:[Id]:STRING=[7]:String
    1:[AccountNo]:STRING=[107]:String
    2:[LastName]:STRING=[John]:String
    3:[FirstName]:STRING=[Doe]:String
    4:[Balance]:STRING=[9223372036854]:String
    5:[CreditLimit]:STRING=[900000.98765]:String
    6:[Full_Name]:STRING=[John Doe]:String
}

-----------------------------------------------
7 - Record (MODIFIED) {
    0:[Id]:STRING=[8]:String
    1:[AccountNo]:STRING=[108]:String
    2:[LastName]:STRING=[Jane]:String
    3:[FirstName]:STRING=[Doe]:String
    4:[Balance]:STRING=[789456123123456]:String
    5:[CreditLimit]:STRING=[8000000000.887765]:String
    6:[Full_Name]:STRING=[Jane Doe]:String
}

-----------------------------------------------
8 - Record (MODIFIED) {
    0:[Id]:STRING=[9]:String
    1:[AccountNo]:STRING=[109]:String
    2:[LastName]:STRING=[James]:String
    3:[FirstName]:STRING=[Bond]:String
    4:[Balance]:STRING=[7894561230123]:String
    5:[CreditLimit]:STRING=[10000.99887]:String
    6:[Full_Name]:STRING=[James Bond]:String
}

-----------------------------------------------
9 records
11:11:47,310 DEBUG [main] datapipeline:661 - job::Success