Map Data Using Automatic Mapping And Exclude Fields

This example shows you how to automatically map data from source-to-target using Data Pipeline.

This demo code uses DataMappingReader to read records from an input CSV file.

If there are many fields in the source, then rather than defining all fields in DataMapping, use the autoMapping property to automatically map all the incoming source fields to their target.

When this flag is set, your custom mappings can use the automatically mapped fields in their source expressions.

DataMapping now also allows you to add a set of excluded fields to be removed after your custom mapping is performed, but before the target data is returned. With this you can rename a field by: 1) mapping fields automatically with the new flag, 2) rename a field by adding a new explicit field mapping, and 3) remove the old automatically mapped field by adding it to the excluded fields.

Java Code listing

package com.northconcepts.datapipeline.foundations.examples.datamapping;

import java.io.File;

import com.northconcepts.datapipeline.core.DataReader;
import com.northconcepts.datapipeline.core.StreamWriter;
import com.northconcepts.datapipeline.csv.CSVReader;
import com.northconcepts.datapipeline.foundations.datamapping.DataMapping;
import com.northconcepts.datapipeline.foundations.datamapping.DataMappingReader;
import com.northconcepts.datapipeline.job.Job;

public class MapDataUsingAutoMappingAndExcludedFields {

    public static void main(String[] args) {
        DataMapping mapping = new DataMapping()
                .setAutoMapping(true) // enable auto fields mapping
                .addFieldMapping("Full_Name", "source.LastName+ ' ' + source.FirstName")
                .addExcludedFields("AccountCreated", "Rating");
        
        DataReader reader = new CSVReader(new File("example/data/input/bank_account.csv"))
                .setAllowMultiLineText(true)
                .setFieldNamesInFirstRow(true);

        reader = new DataMappingReader(reader, mapping);

        Job.run(reader, new StreamWriter(System.out));
    }

}

Code Walkthrough

1. DataMapping is used to enable auto mapping of fields from source to target.
2. Set the autoMapping flag on DataMapping to enable auto mapping.
3. Add custom fields to DataMapping.
4. Exclude some fields after auto mapping and custom mapping performed.
5. Run the job.

Console output

19:30:20,443 DEBUG [main] datapipeline:37 - DataPipeline v8.1.0-SNAPSHOT by North Concepts Inc.
19:30:20,452 DEBUG [main] datapipeline:615 - Job[1,job-1,Sat Feb 18 19:30:20 IST 2023]::Start
-----------------------------------------------
0 - Record (MODIFIED) {
    0:[Id]:STRING=[1]:String
    1:[AccountNo]:STRING=[101]:String
    2:[LastName]:STRING=[Reeves]:String
    3:[FirstName]:STRING=[Keanu]:String
    4:[Balance]:STRING=[12]:String
    5:[CreditLimit]:STRING=[1000.23]:String
    6:[Full_Name]:STRING=[Reeves Keanu]:String
}

-----------------------------------------------
1 - Record (MODIFIED) {
    0:[Id]:STRING=[2]:String
    1:[AccountNo]:STRING=[102]:String
    2:[LastName]:STRING=[Butler]:String
    3:[FirstName]:STRING=[Gerard]:String
    4:[Balance]:STRING=[123456]:String
    5:[CreditLimit]:STRING=[1234567890.98]:String
    6:[Full_Name]:STRING=[Butler Gerard]:String
}

-----------------------------------------------
2 - Record (MODIFIED) {
    0:[Id]:STRING=[3]:String
    1:[AccountNo]:STRING=[103]:String
    2:[LastName]:STRING=[Hewitt]:String
    3:[FirstName]:STRING=[Jennifer]:String
    4:[Balance]:STRING=[4294967295]:String
    5:[CreditLimit]:STRING=[9876543210.776655]:String
    6:[Full_Name]:STRING=[Hewitt Jennifer]:String
}

-----------------------------------------------
3 - Record (MODIFIED) {
    0:[Id]:STRING=[4]:String
    1:[AccountNo]:STRING=[104]:String
    2:[LastName]:STRING=[Pinkett-Smith]:String
    3:[FirstName]:STRING=[Jada]:String
    4:[Balance]:STRING=[184467440737095]:String
    5:[CreditLimit]:STRING=[-1.23457E+18]:String
    6:[Full_Name]:STRING=[Pinkett-Smith Jada]:String
}

-----------------------------------------------
4 - Record (MODIFIED) {
    0:[Id]:STRING=[5]:String
    1:[AccountNo]:STRING=[105]:String
    2:[LastName]:STRING=[Murray]:String
    3:[FirstName]:STRING=[Bill]:String
    4:[Balance]:STRING=[9223372036854]:String
    5:[CreditLimit]:STRING=[-112233.99]:String
    6:[Full_Name]:STRING=[Murray Bill]:String
}

-----------------------------------------------
5 - Record (MODIFIED) {
    0:[Id]:STRING=[6]:String
    1:[AccountNo]:STRING=[106]:String
    2:[LastName]:STRING=[Murray]:String
    3:[FirstName]:STRING=[Bill]:String
    4:[Balance]:STRING=[1]:String
    5:[CreditLimit]:STRING=[-778899.12345]:String
    6:[Full_Name]:STRING=[Murray Bill]:String
}

-----------------------------------------------
6 - Record (MODIFIED) {
    0:[Id]:STRING=[7]:String
    1:[AccountNo]:STRING=[107]:String
    2:[LastName]:STRING=[John]:String
    3:[FirstName]:STRING=[Doe]:String
    4:[Balance]:STRING=[9223372036854]:String
    5:[CreditLimit]:STRING=[900000.98765]:String
    6:[Full_Name]:STRING=[John Doe]:String
}

-----------------------------------------------
7 - Record (MODIFIED) {
    0:[Id]:STRING=[8]:String
    1:[AccountNo]:STRING=[108]:String
    2:[LastName]:STRING=[Jane]:String
    3:[FirstName]:STRING=[Doe]:String
    4:[Balance]:STRING=[789456123123456]:String
    5:[CreditLimit]:STRING=[8000000000.887765]:String
    6:[Full_Name]:STRING=[Jane Doe]:String
}

-----------------------------------------------
8 - Record (MODIFIED) {
    0:[Id]:STRING=[9]:String
    1:[AccountNo]:STRING=[109]:String
    2:[LastName]:STRING=[James]:String
    3:[FirstName]:STRING=[Bond]:String
    4:[Balance]:STRING=[7894561230123]:String
    5:[CreditLimit]:STRING=[10000.99887]:String
    6:[Full_Name]:STRING=[James Bond]:String
}

-----------------------------------------------
9 records
19:30:20,468 DEBUG [main] datapipeline:661 - job::Success