Write my own Transformer

Updated: Aug 24, 2023

View on GitHub

This example offers a customizable Transformer component that enables you to modify the default data transformation logic. By utilizing this functionality, you can implement your own specialized data transformation rules tailored to specific data processing requirements and business logic.

This can be used in scenarios where the default data transformation provided by the library does not fully meet your needs. You may have unique data formats, complex data mappings, or specific data manipulation requirements that necessitate custom transformation rules. The ability to change the default Transformer allows you to achieve integration with your existing data infrastructure and ensure the data is processed in the desired format for further analysis.

 

Input CSV File

Account,LastName,FirstName,Balance,CreditLimit,AccountCreated,Rating
101,Reeves,Keanu,9315.45,10000.00,1/17/1998,A
312,Butler,Gerard,90.00,1000.00,8/6/2003,B
868,Hewitt,Jennifer Love,0,17000.00,5/25/1985,B
761,Pinkett-Smith,Jada,49654.87,100000.00,12/5/2006,A
317,Murray,Bill,789.65,5000.00,2/5/2007,C

 

Java Code Listing

WriteMyOwnTransformer.java

package com.northconcepts.datapipeline.examples.cookbook.customization;

import java.io.File;

import com.northconcepts.datapipeline.core.DataReader;
import com.northconcepts.datapipeline.core.StreamWriter;
import com.northconcepts.datapipeline.csv.CSVReader;
import com.northconcepts.datapipeline.job.Job;
import com.northconcepts.datapipeline.transform.TransformingReader;

public class WriteMyOwnTransformer {
    
    public static void main(String[] args) throws Throwable {
        DataReader reader = new CSVReader(new File("example/data/input/credit-balance-01.csv"))
            .setFieldNamesInFirstRow(true);
        
        reader = new TransformingReader(reader)
            .setExceptionOnFailure(false)
            .add(new MyTransformer());
        
        Job.run(reader, new StreamWriter(System.out));
    }
    
}

 

MyTransformer.java

package com.northconcepts.datapipeline.examples.cookbook.customization;

import com.northconcepts.datapipeline.core.Field;
import com.northconcepts.datapipeline.core.Record;
import com.northconcepts.datapipeline.transform.Transformer;

public class MyTransformer extends Transformer {

    public MyTransformer() {
    }

    public boolean transform(Record record) throws Throwable {
        Field creditLimit = record.getField("CreditLimit");
        // increase everyone's limit by 10%
        double newValue = Double.parseDouble(creditLimit.getValueAsString()) * 1.10;
        creditLimit.setValue(newValue);
        return true;
    }

}

 

Code Walkthrough

1. CSVReader is created corresponding to the input file credit-balance-01.csv.
2. The setFieldNamesInFirstRow(true) method is invoked to specify that the names specified in the first row should be used as field names.
3. TransformingReader instance is created to sequentially apply changes to the incoming data. 
4. MyTransformer which is a custom class to apply desired transformations is used to further process the data. Inside transform() method body, the CreditLimit value is increased by 10%, but this logic can be customized based on our needs.
5. Job.run() is used to transfer the data from reader to StreamWriter(System.out). See how to compile and run data pipeline jobs. 

 

Console Output

-----------------------------------------------
0 - Record (MODIFIED) {
    0:[Account]:STRING=[101]:String
    1:[LastName]:STRING=[Reeves]:String
    2:[FirstName]:STRING=[Keanu]:String
    3:[Balance]:STRING=[9315.45]:String
    4:[CreditLimit]:DOUBLE=[11000.0]:Double
    5:[AccountCreated]:STRING=[1/17/1998]:String
    6:[Rating]:STRING=[A]:String
}

-----------------------------------------------
1 - Record (MODIFIED) {
    0:[Account]:STRING=[312]:String
    1:[LastName]:STRING=[Butler]:String
    2:[FirstName]:STRING=[Gerard]:String
    3:[Balance]:STRING=[90.00]:String
    4:[CreditLimit]:DOUBLE=[1100.0]:Double
    5:[AccountCreated]:STRING=[8/6/2003]:String
    6:[Rating]:STRING=[B]:String
}

-----------------------------------------------
2 - Record (MODIFIED) {
    0:[Account]:STRING=[868]:String
    1:[LastName]:STRING=[Hewitt]:String
    2:[FirstName]:STRING=[Jennifer Love]:String
    3:[Balance]:STRING=[0]:String
    4:[CreditLimit]:DOUBLE=[18700.0]:Double
    5:[AccountCreated]:STRING=[5/25/1985]:String
    6:[Rating]:STRING=[B]:String
}

-----------------------------------------------
3 - Record (MODIFIED) {
    0:[Account]:STRING=[761]:String
    1:[LastName]:STRING=[Pinkett-Smith]:String
    2:[FirstName]:STRING=[Jada]:String
    3:[Balance]:STRING=[49654.87]:String
    4:[CreditLimit]:DOUBLE=[110000.00000000001]:Double
    5:[AccountCreated]:STRING=[12/5/2006]:String
    6:[Rating]:STRING=[A]:String
}

-----------------------------------------------
4 - Record (MODIFIED) {
    0:[Account]:STRING=[317]:String
    1:[LastName]:STRING=[Murray]:String
    2:[FirstName]:STRING=[Bill]:String
    3:[Balance]:STRING=[789.65]:String
    4:[CreditLimit]:DOUBLE=[5500.0]:Double
    5:[AccountCreated]:STRING=[2/5/2007]:String
    6:[Rating]:STRING=[C]:String
}

-----------------------------------------------
5 records