Add my own Functions

This example shows how to add your own functions to the DataPipeline Expression Language (DPEL).  You can use this to extend the library's capabilities with your own logic. This can be used when specific data manipulation tasks or calculations are required, allowing you to enhance the DataPipeline's functionality to match your project's requirements precisely.

Input CSV File

Account,LastName,FirstName,Balance,CreditLimit,AccountCreated,Rating
101,Reeves,Keanu,9315.45,10000.00,1/17/1998,A
312,Butler,Gerard,90.00,1000.00,8/6/2003,B
868,Hewitt,Jennifer Love,0,17000.00,5/25/1985,B
761,Pinkett-Smith,Jada,49654.87,100000.00,12/5/2006,A
317,Murray,Bill,789.65,5000.00,2/5/2007,C

Java Code Listing

package com.northconcepts.datapipeline.examples.cookbook.customization;

import java.io.File;

import com.northconcepts.datapipeline.core.DataReader;
import com.northconcepts.datapipeline.core.Functions;
import com.northconcepts.datapipeline.core.StreamWriter;
import com.northconcepts.datapipeline.csv.CSVReader;
import com.northconcepts.datapipeline.job.Job;
import com.northconcepts.datapipeline.transform.SetCalculatedField;
import com.northconcepts.datapipeline.transform.TransformingReader;

public class AddMyOwnFunctions {
    
    // functions are public, static methods
    public static String initials(String firstName, String lastName) {
        return firstName.substring(0, 1).toUpperCase() +
            " " +
            lastName.substring(0, 1).toUpperCase();
    }
    
    public static void main(String[] args) throws Throwable {
        // register new function 'initials'
        Functions.add("initials", "com.northconcepts.datapipeline.examples.cookbook.customization.AddMyOwnFunctions.initials");
        
        DataReader reader = new CSVReader(new File("example/data/input/credit-balance-01.csv"))
            .setFieldNamesInFirstRow(true);
        
        TransformingReader transformingReader = new TransformingReader(reader);
        
        // use the new function in a SetCalculatedField or FilterExpression 
        transformingReader.add(new SetCalculatedField("name", 
                "initials(FirstName, LastName)"));
        
        Job.run(transformingReader, new StreamWriter(System.out));
    }
    
}

Code Walkthrough

1. A custom function is introduced. It should be public and static. In the given example, as the method name shows, the initials from the first name and last name are returned as the result of the method.
2. In the main method, the above method is registered with its alias and signature (i.e. path to the method).
3. CSVReader is created corresponding to the input file credit-balance-02.csv.
4. The setFieldNamesInFirstRow(true) method is invoked to specify that the names specified in the first row should be used as field names.
5. TransformingReader is created to apply changes to records from the source.
6. initials method is used for the new field name inside SetCalculatedField instance.
7. Job.run() is used to transfer the data from transformingReader to StreamWriter(System.out). See how to compile and run data pipeline jobs. 

Console Output

-----------------------------------------------
0 - Record (MODIFIED) {
    0:[Account]:STRING=[101]:String
    1:[LastName]:STRING=[Reeves]:String
    2:[FirstName]:STRING=[Keanu]:String
    3:[Balance]:STRING=[9315.45]:String
    4:[CreditLimit]:STRING=[10000.00]:String
    5:[AccountCreated]:STRING=[1/17/1998]:String
    6:[Rating]:STRING=[A]:String
    7:[name]:STRING=[K R]:String
}

-----------------------------------------------
1 - Record (MODIFIED) {
    0:[Account]:STRING=[312]:String
    1:[LastName]:STRING=[Butler]:String
    2:[FirstName]:STRING=[Gerard]:String
    3:[Balance]:STRING=[90.00]:String
    4:[CreditLimit]:STRING=[1000.00]:String
    5:[AccountCreated]:STRING=[8/6/2003]:String
    6:[Rating]:STRING=[B]:String
    7:[name]:STRING=[G B]:String
}

-----------------------------------------------
2 - Record (MODIFIED) {
    0:[Account]:STRING=[868]:String
    1:[LastName]:STRING=[Hewitt]:String
    2:[FirstName]:STRING=[Jennifer Love]:String
    3:[Balance]:STRING=[0]:String
    4:[CreditLimit]:STRING=[17000.00]:String
    5:[AccountCreated]:STRING=[5/25/1985]:String
    6:[Rating]:STRING=[B]:String
    7:[name]:STRING=[J H]:String
}

-----------------------------------------------
3 - Record (MODIFIED) {
    0:[Account]:STRING=[761]:String
    1:[LastName]:STRING=[Pinkett-Smith]:String
    2:[FirstName]:STRING=[Jada]:String
    3:[Balance]:STRING=[49654.87]:String
    4:[CreditLimit]:STRING=[100000.00]:String
    5:[AccountCreated]:STRING=[12/5/2006]:String
    6:[Rating]:STRING=[A]:String
    7:[name]:STRING=[J P]:String
}

-----------------------------------------------
4 - Record (MODIFIED) {
    0:[Account]:STRING=[317]:String
    1:[LastName]:STRING=[Murray]:String
    2:[FirstName]:STRING=[Bill]:String
    3:[Balance]:STRING=[789.65]:String
    4:[CreditLimit]:STRING=[5000.00]:String
    5:[AccountCreated]:STRING=[2/5/2007]:String
    6:[Rating]:STRING=[C]:String
    7:[name]:STRING=[B M]:String
}

-----------------------------------------------
5 records