Write my own Lookup

This example shows how to create a customizable Lookup component, allowing you to modify the default behavior of looking up data. By leveraging this feature, you can implement your own specialized data-lookup logic tailored to specific use cases and requirements.

Lookup searches for data in another source to merge with each record passing through this lookup's transformer -- the streaming version of a join.

Users can leverage the example to enrich mapped data by performing lookups to external sources such as databases, APIs, or reference tables. This enables the retrieval of additional information, such as customer details, product attributes, or financial data, and enhances the mapped data with more comprehensive and relevant insights.

Input CSV File

Account,LastName,FirstName,Balance,CreditLimit,AccountCreated,Rating
101,Reeves,Keanu,9315.45,10000.00,1/17/1998,A
312,Butler,Gerard,90.00,1000.00,8/6/2003,B
868,Hewitt,Jennifer Love,0,17000.00,5/25/1985,B
761,Pinkett-Smith,Jada,49654.87,100000.00,12/5/2006,A
317,Murray,Bill,789.65,5000.00,2/5/2007,C

Java Code Listing

WriteMyOwnLookup.java

package com.northconcepts.datapipeline.examples.cookbook.customization;

import java.io.File;

import com.northconcepts.datapipeline.core.DataReader;
import com.northconcepts.datapipeline.core.FieldList;
import com.northconcepts.datapipeline.core.StreamWriter;
import com.northconcepts.datapipeline.csv.CSVReader;
import com.northconcepts.datapipeline.job.Job;
import com.northconcepts.datapipeline.transform.TransformingReader;
import com.northconcepts.datapipeline.transform.lookup.LookupTransformer;

public class WriteMyOwnLookup {
    
    public static void main(String[] args) throws Throwable {
        DataReader reader = new CSVReader(new File("example/data/input/credit-balance-01.csv"))
            .setFieldNamesInFirstRow(true);
        
        MyLookup lookup = new MyLookup();
        
        reader = new TransformingReader(reader)
            .add(new LookupTransformer(new FieldList().add("Rating"), lookup));
        
        Job.run(reader, new StreamWriter(System.out));
    }
    
}

MyLookup.java

package com.northconcepts.datapipeline.examples.cookbook.customization;

import com.northconcepts.datapipeline.core.Field;
import com.northconcepts.datapipeline.core.Record;
import com.northconcepts.datapipeline.core.RecordList;
import com.northconcepts.datapipeline.transform.lookup.Lookup;

public class MyLookup extends Lookup {
	
    public RecordList get(Object ... keys) {
        if (keys == null || keys.length != 1) {
            return null;
        }
        
        Record record = new Record();
        Field ratingClass = record.addField().setName("ratingClass");
        
        char rating = keys[0].toString().charAt(0);
        
        switch (rating) {
        case 'A': ratingClass.setValue("Class A"); break;           
        case 'B': ratingClass.setValue("Class B"); break;           
        case 'C': ratingClass.setValue("Class C"); break;           
        default: return null;           
        }
        
        RecordList list = new RecordList();
        list.add(record);
        return list;
    }

}

Code Walkthrough

1. CSVReader is created corresponding to the input file credit-balance-01.csv.
2. The setFieldNamesInFirstRow(true) method is invoked to specify that the names specified in the first row should be used as field names.
3. MyLookup instance is created to apply custom lookup logic.
4. TransformingReader is created to apply transformations to records from the source.
5. Custom lookup object is called to look up the "Rating" field of the dataset.
6. Job.run() is used to transfer the data from reader to StreamWriter(System.out). See how to compile and run data pipeline jobs. 

Console Output

-----------------------------------------------
0 - Record {
    0:[Account]:STRING=[101]:String
    1:[LastName]:STRING=[Reeves]:String
    2:[FirstName]:STRING=[Keanu]:String
    3:[Balance]:STRING=[9315.45]:String
    4:[CreditLimit]:STRING=[10000.00]:String
    5:[AccountCreated]:STRING=[1/17/1998]:String
    6:[Rating]:STRING=[A]:String
    7:[ratingClass]:STRING=[Class A]:String
}

-----------------------------------------------
1 - Record {
    0:[Account]:STRING=[312]:String
    1:[LastName]:STRING=[Butler]:String
    2:[FirstName]:STRING=[Gerard]:String
    3:[Balance]:STRING=[90.00]:String
    4:[CreditLimit]:STRING=[1000.00]:String
    5:[AccountCreated]:STRING=[8/6/2003]:String
    6:[Rating]:STRING=[B]:String
    7:[ratingClass]:STRING=[Class B]:String
}

-----------------------------------------------
2 - Record {
    0:[Account]:STRING=[868]:String
    1:[LastName]:STRING=[Hewitt]:String
    2:[FirstName]:STRING=[Jennifer Love]:String
    3:[Balance]:STRING=[0]:String
    4:[CreditLimit]:STRING=[17000.00]:String
    5:[AccountCreated]:STRING=[5/25/1985]:String
    6:[Rating]:STRING=[B]:String
    7:[ratingClass]:STRING=[Class B]:String
}

-----------------------------------------------
3 - Record {
    0:[Account]:STRING=[761]:String
    1:[LastName]:STRING=[Pinkett-Smith]:String
    2:[FirstName]:STRING=[Jada]:String
    3:[Balance]:STRING=[49654.87]:String
    4:[CreditLimit]:STRING=[100000.00]:String
    5:[AccountCreated]:STRING=[12/5/2006]:String
    6:[Rating]:STRING=[A]:String
    7:[ratingClass]:STRING=[Class A]:String
}

-----------------------------------------------
4 - Record {
    0:[Account]:STRING=[317]:String
    1:[LastName]:STRING=[Murray]:String
    2:[FirstName]:STRING=[Bill]:String
    3:[Balance]:STRING=[789.65]:String
    4:[CreditLimit]:STRING=[5000.00]:String
    5:[AccountCreated]:STRING=[2/5/2007]:String
    6:[Rating]:STRING=[C]:String
    7:[ratingClass]:STRING=[Class C]:String
}

-----------------------------------------------
5 records