Select and Arrange Fields

In this example, you will learn how you can use DataPipline to customize datasets by selecting specific fields and arranging them in a desired order. It provides you with the flexibility to extract and organize data according to their specific requirements, allowing for tailored datasets that meet your unique needs.

Users can customize datasets to share with specific stakeholders or collaborators. By selecting and arranging relevant fields, users can create focused datasets that contain only the necessary information for the intended recipients, streamlining data sharing and collaboration processes.

Input CSV file

Account,LastName,FirstName,Balance,CreditLimit,AccountCreated,Rating
101,Reeves,Keanu,9315.45,10000.00,1/17/1998,A
312,Butler,Gerard,90.00,1000.00,8/6/2003,B
868,Hewitt,Jennifer Love,0,17000.00,5/25/1985,B
761,Pinkett-Smith,Jada,49654.87,100000.00,12/5/2006,A
317,Murray,Bill,789.65,5000.00,2/5/2007,C

Java Code Listing

package com.northconcepts.datapipeline.examples.cookbook;

import java.io.File;

import com.northconcepts.datapipeline.core.DataReader;
import com.northconcepts.datapipeline.core.StreamWriter;
import com.northconcepts.datapipeline.csv.CSVReader;
import com.northconcepts.datapipeline.job.Job;
import com.northconcepts.datapipeline.transform.SelectFields;
import com.northconcepts.datapipeline.transform.TransformingReader;

public class SelectAndArrangeFields {
    
    public static void main(String[] args) throws Throwable {
        DataReader reader = new CSVReader(new File("example/data/input/credit-balance-01.csv"))
            .setFieldNamesInFirstRow(true);
        
        // remove all fields except 'FirstName', 'LastName', and 'Rating'; 
        // arrange them in that order
        reader = new TransformingReader(reader)
            .add(new SelectFields("FirstName", "LastName", "Rating"));
        
        Job.run(reader, new StreamWriter(System.out));
    }
    
}

Code walkthrough

1. CSVReader is created corresponding to the input file credit-balance-01.csv.
2. The CSVReader.setFieldNamesInFirstRow(true) method is invoked to specify that the names specified in the first row should be used as field names.
3. A TransformingReader is created to sequentially apply one or more transformations to the incoming data.
4. SelectFields is created and supplied with fieldNames as a variable number of String arguments (var args). Its function is to remove all fields except the fieldNames arguments and arrange them in that given order. In the example, only 'FirstName', 'LastName', and 'Rating' fields are retained.
5.  Data is transferred from the reader to the StreamWriter(System.out) via Job.run() method.

Output

-----------------------------------------------
0 - Record (MODIFIED) {
    0:[FirstName]:STRING=[Keanu]:String
    1:[LastName]:STRING=[Reeves]:String
    2:[Rating]:STRING=[A]:String
}

-----------------------------------------------
1 - Record (MODIFIED) {
    0:[FirstName]:STRING=[Gerard]:String
    1:[LastName]:STRING=[Butler]:String
    2:[Rating]:STRING=[B]:String
}

-----------------------------------------------
2 - Record (MODIFIED) {
    0:[FirstName]:STRING=[Jennifer Love]:String
    1:[LastName]:STRING=[Hewitt]:String
    2:[Rating]:STRING=[B]:String
}

-----------------------------------------------
3 - Record (MODIFIED) {
    0:[FirstName]:STRING=[Jada]:String
    1:[LastName]:STRING=[Pinkett-Smith]:String
    2:[Rating]:STRING=[A]:String
}

-----------------------------------------------
4 - Record (MODIFIED) {
    0:[FirstName]:STRING=[Bill]:String
    1:[LastName]:STRING=[Murray]:String
    2:[Rating]:STRING=[C]:String
}

-----------------------------------------------
5 records