Add Non-Persistent Data to Records and Fields

This example shows how to add non-persistent data to both records and fields in a dataset. This means that users can supplement the existing data with additional information at either the record level or the individual field level. This functionality can be beneficial in various real-life use cases, such as augmenting customer profiles with temporary attributes, attaching contextual metadata to specific data fields, or incorporating calculated values for analysis or reporting purposes.

Java Code Listing

package com.northconcepts.datapipeline.examples.cookbook;

import java.io.File;

import com.northconcepts.datapipeline.core.DataReader;
import com.northconcepts.datapipeline.core.DataWriter;
import com.northconcepts.datapipeline.core.Record;
import com.northconcepts.datapipeline.core.RecordList;
import com.northconcepts.datapipeline.csv.CSVWriter;
import com.northconcepts.datapipeline.job.Job;
import com.northconcepts.datapipeline.memory.MemoryReader;

public class AddNonPersistentDataToRecordsAndFields {

    public static void main(String[] args) {
        Record record1 = new Record();
        record1.setField("name", "John Wayne");
        record1.getField("name").setSessionProperty("comment", "Wild West"); //non persistent data on field
        record1.setField("balance", 156.35);
 
        Record record2 = new Record();
        record2.setField("name", "Peter Parker");
        record2.setField("balance", 0.96);
        record2.setSessionProperty("comment", "Marvel Super Hero"); //non persistent data on record
         
        DataReader reader = new MemoryReader(new RecordList(record1, record2));
        DataWriter writer = new CSVWriter(new File("example/data/output/credit-balance-02.csv"));
     
        Job.run(reader, writer);

    }

}

Code Walkthrough

1. Record instance is created and filled with data in a fieldName-value pattern via setField() method. In the given example, the "name" and "balance" fields are added with values.
2. record1.getField("name").setSessionProperty("comment", "Wild West") is used to create non-persistent data with the field comment and value Wild West in the name field. setSessionProperty is used to define non-persistent data.
3. The second Record object called record2 is created and two fields are added.
4. record2.setSessionProperty("comment", "Marvel Super Hero") is used to create non-persistent comment field with the value Marvel Super Hero in record2.
5. MemoryReader is used to obtain records from an in-memory RecordList i.e. in this case record1 and record2.
6. CSVWriter is created corresponding to the output file credit-balance-02.csv.
7. Job.run(reader, writer) is used to transfer the data from the record1 to record2.
8. Non-persistent data is not included in the output file. However, they can be accessed and printed in the console via the following methods:
   1. System.out.println(record1.getField("name").getSessionProperty("comment")) - non-persistent data in the field;
   2. System.out.println(record2.getSessionProperty("comment")) - non-persistent data in the record.

Record

The Record holds persistent data in key-value fields as it flows through the pipeline. Values in a field can be:

• Single values (boolean, date, string, etc. -- see FieldType)
• Records.
• An array of single values, records, and arrays.

CSV Output

name,balance
John Wayne,156.35
Peter Parker,0.96