Add Nonpersistent Data to Records and Fields

Updated: Jun 1, 2023

View on GitHub

In this example you will learn how to use DataPipline to add non-persistent data to both a record and a field. For this example MemoryReader will be used to read the record data and CSVWriter will be used to write to a CSV file.

Java Code listing

/*
 * Copyright (c) 2006-2022 North Concepts Inc.  All rights reserved.
 * Proprietary and Confidential.  Use is subject to license terms.
 * 
 * https://northconcepts.com/data-pipeline/licensing/
 */
package com.northconcepts.datapipeline.examples.cookbook;

import java.io.File;

import com.northconcepts.datapipeline.core.DataReader;
import com.northconcepts.datapipeline.core.DataWriter;
import com.northconcepts.datapipeline.core.Record;
import com.northconcepts.datapipeline.core.RecordList;
import com.northconcepts.datapipeline.csv.CSVWriter;
import com.northconcepts.datapipeline.job.Job;
import com.northconcepts.datapipeline.memory.MemoryReader;

public class AddNonPersistentDataToRecordsAndFields {

    public static void main(String[] args) {
        Record record1 = new Record();
        record1.setField("name", "John Wayne");
        record1.getField("name").setSessionProperty("comment", "Wild West"); //non persistent data on field
        record1.setField("balance", 156.35);
 
        Record record2 = new Record();
        record2.setField("name", "Peter Parker");
        record2.setField("balance", 0.96);
        record2.setSessionProperty("comment", "Marvel Super Hero"); //non persistent data on record
         
        DataReader reader = new MemoryReader(new RecordList(record1, record2));
        DataWriter writer = new CSVWriter(new File("example/data/output/credit-balance-02.csv"));
     
        Job.run(reader, writer);

    }

}

Code Walkthrough

1. record1.setField("name", "John Wayne") is used to add a name field with the data John Wayne in record1.
2. record1.getField("name").setSessionProperty("comment", "Wild West") is used to create non-persistent data with the field comment and value Wild West in the name field. setSessionProperty is used to create the non-persistent data.
3. record2.setField("balance", 0.96) is used to create balance field with value 0.96 in record2.
4. record2.setSessionProperty("comment", "Marvel Super Hero") is used to create comment field with value Marvel Super Hero in record2.
5. System.out.println(record1.getField("name").getSessionProperty("comment")) can be used incase you want to see the non-persistent data in the field and System.out.println(record2.getSessionProperty("comment")) is used to print the non-persistent data in the record.
6. MemoryReader is used to obtain records from an in-memory RecordList i.e. in this case record1 and record2.
7. CSVWriter is created corresponding to the output file credit-balance-02.csv.
8. Job.run(reader, writer) is used to tranfer the data from the record1 to record2.

Record

Record holds persistent data in key-value fields as it flows through the pipeline. Values in a field can be:

• Single values (boolean, date, string, etc. -- see FieldType)
• Records.
• Array of single values, records, and arrays.

CSV Output

name,balance
John Wayne,156.35
Peter Parker,0.96