Write to Memory

This example shows you how to write data to memory using the MemoryWriter class. The demo code reads the contents of a CSV file and writes it to memory. However other input sources like CSV, Excel,XML, etc can also be used to write to memory.

This example can easily be modified to show how to read from memory.

Input CSV file

Account,LastName,FirstName,Balance,CreditLimit,AccountCreated,Rating
101,Reeves,Keanu,9315.45,10000.00,1/17/1998,A
312,Butler,Gerard,90.00,1000.00,8/6/2003,B
868,Hewitt,Jennifer Love,0,17000.00,5/25/1985,B
761,Pinkett-Smith,Jada,49654.87,100000.00,12/5/2006,A
317,Murray,Bill,789.65,5000.00,2/5/2007,C

Java Code Listing

package com.northconcepts.datapipeline.examples.cookbook;

import java.io.File;

import org.apache.log4j.Logger;

import com.northconcepts.datapipeline.core.DataEndpoint;
import com.northconcepts.datapipeline.core.DataReader;
import com.northconcepts.datapipeline.core.RecordList;
import com.northconcepts.datapipeline.csv.CSVReader;
import com.northconcepts.datapipeline.job.Job;
import com.northconcepts.datapipeline.memory.MemoryWriter;

public class WriteToMemory {
    
    public static final Logger log = DataEndpoint.log; 

    public static void main(String[] args) {
        DataReader reader = new CSVReader(new File("example/data/input/credit-balance-01.csv"))
            .setFieldNamesInFirstRow(true);

        MemoryWriter memoryWriter = new  MemoryWriter();
    
        Job.run(reader, memoryWriter);
        
        RecordList recordList = memoryWriter.getRecordList();
        for (int i = 0; i < recordList.getRecordCount(); i++) {
            log.info(recordList.get(i));
        }
    }

}

Code Walkthrough

1. A CSVReader is created using the file path of the input file credit-balance-01.csv.
2. A new MemoryWriter is created.
3. Data is transferred from the input CSV file to memory via JobTemplate.DEFAULT.transfer method.
4. The RecordList is retrieved from the MemoryWriter.
5. A for loop iterates through the RecordList and each Record is displayed on the console via the Datapipeline logger.

MemoryWriter

A MemoryWriter is an output writer which can be used to create data in memory. When data is written via the MemoryWriter, a new RecordList is created which can be retrieved via memoryWriter.getRecordList.
RecordList is a data structure for storing Record objects. Record encapsulates a single line of information and is the basic unit of data transfer. Data transfers of all types occur as Record objects i.e. data from an input source is read as a Record object and written to the output source as a Record object.

Console output

14:25:52,118 DEBUG [main] datapipeline:60 - job::Start
14:25:52,128 DEBUG [main] datapipeline:72 - job::Success
14:25:52,129  INFO [main] datapipeline:35 - Record {
    0:[Account]:STRING=[101]:String
    1:[LastName]:STRING=[Reeves]:String
    2:[FirstName]:STRING=[Keanu]:String
    3:[Balance]:STRING=[9315.45]:String
    4:[CreditLimit]:STRING=[10000.00]:String
    5:[AccountCreated]:STRING=[1/17/1998]:String
    6:[Rating]:STRING=[A]:String
}

14:25:52,129  INFO [main] datapipeline:35 - Record {
    0:[Account]:STRING=[312]:String
    1:[LastName]:STRING=[Butler]:String
    2:[FirstName]:STRING=[Gerard]:String
    3:[Balance]:STRING=[90.00]:String
    4:[CreditLimit]:STRING=[1000.00]:String
    5:[AccountCreated]:STRING=[8/6/2003]:String
    6:[Rating]:STRING=[B]:String
}

14:25:52,129  INFO [main] datapipeline:35 - Record {
    0:[Account]:STRING=[868]:String
    1:[LastName]:STRING=[Hewitt]:String
    2:[FirstName]:STRING=[Jennifer Love]:String
    3:[Balance]:STRING=[0]:String
    4:[CreditLimit]:STRING=[17000.00]:String
    5:[AccountCreated]:STRING=[5/25/1985]:String
    6:[Rating]:STRING=[B]:String
}

14:25:52,129  INFO [main] datapipeline:35 - Record {
    0:[Account]:STRING=[761]:String
    1:[LastName]:STRING=[Pinkett-Smith]:String
    2:[FirstName]:STRING=[Jada]:String
    3:[Balance]:STRING=[49654.87]:String
    4:[CreditLimit]:STRING=[100000.00]:String
    5:[AccountCreated]:STRING=[12/5/2006]:String
    6:[Rating]:STRING=[A]:String
}

14:25:52,130  INFO [main] datapipeline:35 - Record {
    0:[Account]:STRING=[317]:String
    1:[LastName]:STRING=[Murray]:String
    2:[FirstName]:STRING=[Bill]:String
    3:[Balance]:STRING=[789.65]:String
    4:[CreditLimit]:STRING=[5000.00]:String
    5:[AccountCreated]:STRING=[2/5/2007]:String
    6:[Rating]:STRING=[C]:String
}