Open and Close Several Data Readers and Data Writers at Once

Updated: Feb 21, 2022

View on GitHub

This example shows you how to open and close several data readers/writers at once using the DataEndpointGroup class.

The demo code reads a CSV file and writes its contents to a database table via CSVReader and JdbcWriter. It uses the DataEndpointGroup to open/close both the CSVReader and JdbcWriter in a single method call. However, other Reader/Writer types can be used instead of or in addition to these reader/writer classes.

Input CSV file

Account,LastName,FirstName,Balance,CreditLimit,AccountCreated,Rating
101,Reeves,Keanu,9315.45,10000.00,1/17/1998,A
312,Butler,Gerard,90.00,1000.00,8/6/2003,B
868,Hewitt,Jennifer Love,0,17000.00,5/25/1985,B
761,Pinkett-Smith,Jada,49654.87,100000.00,12/5/2006,A
317,Murray,Bill,789.65,5000.00,2/5/2007,C

Java Code Listing

package com.northconcepts.datapipeline.examples.cookbook;

import java.io.File;
import java.sql.Connection;
import java.sql.Driver;
import java.util.Properties;

import com.northconcepts.datapipeline.core.DataEndpointGroup;
import com.northconcepts.datapipeline.core.DataReader;
import com.northconcepts.datapipeline.core.DataWriter;
import com.northconcepts.datapipeline.core.Record;
import com.northconcepts.datapipeline.csv.CSVReader;
import com.northconcepts.datapipeline.jdbc.JdbcWriter;

public class OpenAndCloseSeveralDataReadersAndDataWritersAtOnce {
    
    public static void main(String[] args) throws Throwable {
        // connect to the database
        Driver driver = (Driver) Class.forName("sun.jdbc.odbc.JdbcOdbcDriver").newInstance();
        Properties properties = new Properties();
        properties.put("user", "scott");
        properties.put("password", "tiger");
        Connection connection = driver.connect("jdbc:odbc:dp-cookbook", properties);

        DataReader reader = new CSVReader(new File("example/data/input/credit-balance-01.csv"))
            .setFieldNamesInFirstRow(true);
        
        DataWriter writer = new  JdbcWriter(connection, "dp_credit_balance")
            .setAutoCloseConnection(true)
            .setBatchSize(100);
        
        DataEndpointGroup group = new DataEndpointGroup()
            .add(reader)
            .add(writer);

        group.open();
        try {
            Record record;
            while ((record = reader.read()) != null) {
                writer.write(record);
            }
        } finally {
            group.close();
        }
    }

}

Code Walkthrough

1. Lines 19-23 obtain a JDBC Driver and Connection to the database where the output is to be written, this is standard JDBC code.
2. A CSVReader is created to read the input file credit-balance-01.csv and assigned to a DataReader.
3. JdbcWriter is created corresponding to the output database table i.e. dp_credit_balance and assigned to a DataWriter.
4. A DataEndpointGroup is created and opened.
5. A while loop iterates through the input data and transfers the records from the reader to the writer as Record objects.
6. The DataEndpointGroup is then closed in a finally block.

DataEndpointGroup

DataEndpointGroup class is used to group together several readers and writers and open/close them using single method calls. DataEndpointGroup has an add method which accepts as parameters a variable number of DataEndPoint instances and returns a DataEndpointGroup. The DataEndpointGroup.open and DataEndpointGroup.close methods can then be used to open/close the added readers/writers in a single method call.

In the demo code, this method is invoked using the DataReader and and DataWriter references which are sub-classes of DataEndPoint. Note that is it not really necessary to use DataReader and DataWriter references, we can directly use CSVReader and JdbcWriter. The DataReader and DataWriter references are only used here to only show that any sub-class be used.

Database output

The output of this program would be the contents of CSV file written to the dp_credit_balance database table.