Write to A Data Writer using Tee Reader

Updated: Feb 21, 2022

View on GitHub

This example shows you how TeeReader can be used to write to a DataWriter anywhere in the reading part of a pipeline. This allows you to record the data at that point in the pipeline to a data sink.

TeeReader is inspired by the Tee tool in Linux. Tee reads from the standard input and writes to both the standard output and one or more files simultaneously.

Similarly, TeeReader can be used in a pipeline that reads a file (purchase.csv) and writes to multiple files (purchases-total.csv and purchases-details.csv) at different stages in the flow.

Java Code Listing

package com.northconcepts.datapipeline.examples.cookbook;

import java.io.File;

import com.northconcepts.datapipeline.core.DataReader;
import com.northconcepts.datapipeline.core.DataWriter;
import com.northconcepts.datapipeline.core.TeeReader;
import com.northconcepts.datapipeline.csv.CSVReader;
import com.northconcepts.datapipeline.csv.CSVWriter;
import com.northconcepts.datapipeline.job.Job;
import com.northconcepts.datapipeline.transform.SelectFields;
import com.northconcepts.datapipeline.transform.TransformingWriter;

public class WriteToADataWriterUsingTeeReader {
	
	private static final String INPUT = "example/data/input";
    private static final String OUTPUT = "example/data/output";

	public static void main(String[] args) throws Throwable {
		
		DataReader reader = new CSVReader(new File(INPUT, "purchases.csv"))
                .setFieldNamesInFirstRow(true);
		
		DataWriter writer1 = new CSVWriter(new File(OUTPUT, "purchases-total.csv"))
				.setFieldNamesInFirstRow(false);
		
		writer1 = new TransformingWriter(writer1)
				.add(new SelectFields("product_name", "total"));
		
		DataWriter writer2 = new CSVWriter(new File(OUTPUT, "purchases-details.csv"))
				.setFieldNamesInFirstRow(false);
		
		reader = new TeeReader(reader, writer1, true);
		Job job = Job.run(reader, writer2);

		System.out.println(job);
	}

}