Examples >

Throttle Data being Read

This example shows you how to limit the rate (bytes/second) at which data is read using the ThrottledReader class.

The demo code reads a CSV file via ThrottledReader and writes its contents to a new CSV file. However, the ThrottledReader can be used as a wrapper across other input types like Excel and XML as well.

This example can easily be modified to show how to throttle data being written.

Input CSV file

868,Hewitt,Jennifer Love,0,17000.00,5/25/1985,B

Java Code Listing

 * Copyright (c) 2006-2018 North Concepts Inc.  All rights reserved.
 * Proprietary and Confidential.  Use is subject to license terms.
 * http://northconcepts.com/data-pipeline/licensing/
package com.northconcepts.datapipeline.examples.cookbook;

import java.io.File;

import org.apache.log4j.Logger;

import com.northconcepts.datapipeline.core.DataEndpoint;
import com.northconcepts.datapipeline.core.DataReader;
import com.northconcepts.datapipeline.core.DataWriter;
import com.northconcepts.datapipeline.csv.CSVReader;
import com.northconcepts.datapipeline.csv.CSVWriter;
import com.northconcepts.datapipeline.job.Job;
import com.northconcepts.datapipeline.throttle.ThrottledReader;

public class ThrottleDataBeingRead {
    public static final Logger log = DataEndpoint.log; 

    private static final int MAX_BYTES_PER_SECOND = 1024;

    public static void main(String[] args) throws Throwable {
        DataReader reader = new CSVReader(new File("example/data/input/credit-balance-01.csv"))
        DataWriter writer = new CSVWriter(new File("example/data/output/credit-balance-04.csv"))

        ThrottledReader throttledReader = new ThrottledReader(reader, MAX_BYTES_PER_SECOND);

        Job.run(throttledReader, writer);
        log.info("read rate: " + throttledReader.getMeter().getUnitsPerSecondAsString());


Code walkthrough

  1. A static constant called MAX_BYTES_PER_SECOND is defined which specifies the maximum number of bytes per second which may be read by the ThrottledReader.
  2. A CSVReader is created corresponding to the input CSV file credit-balance-01.csv.
  3. A CSVWriter is created corresponding to the output CSV file credit-balance-04.csv.
  4. A new ThrottledReader is created using the CSVReader and MAX_BYTES_PER_SECOND.
  5. Data is then transferred from the input CSV file to the output CSV file via JobTemplate.DEFAULT.transfer method.
  6. The rate at which data is read is logged.


A ThrottledReader is an input reader which can be used to limit the rate at which data is read. It can be created as a wrapper around any input reader and a data transfer rate can be specified while creating it. So the ThrottledReader will never read data at a rate higher than what is specified while creating it.

Displaying transfer rate

The ThrottledReader is a sub-class of MeteredReader and therefore inherits the getMeter() method which returns a Meter instance. Meter is a generic counter which has the capability of recording statistics related to data transfer. The Meter.getUnitsPerSecondAsString method can be used to retrieve the actual data transfer rate as kilo-bytes/second. This data transfer rate will always be lower than that specified while creating the ThrottledReader.

Console Output

18:53:34,931 DEBUG [main] datapipeline:37 - Data Pipeline v2.3.3 by North Concepts Inc.
18:53:34,959 DEBUG [main] datapipeline:60 - job::Start
18:53:35,867 DEBUG [main] datapipeline:72 - job::Success
18:53:35,893  INFO [main] datapipeline:38 - read rate: 0.7 kilo-bytes/s
Mobile Analytics