Throttle Data being Read

Updated: Feb 21, 2022

View on GitHub

This example shows you how to limit the rate (bytes/second) at which data is read using the ThrottledReader class.

The demo code reads a CSV file via ThrottledReader and writes its contents to a new CSV file. However, the ThrottledReader can be used as a wrapper across other input types like Excel and XML as well.

This example can easily be modified to show how to throttle data being written.

Input CSV file

Account,LastName,FirstName,Balance,CreditLimit,AccountCreated,Rating
101,Reeves,Keanu,9315.45,10000.00,1/17/1998,A
312,Butler,Gerard,90.00,1000.00,8/6/2003,B
868,Hewitt,Jennifer Love,0,17000.00,5/25/1985,B
761,Pinkett-Smith,Jada,49654.87,100000.00,12/5/2006,A
317,Murray,Bill,789.65,5000.00,2/5/2007,C

Java Code Listing

package com.northconcepts.datapipeline.examples.cookbook;

import java.io.File;

import org.apache.log4j.Logger;

import com.northconcepts.datapipeline.core.DataEndpoint;
import com.northconcepts.datapipeline.core.DataReader;
import com.northconcepts.datapipeline.core.DataWriter;
import com.northconcepts.datapipeline.csv.CSVReader;
import com.northconcepts.datapipeline.csv.CSVWriter;
import com.northconcepts.datapipeline.job.Job;
import com.northconcepts.datapipeline.throttle.ThrottledReader;

public class ThrottleDataBeingRead {
    
    public static final Logger log = DataEndpoint.log; 

    private static final int MAX_BYTES_PER_SECOND = 1024;

    public static void main(String[] args) throws Throwable {
        DataReader reader = new CSVReader(new File("example/data/input/credit-balance-01.csv"))
            .setFieldNamesInFirstRow(true);
        
        DataWriter writer = new CSVWriter(new File("example/data/output/credit-balance-04.csv"))
            .setFieldNamesInFirstRow(true);

        ThrottledReader throttledReader = new ThrottledReader(reader, MAX_BYTES_PER_SECOND);

        Job.run(throttledReader, writer);
        log.info("read rate: " + throttledReader.getMeter().getUnitsPerSecondAsString());
    }

}

Code walkthrough

1. A static constant called MAX_BYTES_PER_SECOND is defined which specifies the maximum number of bytes per second which may be read by the ThrottledReader.
2. A CSVReader is created corresponding to the input CSV file credit-balance-01.csv.
3. A CSVWriter is created corresponding to the output CSV file credit-balance-04.csv.
4. A new ThrottledReader is created using the CSVReader and MAX_BYTES_PER_SECOND.
5. Data is then transferred from the input CSV file to the output CSV file via JobTemplate.DEFAULT.transfer method.
6. The rate at which data is read is logged.

ThrottledReader

A ThrottledReader is an input reader which can be used to limit the rate at which data is read. It can be created as a wrapper around any input reader and a data transfer rate can be specified while creating it. So the ThrottledReader will never read data at a rate higher than what is specified while creating it.

Displaying transfer rate

The ThrottledReader is a sub-class of MeteredReader and therefore inherits the getMeter() method which returns a Meter instance. Meter is a generic counter which has the capability of recording statistics related to data transfer. The Meter.getUnitsPerSecondAsString method can be used to retrieve the actual data transfer rate as kilo-bytes/second. This data transfer rate will always be lower than that specified while creating the ThrottledReader.

Console Output

18:53:34,931 DEBUG [main] datapipeline:37 - Data Pipeline v2.3.3 by North Concepts Inc.
18:53:34,959 DEBUG [main] datapipeline:60 - job::Start
18:53:35,867 DEBUG [main] datapipeline:72 - job::Success
18:53:35,893  INFO [main] datapipeline:38 - read rate: 0.7 kilo-bytes/s