Throttle Data being Written

Updated: Feb 21, 2022

This example shows you how to limit the rate (bytes/second) at which data is written using the ThrottledWriter class.

The demo code reads a CSV file and writes its contents to a new CSV file via ThrottledWriter. However, the ThrottledWriter can be used as a wrapper across other output types like Excel and XML as well.

This example can easily be modified to show how to throttle data being read.

Input CSV file

868,Hewitt,Jennifer Love,0,17000.00,5/25/1985,B

Java Code Listing

package com.northconcepts.datapipeline.examples.cookbook;


import org.apache.log4j.Logger;

import com.northconcepts.datapipeline.core.DataEndpoint;
import com.northconcepts.datapipeline.core.DataReader;
import com.northconcepts.datapipeline.core.DataWriter;
import com.northconcepts.datapipeline.csv.CSVReader;
import com.northconcepts.datapipeline.csv.CSVWriter;
import com.northconcepts.datapipeline.job.Job;
import com.northconcepts.datapipeline.throttle.ThrottledWriter;

public class ThrottleDataBeingWritten {
    public static final Logger log = DataEndpoint.log; 

    private static final int MAX_BYTES_PER_SECOND = 256;

    public static void main(String[] args) throws Throwable {
        DataReader reader = new CSVReader(new File("example/data/input/credit-balance-01.csv"))
        DataWriter writer = new CSVWriter(new File("example/data/output/credit-balance-04.csv"))

        ThrottledWriter throttledWriter = new ThrottledWriter(writer, MAX_BYTES_PER_SECOND);, throttledWriter);
       "write rate: " + throttledWriter.getMeter().getUnitsPerSecondAsString());


Code Walkthrough

  1. A static constant called MAX_BYTES_PER_SECOND is defined which specifies the maximum number of bytes per second which may be read by the ThrottledWriter.
  2. A CSVReader is created corresponding to the input CSV file credit-balance-01.csv.
  3. A CSVWriter is created corresponding to the output CSV file credit-balance-04.csv.
  4. A new ThrottledWriter is created using the CSVWriter and MAX_BYTES_PER_SECOND.
  5. Data is then transferred from the input CSV file to the output CSV file via JobTemplate.DEFAULT.transfer method.
  6. The rate at which data is written is logged.


A ThrottledWriter is an output writer which can be used to limit the rate at which data is written. It can be created as a wrapper around any output writer and a data transfer rate can be specified while creating it. So the ThrottledWriter will never write data at a rate higher than what is specified while creating it.

Displaying transfer rate

The ThrottledWriter is a sub-class of MeteredWriter and therefore inherits the getMeter() method which returns a Meter instance. Meter is a generic counter which has the capability of recording statistics related to data transfer. The Meter.getUnitsPerSecondAsString method can be used to retrieve the actual data transfer rate as kilo-bytes/second. This data transfer rate will always be lower than that specified while creating the ThrottledWriter.

Console output

11:50:23,782 DEBUG [main] datapipeline:37 - Data Pipeline v2.3.3 by North Concepts Inc.
11:50:23,794 DEBUG [main] datapipeline:60 - job::Start
11:50:25,543 DEBUG [main] datapipeline:72 - job::Success
11:50:25,545  INFO [main] datapipeline:39 - write rate: 0.3 kilo-bytes/s
Mobile Analytics