Throttle Data being Written

Updated: Feb 21, 2022

View on GitHub

This example shows you how to limit the rate (bytes/second) at which data is written using the ThrottledWriter class.

The demo code reads a CSV file and writes its contents to a new CSV file via ThrottledWriter. However, the ThrottledWriter can be used as a wrapper across other output types like Excel and XML as well.

This example can easily be modified to show how to throttle data being read.

Input CSV file

Account,LastName,FirstName,Balance,CreditLimit,AccountCreated,Rating
101,Reeves,Keanu,9315.45,10000.00,1/17/1998,A
312,Butler,Gerard,90.00,1000.00,8/6/2003,B
868,Hewitt,Jennifer Love,0,17000.00,5/25/1985,B
761,Pinkett-Smith,Jada,49654.87,100000.00,12/5/2006,A
317,Murray,Bill,789.65,5000.00,2/5/2007,C

Java Code Listing

package com.northconcepts.datapipeline.examples.cookbook;

import java.io.File;

import org.apache.log4j.Logger;

import com.northconcepts.datapipeline.core.DataEndpoint;
import com.northconcepts.datapipeline.core.DataReader;
import com.northconcepts.datapipeline.core.DataWriter;
import com.northconcepts.datapipeline.csv.CSVReader;
import com.northconcepts.datapipeline.csv.CSVWriter;
import com.northconcepts.datapipeline.job.Job;
import com.northconcepts.datapipeline.throttle.ThrottledWriter;

public class ThrottleDataBeingWritten {
    
    public static final Logger log = DataEndpoint.log; 

    private static final int MAX_BYTES_PER_SECOND = 256;

    public static void main(String[] args) throws Throwable {
        DataReader reader = new CSVReader(new File("example/data/input/credit-balance-01.csv"))
            .setFieldNamesInFirstRow(true);
        
        DataWriter writer = new CSVWriter(new File("example/data/output/credit-balance-04.csv"))
            .setFieldNamesInFirstRow(true);

        ThrottledWriter throttledWriter = new ThrottledWriter(writer, MAX_BYTES_PER_SECOND);

        Job.run(reader, throttledWriter);
        
        log.info("write rate: " + throttledWriter.getMeter().getUnitsPerSecondAsString());
    }

}

Code Walkthrough

1. A static constant called MAX_BYTES_PER_SECOND is defined which specifies the maximum number of bytes per second which may be read by the ThrottledWriter.
2. A CSVReader is created corresponding to the input CSV file credit-balance-01.csv.
3. A CSVWriter is created corresponding to the output CSV file credit-balance-04.csv.
4. A new ThrottledWriter is created using the CSVWriter and MAX_BYTES_PER_SECOND.
5. Data is then transferred from the input CSV file to the output CSV file via JobTemplate.DEFAULT.transfer method.
6. The rate at which data is written is logged.

ThrottledWriter

A ThrottledWriter is an output writer which can be used to limit the rate at which data is written. It can be created as a wrapper around any output writer and a data transfer rate can be specified while creating it. So the ThrottledWriter will never write data at a rate higher than what is specified while creating it.

Displaying transfer rate

The ThrottledWriter is a sub-class of MeteredWriter and therefore inherits the getMeter() method which returns a Meter instance. Meter is a generic counter which has the capability of recording statistics related to data transfer. The Meter.getUnitsPerSecondAsString method can be used to retrieve the actual data transfer rate as kilo-bytes/second. This data transfer rate will always be lower than that specified while creating the ThrottledWriter.

Console output

11:50:23,782 DEBUG [main] datapipeline:37 - Data Pipeline v2.3.3 by North Concepts Inc.
11:50:23,794 DEBUG [main] datapipeline:60 - job::Start
11:50:25,543 DEBUG [main] datapipeline:72 - job::Success
11:50:25,545  INFO [main] datapipeline:39 - write rate: 0.3 kilo-bytes/s