Read using TimedReader

In this example, you are going to learn how to read records for a specific duration of time using TimedReader. This is applicable where you want to read records from a source for only 10 seconds for example. TimedReader works by reading an underlying DataReader for a maximum period of time.

You can apply this example for use cases like real-time data streaming, event processing, log monitoring, and live data analytics, where time-sensitive data acquisition and handling are paramount for timely insights and decision-making within Java-based applications.

Java Code Listing

package com.northconcepts.datapipeline.examples.cookbook;

import java.util.concurrent.TimeUnit;

import com.northconcepts.datapipeline.core.DataReader;
import com.northconcepts.datapipeline.core.NullWriter;
import com.northconcepts.datapipeline.core.TimedReader;
import com.northconcepts.datapipeline.examples.cookbook.common.card.CardReader;
import com.northconcepts.datapipeline.job.Job;

public class ReadUsingTimedReader {

    public static void main(String[] args) {
        // Create a CardReader to retrieve records with specified delay in read.
        DataReader reader = new CardReader()
                .setMaxRecords(10000)
                .setReadPeriod(TimeUnit.SECONDS.toMillis(1))
                .setStartingDate("2016-01-14 09:00:00.000")
                .setDateIncrement(500);

        // Use TimedReader to read for limited time.
        reader = new TimedReader(reader, TimeUnit.SECONDS.toMillis(10)); // Read only for 10 seconds.

        Job job = Job.run(reader, new NullWriter());

        System.out.println("Total Records retrieved :- " + job.getRecordsTransferred());
        System.out.println("Running Time in milli seconds :- " + job.getRunningTimeAsString());

    }
}

Code walkthrough

1. CardReader is used to retrieve the records with a specified delay. It generates fake records.
2. .setMaxRecords(10000) sets the maximum records that can be read i.e. 10000.
3. .setReadPeriod(TimeUnit.SECONDS.toMillis(1)) sets the time taken to read a record to 1 second.
4. .setStartingDate("2016-01-14 09:00:00.000") sets the starting date of reading the records. i.e. only records with this date or with dates greater than this will be read.
5. .setDateIncrement(500) increments the starting date by 500.
6. TimedReader is used to read data for a limited time. In this case, it reads data from the reader for only 10 seconds.
7. Job job = Job.run(reader, new NullWriter()) is used to copy the data from the reader to the NullWriter.
8. job.getRecordsTransferred() returns the total records that have been retrieved.
9. job.getRunningTimeAsString()) returns the total running time of the job in milliseconds.

Console Output

Total Records retrieved :- 10
Running Time in milli seconds :- 10 Seconds, 27 Milliseconds