Debug my Code

Updated: Aug 24, 2023

View on GitHub

This example shows you how to debug code from the data pipeline using the DebugReader. This class provides a debugging mechanism for pipelines by allowing the contents of input files or other data transfers to be printed to the console. It helps you to inspect and verify the data flowing through the pipeline, aiding in the identification and resolution of issues during the debugging process.

When performing complex data transformations or manipulations within a pipeline, DataPipeline can help validate the correctness of the transformation logic. By printing the intermediate or final data to the console, you can visually inspect the transformed data, ensuring it aligns with their expectations and requirements.

Other listed examples demonstrate how to proceed after encountering an error or handle exceptions.

 

Input CSV file

Account,LastName,FirstName,Balance,CreditLimit,AccountCreated,Rating
101,Reeves,Keanu,9315.45,10000.00,1/17/1998,A
312,"Butler,Gerard,90.00,1000.00,8/6/2003,B
868,Hewitt,Jennifer Love,0,17000.00,5/25/1985,B
761,Pinkett-Smith,Jada,49654.87,100000.00,12/5/2006,A
317,Murray,Bill,789.65,5000.00,2/5/2007,C

 

Java Code Listing

package com.northconcepts.datapipeline.examples.cookbook;

import java.io.File;

import com.northconcepts.datapipeline.core.DataReader;
import com.northconcepts.datapipeline.core.DataWriter;
import com.northconcepts.datapipeline.core.DebugReader;
import com.northconcepts.datapipeline.csv.CSVReader;
import com.northconcepts.datapipeline.csv.CSVWriter;
import com.northconcepts.datapipeline.job.Job;

public class DebugMyCode {
    
    public static void main(String[] args) throws Throwable {
        DataReader reader = new CSVReader(new File("example/data/input/credit-balance-01.csv"))
            .setFieldNamesInFirstRow(true);

        DataWriter writer = new CSVWriter(new File("example/data/output/credit-balance-04.csv"))
            .setFieldNamesInFirstRow(true);

        // print each record to System.out as they are read, 
        // prefixing the output with the text "from file"
        DebugReader debugReader = new DebugReader(reader, "from file", System.out);
    
        Job.run(debugReader, writer);
    }

}

 

Code Walkthrough

1. A CSVReader is created using the file path of the input file credit-balance-01.csv.
2. A CSVWriter is created using the file path of the output file credit-balance-04.csv.
3. A DebugReader wrapped around the CSVReader is created. Each record is printed on System.out as they are read, prefixing the output with the text "from file".
4. Data is transferred from the input CSV file to the output CSV file via JobTemplate.DEFAULT.transfer method.

DebugReader

A DebugReader can be wrapped around any Reader for debugging purposes. It modifies the input data and displays it in the specified OutputStream. It can be created using the following parameters:

• DataReader - This is the reader corresponding to the input file whose contents are being debugged.
• String - This is the text that is to be prefixed to the output. Any text can be used here.
• OutputStream - This is the output stream where the output will be printed. System.out is used for demo purpose, it can be replaced with any OutputStream.

 

Output CSV file

Account,LastName,FirstName,Balance,CreditLimit,AccountCreated,Rating
101,Reeves,Keanu,9315.45,10000.00,1/17/1998,A
312,Butler,Gerard,90.00,1000.00,8/6/2003,B
868,Hewitt,Jennifer Love,0,17000.00,5/25/1985,B
761,Pinkett-Smith,Jada,49654.87,100000.00,12/5/2006,A
317,Murray,Bill,789.65,5000.00,2/5/2007,C

 

Console output

==========================================================
from file
------------------------------------------
Record {
    0:[Account]:STRING=[101]:String
    1:[LastName]:STRING=[Reeves]:String
    2:[FirstName]:STRING=[Keanu]:String
    3:[Balance]:STRING=[9315.45]:String
    4:[CreditLimit]:STRING=[10000.00]:String
    5:[AccountCreated]:STRING=[1/17/1998]:String
    6:[Rating]:STRING=[A]:String
}

==========================================================
from file
------------------------------------------
Record {
    0:[Account]:STRING=[312]:String
    1:[LastName]:STRING=[Butler]:String
    2:[FirstName]:STRING=[Gerard]:String
    3:[Balance]:STRING=[90.00]:String
    4:[CreditLimit]:STRING=[1000.00]:String
    5:[AccountCreated]:STRING=[8/6/2003]:String
    6:[Rating]:STRING=[B]:String
}

==========================================================
from file
------------------------------------------
Record {
    0:[Account]:STRING=[868]:String
    1:[LastName]:STRING=[Hewitt]:String
    2:[FirstName]:STRING=[Jennifer Love]:String
    3:[Balance]:STRING=[0]:String
    4:[CreditLimit]:STRING=[17000.00]:String
    5:[AccountCreated]:STRING=[5/25/1985]:String
    6:[Rating]:STRING=[B]:String
}

==========================================================
from file
------------------------------------------
Record {
    0:[Account]:STRING=[761]:String
    1:[LastName]:STRING=[Pinkett-Smith]:String
    2:[FirstName]:STRING=[Jada]:String
    3:[Balance]:STRING=[49654.87]:String
    4:[CreditLimit]:STRING=[100000.00]:String
    5:[AccountCreated]:STRING=[12/5/2006]:String
    6:[Rating]:STRING=[A]:String
}

==========================================================
from file
------------------------------------------
Record {
    0:[Account]:STRING=[317]:String
    1:[LastName]:STRING=[Murray]:String
    2:[FirstName]:STRING=[Bill]:String
    3:[Balance]:STRING=[789.65]:String
    4:[CreditLimit]:STRING=[5000.00]:String
    5:[AccountCreated]:STRING=[2/5/2007]:String
    6:[Rating]:STRING=[C]:String
}

21:48:02,252 DEBUG [main] datapipeline:72 - job::Success