Read Json Lines

Updated: Jan 22, 2023

View on GitHub

This example shows you how to read data from a JSON Lines file in Java using the JsonReader class.

This demo code reads objects from a JSON Lines file via JsonReader and then writes an array of objects to the console via StreamWriter class.

This example can easily be modified to show how to read Json Lines as record using JsonRecordReader class.

Input JSON file

{"Employee": [{"empMID": "mock:1", "comments": [], "col1": "something", "contact": [{"address":"2400 waterview", "freetext":true}], "gender": "male"}, {"empMID": "mock:2", "comments": [], "col1": "something", "contact": [{"address":"2200 waterview", "freetext":true}], "gender": "female"}], "cola": false, "colb": false}
{"Employee": [{"empMID": "mock:10", "comments": [], "col1": "something", "contact": [{"address":"2410 waterview", "freetext":true}], "gender": "female"}, {"empMID": "mock:11", "comments": [], "col1": "something", "contact": [{"address":"2210 waterview", "freetext":true}], "gender": "female"}], "cola": false, "colb": false}

Java Code Listing

package com.northconcepts.datapipeline.examples.cookbook;

import com.northconcepts.datapipeline.core.DataReader;
import com.northconcepts.datapipeline.core.DataWriter;
import com.northconcepts.datapipeline.core.StreamWriter;
import com.northconcepts.datapipeline.job.Job;
import com.northconcepts.datapipeline.json.JsonReader;

import java.io.File;

public class ReadJsonLines {
    public static void main(String[] args) {
        DataReader reader = new JsonReader(new File("data/input/json-input-with-lines.jsonl"))
                .addField("empMID", "/object/Employee/array/object/empMID")
                .addField("comments", "/object/Employee/array/object/comments")
                .addField("col1", "/object/Employee/array/object/col1")
                .addField("address", "/object/Employee/array/object/contact/array/object/address")
                .addRecordBreak("/object/Employee/array/object");

        DataWriter writer = StreamWriter.newSystemOutWriter();

        Job.run(reader, writer);
    }
}

Code Walkthrough

1. First, JsonReader object is created corresponding to the input file json-input-with-lines.jsonl.
2. .addField() method is used to add the specific field that you want to extract from the JSON file e.g. for this example, four fields have been extracted from the input file i.e. empMID,comments,col1 and address.
3. In order to specify where the fields are located in the file we define a paths e.g. /object/Employee/array/object/empMID is used to specify the path to empMID field. Incase you need to see the paths to the specific fields in the file you need to run your code when .setDebug(true), it is false by default. e.g ...addRecordBreak("/object/Employee/array/object").setDebug(true);
4. .addRecordBreak() method is used to separate the records.
5. Data is extracted from the JSON file via Job.run().
6. Array of objects are then printed on the console via StreamWriter class.

Console Output

-----------------------------------------------
0 - Record (MODIFIED) {
    0:[empMID]:STRING=[mock:1]:String
    1:[comments]:UNDEFINED=[null]
    2:[col1]:STRING=[something]:String
    3:[address]:STRING=[2400 waterview]:String
}

-----------------------------------------------
1 - Record (MODIFIED) {
    0:[empMID]:STRING=[mock:2]:String
    1:[comments]:UNDEFINED=[null]
    2:[col1]:STRING=[something]:String
    3:[address]:STRING=[2200 waterview]:String
}

-----------------------------------------------
2 - Record (MODIFIED) {
    0:[empMID]:STRING=[mock:10]:String
    1:[comments]:UNDEFINED=[null]
    2:[col1]:STRING=[something]:String
    3:[address]:STRING=[2410 waterview]:String
}

-----------------------------------------------
3 - Record (MODIFIED) {
    0:[empMID]:STRING=[mock:11]:String
    1:[comments]:UNDEFINED=[null]
    2:[col1]:STRING=[something]:String
    3:[address]:STRING=[2210 waterview]:String
}

-----------------------------------------------
4 records