Write Concatenated JSON

Updated: Sep 26, 2022

View on GitHub

This example shows you how to write concatenated JSON using JsonLinesWriter.

Several Records are used to create the sample input data. You can also use a file as input instead as seen in the Write Concatenated JSON With Nested Data example.

Java Code Listing

package com.northconcepts.datapipeline.examples.cookbook;

import com.northconcepts.datapipeline.core.DataWriter;
import com.northconcepts.datapipeline.core.Record;
import com.northconcepts.datapipeline.core.RecordList;
import com.northconcepts.datapipeline.job.Job;
import com.northconcepts.datapipeline.json.JsonLinesWriter;
import com.northconcepts.datapipeline.memory.MemoryReader;


import java.io.File;

public class WriteConcatenatedJson {
    public static void main(String[] args) {
        Record record1 = new Record();
        record1.setField("stageName", "John Wayne");
        record1.setField("realName", "Marion Robert Morrison");
        record1.setField("gender", "male");
        record1.setField("city", "Winterset");
        record1.setField("balance", 156.35);

        Record record2 = new Record();
        record2.setField("stageName", "Spiderman");
        record2.setField("realName", "Peter Parker");
        record2.setField("gender","male");
        record2.setField("city", "New York");
        record2.setField("balance", -0.96);

        MemoryReader reader = new MemoryReader(new RecordList(record1, record2));
        
        DataWriter writer = new JsonLinesWriter(new File("data/output/concatenated-json.jsonl")).setNewLine("");

        Job.run(reader, writer);
    }
}

Code Walkthrough

1. MemoryReader is used to obtain the records from RecordList.
2. A JsonLinesWriter object is created to write to the concatenated-json.jsonl output file. By default JsonLinesWriter will print each record on a new line so .setNewLine("") is added in order to concatenate the JSON data without a line break.
3. Data is then transferred from the reader to the output file via Job.run().

Output

{"stageName":"John Wayne","realName":"Marion Robert Morrison","gender":"male","city":"Winterset","balance":156.35}{"stageName":"Spiderman","realName":"Peter Parker","gender":"male","city":"New York","balance":-0.96}