Read a JSON Stream
This example shows you how to create a JSON stream in Java using the JsonReader class.
This example also shows how the input data can be modified on the fly using the TransformingReader and BasicFieldTransformer classes.
This example can easily be modified to show how to write to a JSON stream.
JSON input
[
{
"id":"358464",
"t":"MSFT",
"e":"NASDAQ",
"l":"31.67",
"l_cur":"31.67",
"s":"2",
"ltt":"4:00PM EDT",
"lt":"Aug 1, 4:00PM EDT",
"c":"-0.17",
"cp":"-0.53",
"ccol":"chr",
"el":"31.69",
"el_cur":"31.69",
"elt":"Aug 1, 7:54PM EDT",
"ec":"+0.02",
"ecp":"0.06",
"eccol":"chg",
"div":"0.23",
"yld":"2.90"
},
{
"id":"419344",
"t":"ORCL",
"e":"NYSE",
"l":"32.75",
"l_cur":"32.75",
"s":"2",
"ltt":"4:00PM EDT",
"lt":"Aug 1, 4:00PM EDT",
"c":"+0.40",
"cp":"1.24",
"ccol":"chg",
"el":"32.70",
"el_cur":"32.70",
"elt":"Aug 1, 7:15PM EDT",
"ec":"-0.05",
"ecp":"-0.15",
"eccol":"chr",
"div":"",
"yld":"1.47"
},
{
"id":"4112",
"t":"ADBE",
"e":"NASDAQ",
"l":"47.70",
"l_cur":"47.70",
"s":"0",
"ltt":"4:00PM EDT",
"lt":"Aug 1, 4:00PM EDT",
"c":"+0.42",
"cp":"0.89",
"ccol":"chg"
}
]
This JSON Stream consists of an array of objects. This code demonstrates reading only the highlighted fields from each object in the array.
Java Code listing
/*
* Copyright (c) 2006-2022 North Concepts Inc. All rights reserved.
* Proprietary and Confidential. Use is subject to license terms.
*
* https://northconcepts.com/data-pipeline/licensing/
*/
package com.northconcepts.datapipeline.examples.cookbook;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.URL;
import com.northconcepts.datapipeline.core.DataReader;
import com.northconcepts.datapipeline.core.DataWriter;
import com.northconcepts.datapipeline.core.StreamWriter;
import com.northconcepts.datapipeline.job.Job;
import com.northconcepts.datapipeline.json.JsonReader;
import com.northconcepts.datapipeline.transform.BasicFieldTransformer;
import com.northconcepts.datapipeline.transform.TransformingReader;
public class ReadAJsonStream {
/* Consumes the following JSON (line breaks added for clarity). Yes, it starts with a new line an two slashes.
// [
{
"id":"358464",
"t":"MSFT",
"e":"NASDAQ",
"l":"31.67",
"l_cur":"31.67",
"s":"2",
"ltt":"4:00PM EDT",
"lt":"Aug 1, 4:00PM EDT",
"c":"-0.17",
"cp":"-0.53",
"ccol":"chr",
"el":"31.69",
"el_cur":"31.69",
"elt":"Aug 1, 7:54PM EDT",
"ec":"+0.02",
"ecp":"0.06",
"eccol":"chg",
"div":"0.23",
"yld":"2.90"
},
{
"id":"419344",
"t":"ORCL",
"e":"NYSE",
"l":"32.75",
"l_cur":"32.75",
"s":"2",
"ltt":"4:00PM EDT",
"lt":"Aug 1, 4:00PM EDT",
"c":"+0.40",
"cp":"1.24",
"ccol":"chg",
"el":"32.70",
"el_cur":"32.70",
"elt":"Aug 1, 7:15PM EDT",
"ec":"-0.05",
"ecp":"-0.15",
"eccol":"chr",
"div":"",
"yld":"1.47"
},
{
"id":"4112",
"t":"ADBE",
"e":"NASDAQ",
"l":"47.70",
"l_cur":"47.70",
"s":"0",
"ltt":"4:00PM EDT",
"lt":"Aug 1, 4:00PM EDT",
"c":"+0.42",
"cp":"0.89",
"ccol":"chg"
}
]
*/
public static void main(String[] args) throws Throwable {
String url = "http://www.google.com/finance/info?client=ig&q=msft,orcl,adbe";
BufferedReader input = new BufferedReader(new InputStreamReader(new URL(url).openStream(), "UTF-8"));
// remove preceding slashes from stream
input.readLine();
input.read();
input.read();
DataReader reader = new JsonReader(input)
.addField("symbol", "//array/object/t")
.addField("exchange", "//array/object/e")
.addField("price", "//array/object/l")
.addField("change", "//array/object/c")
.addRecordBreak("//array/object");
reader = new TransformingReader(reader)
.add(new BasicFieldTransformer("price", "change").stringToDouble())
;
DataWriter writer = new StreamWriter(System.out);
Job.run(reader, writer);
}
}
Code Walkthrough
- A BufferedReader is created to read from the URL
http://www.google.com/finance/info?client=ig&q=msft,orcl,adbe.which points to the JSON stream. - A new JsonReader is created using BufferedReader.
- The fields to be read from the JSON stream are added via the JsonReader.addField method.
- The JsonReader.addRecordBreak method is invoked to demarcate records.
- Transformations are applied to the
priceandchangefields in the input via TransformingReader. - A StreamWriter is created to write to the console.
- Data is transferred from the JSON stream to the console via JobTemplate.DEFAULT.transfer method.
JsonReader
JsonReader is an input reader that can read a JSON stream. It can be created using a Reader object as demonstrated in this code or using a File corresponding to the JSON stream. The main method in this class is the JsonReader.addField method which is used to add the fields to be read from the JSON stream. This method uses the field name and a subset of the XPath 1.0 location paths notation to identify field values. You can selectively add whichever fields you wish to be read from the input JSON stream. Another important method is the JsonReader.addRecordBreak, which tells the reader to return a new record using whatever fields have been assigned. This method is basically used to demarcate records.
Input Data Transformation
A TransformingReader can be used to modify the input data on the fly. In this example, it is created as a wrapper around the JsonReader and modifies the input data in the JSON stream. However, it can be created on top of other readers and can be used to modify other types of input data as well. A TransformingReader basically modifies the input reader using the Transformer objects added to it via the add method.
Transformer
Transformer is an abstract class and does not define much functionality. It has an abstract method called transform which is invoked by the TransformingReader. FieldTransformer is a sub-class of Transformer and overrides the transform method. FieldTransformer has a method called transformField, which is overridden by the BasicFieldTransformer.
BasicFieldTransformer
BasicFieldTransformer which is used in this code is a
sub-class of FieldTransformer. It has several methods like
stringToDouble, nullToValue, etc. These specialized methods adds a new Operation to its collection of operations, then returns the "this" pointer (itself).
In order to perform the data transformation, the TransformingReader
invokes the Transformer.transform method.
This in turn invokes the transformField
method. Thus, these collection of operations are run when the transform->transformField methods are called by the
TransformingReader.
In this example, the
stringToDouble operation
is invoked for the price and change fields which convert these fields from the input JSON stream to double values.
Output stream
12:02:19,378 INFO [main] datapipeline:136 - Record (MODIFIED) {
0:[symbol]:STRING=[MSFT]:String
1:[exchange]:STRING=[NASDAQ]:String
2:[price]:DOUBLE=[41.35]:Double
3:[change]:DOUBLE=[-0.07]:Double
}
12:02:19,417 INFO [main] datapipeline:136 - Record (MODIFIED) {
0:[symbol]:STRING=[ORCL]:String
1:[exchange]:STRING=[NYSE]:String
2:[price]:DOUBLE=[41.13]:Double
3:[change]:DOUBLE=[-0.36]:Double
}
12:02:19,424 INFO [main] datapipeline:136 - Record (MODIFIED) {
0:[symbol]:STRING=[ADBE]:String
1:[exchange]:STRING=[NASDAQ]:String
2:[price]:DOUBLE=[65.48]:Double
3:[change]:DOUBLE=[-0.45]:Double
}
As can be seen from the above output, the "price" and "change" fields are converted to a Double type.
