Read a JSON Stream
This example shows you how to create a JSON stream in Java using the JsonReader
class.
This example also shows how the input data can be modified on the fly using the TransformingReader
and BasicFieldTransformer
classes.
This example can easily be modified to show how to write to a JSON stream.
JSON input
This JSON stream consists of an array of objects.
[ { "id":"358464", "t":"MSFT", "e":"NASDAQ", "l":"31.67", "l_cur":"31.67", "s":"2", "ltt":"4:00PM EDT", "lt":"Aug 1, 4:00PM EDT", "c":"-0.17", "cp":"-0.53", "ccol":"chr", "el":"31.69", "el_cur":"31.69", "elt":"Aug 1, 7:54PM EDT", "ec":"+0.02", "ecp":"0.06", "eccol":"chg", "div":"0.23", "yld":"2.90" }, { "id":"419344", "t":"ORCL", "e":"NYSE", "l":"32.75", "l_cur":"32.75", "s":"2", "ltt":"4:00PM EDT", "lt":"Aug 1, 4:00PM EDT", "c":"+0.40", "cp":"1.24", "ccol":"chg", "el":"32.70", "el_cur":"32.70", "elt":"Aug 1, 7:15PM EDT", "ec":"-0.05", "ecp":"-0.15", "eccol":"chr", "div":"", "yld":"1.47" }, { "id":"4112", "t":"ADBE", "e":"NASDAQ", "l":"47.70", "l_cur":"47.70", "s":"0", "ltt":"4:00PM EDT", "lt":"Aug 1, 4:00PM EDT", "c":"+0.42", "cp":"0.89", "ccol":"chg" } ]
Java Code Listing
The code below demonstrates reading only the highlighted fields from each object in the array.
package com.northconcepts.datapipeline.examples.cookbook; import java.io.BufferedReader; import java.io.InputStreamReader; import java.net.URL; import com.northconcepts.datapipeline.core.DataReader; import com.northconcepts.datapipeline.core.DataWriter; import com.northconcepts.datapipeline.core.StreamWriter; import com.northconcepts.datapipeline.job.Job; import com.northconcepts.datapipeline.json.JsonReader; import com.northconcepts.datapipeline.transform.BasicFieldTransformer; import com.northconcepts.datapipeline.transform.TransformingReader; public class ReadAJsonStream { /* Consumes the following JSON (line breaks added for clarity). Yes, it starts with a new line an two slashes. // [ { "id":"358464", "t":"MSFT", "e":"NASDAQ", "l":"31.67", "l_cur":"31.67", "s":"2", "ltt":"4:00PM EDT", "lt":"Aug 1, 4:00PM EDT", "c":"-0.17", "cp":"-0.53", "ccol":"chr", "el":"31.69", "el_cur":"31.69", "elt":"Aug 1, 7:54PM EDT", "ec":"+0.02", "ecp":"0.06", "eccol":"chg", "div":"0.23", "yld":"2.90" }, { "id":"419344", "t":"ORCL", "e":"NYSE", "l":"32.75", "l_cur":"32.75", "s":"2", "ltt":"4:00PM EDT", "lt":"Aug 1, 4:00PM EDT", "c":"+0.40", "cp":"1.24", "ccol":"chg", "el":"32.70", "el_cur":"32.70", "elt":"Aug 1, 7:15PM EDT", "ec":"-0.05", "ecp":"-0.15", "eccol":"chr", "div":"", "yld":"1.47" }, { "id":"4112", "t":"ADBE", "e":"NASDAQ", "l":"47.70", "l_cur":"47.70", "s":"0", "ltt":"4:00PM EDT", "lt":"Aug 1, 4:00PM EDT", "c":"+0.42", "cp":"0.89", "ccol":"chg" } ] */ public static void main(String[] args) throws Throwable { String url = "http://www.google.com/finance/info?client=ig&q=msft,orcl,adbe"; BufferedReader input = new BufferedReader(new InputStreamReader(new URL(url).openStream(), "UTF-8")); // remove preceding slashes from stream input.readLine(); input.read(); input.read(); DataReader reader = new JsonReader(input) .addField("symbol", "//array/object/t") .addField("exchange", "//array/object/e") .addField("price", "//array/object/l") .addField("change", "//array/object/c") .addRecordBreak("//array/object"); reader = new TransformingReader(reader) .add(new BasicFieldTransformer("price", "change").stringToDouble()) ; DataWriter writer = new StreamWriter(System.out); Job.run(reader, writer); } }
Code Walkthrough
- A
BufferedReader
is created to read from the URLhttp://www.google.com/finance/info?client=ig&q=msft,orcl,adbe.
which points to the JSON stream. - A new
JsonReader
is created usingBufferedReader
. - The fields to be read from the JSON stream are added via the
JsonReader.addField()
method. - The
JsonReader.addRecordBreak()
method is invoked to demarcate records. - Transformations are applied to the
price
andchange
fields in the input viaTransformingReader
. - A
StreamWriter
is created to write to the console. - Data is transferred from the JSON stream to the console via
JobTemplate.DEFAULT.transfer()
method.
JsonReader
JsonReader
is an input reader that can read a JSON stream. It can be created using a Reader
object as demonstrated in this code or using a File
corresponding to the JSON stream. The main method in this class is the JsonReader.addField()
method which is used to add the fields to be read from the JSON stream. This method uses the field name and a subset of the XPath 1.0 location paths notation to identify field values. You can selectively add whichever fields you wish to be read from the input JSON stream. Another important method is the JsonReader.addRecordBreak()
, which tells the reader to return a new record using whatever fields have been assigned. This method is basically used to demarcate records.
Input Data Transformation
A TransformingReader
can be used to modify the input data on the fly. In this example, it is created as a wrapper around the JsonReader
and modifies the input data in the JSON stream. However, it can be created on top of other readers and can be used to modify other types of input data as well. A TransformingReader
basically modifies the input reader using the Transformer
objects added to it via the add()
method.
Transformer
Transformer
is an abstract class and does not define much functionality. It has an abstract method called transform()
which is invoked by the TransformingReader
. FieldTransformer
is a sub-class of Transformer
and overrides the transform()
method. FieldTransformer
has a method called transformField()
, which is overridden by the BasicFieldTransformer
.
BasicFieldTransformer
BasicFieldTransformer
which is used in this code is a sub-class of FieldTransformer
. It has several methods like stringToDouble
, nullToValue
, etc. These specialized methods add a new operation to its collection of operations, then return the this
pointer (itself).
In order to perform the data transformation, the TransformingReader
invokes the Transformer.transform()
method. This in turn invokes the transformField()
method. Thus, these collection of operations are run when the transform->transformField
methods are called by the TransformingReader
.
In this example, the stringToDouble()
operation is invoked for the price
and change
fields which convert these fields from the input JSON stream to double values.
Output Stream
As can be seen from the above output, the price
and change
fields are converted to a Double
type.
12:02:19,378 INFO [main] datapipeline:136 - Record (MODIFIED) { 0:[symbol]:STRING=[MSFT]:String 1:[exchange]:STRING=[NASDAQ]:String 2:[price]:DOUBLE=[41.35]:Double 3:[change]:DOUBLE=[-0.07]:Double } 12:02:19,417 INFO [main] datapipeline:136 - Record (MODIFIED) { 0:[symbol]:STRING=[ORCL]:String 1:[exchange]:STRING=[NYSE]:String 2:[price]:DOUBLE=[41.13]:Double 3:[change]:DOUBLE=[-0.36]:Double } 12:02:19,424 INFO [main] datapipeline:136 - Record (MODIFIED) { 0:[symbol]:STRING=[ADBE]:String 1:[exchange]:STRING=[NASDAQ]:String 2:[price]:DOUBLE=[65.48]:Double 3:[change]:DOUBLE=[-0.45]:Double }