Read a JSON Stream

This example shows you how to create a JSON stream in Java using the JsonReader class.

This example also shows how the input data can be modified on the fly using the TransformingReader and BasicFieldTransformer classes.

This example can easily be modified to show how to write to a JSON stream.

JSON input

This JSON stream consists of an array of objects.

[
   {
      "id":"358464",
      "t":"MSFT",
      "e":"NASDAQ",
      "l":"31.67",
      "l_cur":"31.67",
      "s":"2",
      "ltt":"4:00PM EDT",
      "lt":"Aug 1, 4:00PM EDT",
      "c":"-0.17",
      "cp":"-0.53",
      "ccol":"chr",
      "el":"31.69",
      "el_cur":"31.69",
      "elt":"Aug 1, 7:54PM EDT",
      "ec":"+0.02",
      "ecp":"0.06",
      "eccol":"chg",
      "div":"0.23",
      "yld":"2.90"
   },
   {
      "id":"419344",
      "t":"ORCL",
      "e":"NYSE",
      "l":"32.75",
      "l_cur":"32.75",
      "s":"2",
      "ltt":"4:00PM EDT",
      "lt":"Aug 1, 4:00PM EDT",
      "c":"+0.40",
      "cp":"1.24",
      "ccol":"chg",
      "el":"32.70",
      "el_cur":"32.70",
      "elt":"Aug 1, 7:15PM EDT",
      "ec":"-0.05",
      "ecp":"-0.15",
      "eccol":"chr",
      "div":"",
      "yld":"1.47"
   },
   {
      "id":"4112",
      "t":"ADBE",
      "e":"NASDAQ",
      "l":"47.70",
      "l_cur":"47.70",
      "s":"0",
      "ltt":"4:00PM EDT",
      "lt":"Aug 1, 4:00PM EDT",
      "c":"+0.42",
      "cp":"0.89",
      "ccol":"chg"
   }
]

Java Code Listing

The code below demonstrates reading only the highlighted fields from each object in the array.

package com.northconcepts.datapipeline.examples.cookbook;

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.URL;

import com.northconcepts.datapipeline.core.DataReader;
import com.northconcepts.datapipeline.core.DataWriter;
import com.northconcepts.datapipeline.core.StreamWriter;
import com.northconcepts.datapipeline.job.Job;
import com.northconcepts.datapipeline.json.JsonReader;
import com.northconcepts.datapipeline.transform.BasicFieldTransformer;
import com.northconcepts.datapipeline.transform.TransformingReader;

public class ReadAJsonStream {

/*   Consumes the following JSON (line breaks added for clarity).  Yes, it starts with a new line an two slashes.    

// [
   {
      "id":"358464",
      "t":"MSFT",
      "e":"NASDAQ",
      "l":"31.67",
      "l_cur":"31.67",
      "s":"2",
      "ltt":"4:00PM EDT",
      "lt":"Aug 1, 4:00PM EDT",
      "c":"-0.17",
      "cp":"-0.53",
      "ccol":"chr",
      "el":"31.69",
      "el_cur":"31.69",
      "elt":"Aug 1, 7:54PM EDT",
      "ec":"+0.02",
      "ecp":"0.06",
      "eccol":"chg",
      "div":"0.23",
      "yld":"2.90"
   },
   {
      "id":"419344",
      "t":"ORCL",
      "e":"NYSE",
      "l":"32.75",
      "l_cur":"32.75",
      "s":"2",
      "ltt":"4:00PM EDT",
      "lt":"Aug 1, 4:00PM EDT",
      "c":"+0.40",
      "cp":"1.24",
      "ccol":"chg",
      "el":"32.70",
      "el_cur":"32.70",
      "elt":"Aug 1, 7:15PM EDT",
      "ec":"-0.05",
      "ecp":"-0.15",
      "eccol":"chr",
      "div":"",
      "yld":"1.47"
   },
   {
      "id":"4112",
      "t":"ADBE",
      "e":"NASDAQ",
      "l":"47.70",
      "l_cur":"47.70",
      "s":"0",
      "ltt":"4:00PM EDT",
      "lt":"Aug 1, 4:00PM EDT",
      "c":"+0.42",
      "cp":"0.89",
      "ccol":"chg"
   }
]

*/
    
    public static void main(String[] args) throws Throwable {
    	String url = "http://www.google.com/finance/info?client=ig&q=msft,orcl,adbe";

    	BufferedReader input = new BufferedReader(new InputStreamReader(new URL(url).openStream(), "UTF-8"));
    	
    	// remove preceding slashes from stream
    	input.readLine();
    	input.read();
    	input.read();
    	

    	DataReader reader = new JsonReader(input)
    	    .addField("symbol", "//array/object/t")
    	    .addField("exchange", "//array/object/e")
    	    .addField("price", "//array/object/l")
    	    .addField("change", "//array/object/c")
    	    .addRecordBreak("//array/object");
        
        reader = new TransformingReader(reader)
        	.add(new BasicFieldTransformer("price", "change").stringToDouble())
        	;
    	
        DataWriter writer = new  StreamWriter(System.out);
   
        Job.run(reader, writer);
    }

}

Code Walkthrough

1. A BufferedReader is created to read from the URL http://www.google.com/finance/info?client=ig&q=msft,orcl,adbe. which points to the JSON stream.
2. A new JsonReader is created using BufferedReader.
3. The fields to be read from the JSON stream are added via the JsonReader.addField() method.
4. The JsonReader.addRecordBreak() method is invoked to demarcate records.
5. Transformations are applied to the price and change fields in the input via TransformingReader.
6. A StreamWriter is created to write to the console.
7. Data is transferred from the JSON stream to the console via JobTemplate.DEFAULT.transfer() method.

JsonReader

JsonReader is an input reader that can read a JSON stream. It can be created using a Reader object as demonstrated in this code or using a File corresponding to the JSON stream. The main method in this class is the JsonReader.addField() method which is used to add the fields to be read from the JSON stream. This method uses the field name and a subset of the XPath 1.0 location paths notation to identify field values. You can selectively add whichever fields you wish to be read from the input JSON stream. Another important method is the JsonReader.addRecordBreak(), which tells the reader to return a new record using whatever fields have been assigned. This method is basically used to demarcate records.

Input Data Transformation

A TransformingReader can be used to modify the input data on the fly. In this example, it is created as a wrapper around the JsonReader and modifies the input data in the JSON stream. However, it can be created on top of other readers and can be used to modify other types of input data as well. A TransformingReader basically modifies the input reader using the Transformer objects added to it via the add() method.

Transformer

Transformer is an abstract class and does not define much functionality. It has an abstract method called transform() which is invoked by the TransformingReader. FieldTransformer is a sub-class of Transformer and overrides the transform() method. FieldTransformer has a method called transformField(), which is overridden by the BasicFieldTransformer.

BasicFieldTransformer

BasicFieldTransformer which is used in this code is a sub-class of FieldTransformer. It has several methods like stringToDouble, nullToValue, etc. These specialized methods add a new operation to its collection of operations, then return the this pointer (itself).

In order to perform the data transformation, the TransformingReader invokes the Transformer.transform() method. This in turn invokes the transformField() method. Thus, these collection of operations are run when the transform->transformField methods are called by the TransformingReader.

In this example, the stringToDouble() operation is invoked for the price and change fields which convert these fields from the input JSON stream to double values.

Output Stream

As can be seen from the above output, the price and change fields are converted to a Double type.

12:02:19,378  INFO [main] datapipeline:136 - Record (MODIFIED) {
    0:[symbol]:STRING=[MSFT]:String
    1:[exchange]:STRING=[NASDAQ]:String
    2:[price]:DOUBLE=[41.35]:Double
    3:[change]:DOUBLE=[-0.07]:Double
}

12:02:19,417  INFO [main] datapipeline:136 - Record (MODIFIED) {
    0:[symbol]:STRING=[ORCL]:String
    1:[exchange]:STRING=[NYSE]:String
    2:[price]:DOUBLE=[41.13]:Double
    3:[change]:DOUBLE=[-0.36]:Double
}

12:02:19,424  INFO [main] datapipeline:136 - Record (MODIFIED) {
    0:[symbol]:STRING=[ADBE]:String
    1:[exchange]:STRING=[NASDAQ]:String
    2:[price]:DOUBLE=[65.48]:Double
    3:[change]:DOUBLE=[-0.45]:Double
}