Create a JsonPipelineInput Declaratively from JSON

This example is designed to facilitate data transformations by leveraging JSON field definitions and applying them to another JSON file. By taking a JSON file containing field definitions and conversion rules, the library creates a JsonPipelineInput object, which allows users to declaratively define the conversion process.

Real-life use cases for this library can be found in data integration and data mapping scenarios. For instance, in a data migration project, where there is a need to transform data from one JSON format to another based on specific rules defined in a JSON file, this library provides a convenient solution. It allows users to define complex mapping rules, data type conversions, and field transformations in a structured and declarative manner, reducing manual effort and ensuring consistent and accurate data transformations. By bridging the gap between JSON field definitions and JSON data, this example enables seamless and efficient data processing and integration.

Input Files

jsonpipelineinput.json

{
  "__class__": "com.northconcepts.datapipeline.foundations.pipeline.input.JsonPipelineInput",
  "fileSource": {
    "__class__": "com.northconcepts.datapipeline.foundations.file.LocalFileSource",
    "name": "input-json",
    "path": "example/data/input/pipeline/json-input.json"
  },
  "duplicateFieldPolicy": "USE_LAST_VALUE",
  "recordBreaks": [
    "//array/object"
  ],
  "fields": [
    {
      "name": "UID",
      "locationPath": "/object/UID",
      "cascadeValues": true,
      "cascadeResetLocationPath": null
    },
    {
      "name": "type",
      "locationPath": "//favorites/*/node()",
      "cascadeValues": false,
      "cascadeResetLocationPath": null
    },
    {
      "name": "favorite-name",
      "locationPath": "//favorites//name",
      "cascadeValues": false,
      "cascadeResetLocationPath": null
    },
    {
      "name": "favorite-id",
      "locationPath": "//favorites//id",
      "cascadeValues": true,
      "cascadeResetLocationPath": "//favorites/books"
    },
    {
      "name": "favorite-category",
      "locationPath": "//favorites//category",
      "cascadeValues": false,
      "cascadeResetLocationPath": null
    }
  ],
  "useBigDecimal": false
}

json-input.json

{
   "UID":"11e03ad7f20543f88d598ab7f31b5b3f",   
   "profile":{
      "favorites":{
         "movies":[
            {
               "name":"Food Inc",
               "id":73304591159.1234567,
               "category":"Movie"
            },
			{
               "name":"Forrest gump",
               "id":25904594452.1234567,
               "category":"Movie"
            }
         ],
         "books":[
            {
               "name":"Robert Whitaker",
               "id":168314056582399.1234567,
               "category":"Author"
            },
            {
               "name":"Sylvia Plath",
               "category":"Author"
            },
            {
               "name":"John Steinbeck",
               "category":"Author"
            }
         ],
         "music":[
            {
               "name":"The Devil Makes Three",
               "id":8364692300.1234567,
               "category":"Musician/band"
            },
            {
               "name":"Scorpions",
               "category":"Hardrock band"
            }
         ]
         
      }
   }	
}

Java Code

package com.northconcepts.datapipeline.foundations.examples.pipeline;


import com.northconcepts.datapipeline.core.StreamWriter;
import com.northconcepts.datapipeline.foundations.pipeline.Pipeline;
import com.northconcepts.datapipeline.foundations.pipeline.input.JsonPipelineInput;
import com.northconcepts.datapipeline.foundations.pipeline.output.DataWriterPipelineOutput;

import java.io.FileInputStream;

public class CreateAJsonPipelineInputDeclarativelyFromJson {

    public static void main(String[] args) throws Throwable {
        Pipeline pipeline = new Pipeline();

        JsonPipelineInput input = (JsonPipelineInput) new JsonPipelineInput()
            .fromJson(new FileInputStream("example/data/input/pipeline/jsonpipelineinput.json"));

        pipeline.setInput(input);
        pipeline.setOutput(new DataWriterPipelineOutput(() -> StreamWriter.newSystemOutWriter()));

        pipeline.run();
    }

}

Code Walkthrough

1. Pipeline object is created.
2. JsonPipelineInput instance is generated using the provided JSON file jsonpipelineinput.json. This object is then incorporated into a Pipeline as the designated input.
3. Subsequently, a new StreamWriter is introduced as the output, facilitating the printing of the Pipeline's results to the console.
4. Within the JSON definition, there is a reference to another JSON file (json-input.json) that serves as a file-source, containing all the relevant data.

Console Output

-----------------------------------------------
0 - Record (MODIFIED) {
    0:[UID]:STRING=[11e03ad7f20543f88d598ab7f31b5b3f]:String
    1:[type]:STRING=[movies]:String
    2:[favorite-name]:STRING=[Food Inc]:String
    3:[favorite-id]:DOUBLE=[7.330459115912346E10]:Double
    4:[favorite-category]:STRING=[Movie]:String
}

-----------------------------------------------
1 - Record (MODIFIED) {
    0:[UID]:STRING=[11e03ad7f20543f88d598ab7f31b5b3f]:String
    1:[type]:STRING=[movies]:String
    2:[favorite-name]:STRING=[Forrest gump]:String
    3:[favorite-id]:DOUBLE=[2.5904594452123455E10]:Double
    4:[favorite-category]:STRING=[Movie]:String
}

-----------------------------------------------
2 - Record (MODIFIED) {
    0:[UID]:STRING=[11e03ad7f20543f88d598ab7f31b5b3f]:String
    1:[type]:STRING=[books]:String
    2:[favorite-name]:STRING=[Robert Whitaker]:String
    3:[favorite-id]:DOUBLE=[1.6831405658239912E14]:Double
    4:[favorite-category]:STRING=[Author]:String
}

-----------------------------------------------
3 - Record (MODIFIED) {
    0:[UID]:STRING=[11e03ad7f20543f88d598ab7f31b5b3f]:String
    1:[type]:STRING=[books]:String
    2:[favorite-name]:STRING=[Sylvia Plath]:String
    3:[favorite-id]:DOUBLE=[1.6831405658239912E14]:Double
    4:[favorite-category]:STRING=[Author]:String
}

-----------------------------------------------
4 - Record (MODIFIED) {
    0:[UID]:STRING=[11e03ad7f20543f88d598ab7f31b5b3f]:String
    1:[type]:STRING=[books]:String
    2:[favorite-name]:STRING=[John Steinbeck]:String
    3:[favorite-id]:DOUBLE=[1.6831405658239912E14]:Double
    4:[favorite-category]:STRING=[Author]:String
}

-----------------------------------------------
5 - Record (MODIFIED) {
    0:[UID]:STRING=[11e03ad7f20543f88d598ab7f31b5b3f]:String
    1:[type]:STRING=[music]:String
    2:[favorite-name]:STRING=[The Devil Makes Three]:String
    3:[favorite-id]:DOUBLE=[8.364692300123457E9]:Double
    4:[favorite-category]:STRING=[Musician/band]:String
}

-----------------------------------------------
6 - Record (MODIFIED) {
    0:[UID]:STRING=[11e03ad7f20543f88d598ab7f31b5b3f]:String
    1:[type]:STRING=[music]:String
    2:[favorite-name]:STRING=[Scorpions]:String
    3:[favorite-id]:DOUBLE=[8.364692300123457E9]:Double
    4:[favorite-category]:STRING=[Hardrock band]:String
}

-----------------------------------------------
7 records