Build DataMappingPipeline Declaratively from Json

This example shows how you can use Data Pipeline to process data through a pipeline, applying a source entity, mapping, and target entity from an input JSON file to achieve the desired output. It provides a structured and customizable approach to data transformation, allowing users to define and execute complex data processing workflows.

By processing data through the pipeline, users can generate refined and structured datasets suitable for reporting and analytics purposes. The library enables the extraction, transformation, and loading of data into target entities that are optimized for reporting and analysis, supporting data-driven decision-making.

Input JSON File

View The Full File

{
  "name":null,
  "description":null,
  "input":{
    "__class__":"com.northconcepts.datapipeline.foundations.pipeline.input.CsvPipelineInput",
    "fieldSeparator":",",
    "startingQuote":"\"",
    "endingQuote":"\"",
    "lineSeparators":"\\n,\\r\\n,\\r",
    "allowMultiLineText":false,
    "allowQuoteInField":false,
    "trimFields":true,
    "skipEmptyRows":false,
    "charset":"UTF-8",
    "fieldNamesInFirstRow":true,
    "fileSource":{
      "__class__":"com.northconcepts.datapipeline.foundations.file.LocalFileSource",
      "name":null,
      "path":"example/data/input/call-center-inbound-call.csv"
    }
  },
  "sourceEntity":{
    "name":"Raw",
    "description":null,
    "allowExtraFieldsInValidation":true,
    "allowExtraFieldsInMapping":true,
    "addMissingOptionalFields":false,
...

Input CSV File

View The Full File

event_type,id,agent_id,phone_number,start_time,end_time,disposition
STARTED,1,7,(437) 689-5268,2016-03-04 22:39,,
ENDED,1,7,(437) 689-5268,2016-03-04 22:39,2016-03-04 22:39,PRODUCT_QUESTION
STARTED,2,19,(343) 8314-0603,2016-03-04 22:39,,
...

Java Code Listing

package com.northconcepts.datapipeline.foundations.examples.pipeline;


import java.io.FileInputStream;

import com.northconcepts.datapipeline.foundations.pipeline.DataMappingPipeline;

public class BuildDataMappingPipelineDeclarativelyFromJson {

    public static void main(String[] args) throws Throwable {
        DataMappingPipeline pipeline = new DataMappingPipeline().fromJson(new FileInputStream("example/data/input/pipeline/datamappingpipeline.json"));
        pipeline.run();
    }

}

Code Walkthrough

1. DataMappingPipeline instance is created with rules imported from the input file datamappingpipeline.json.
2. The pipeline is run.

This functionality operates in a similar manner as the example involving XML files. If you require further details about JSON content and its features, you can refer to that specific example.

Output XLSX File

Event	Call ID	Agent ID	Caller Number	Call Start Time	Call End Time	Disposition
STARTED	1	7	(437) 689-5268	3/4/2016 22:39	3/4/2016 22:39	UNKNOWN
ENDED	1	7	(437) 689-5268	3/4/2016 22:39	3/4/2016 22:39	PRODUCT_QUESTION
STARTED	2	19	(343) 8314-0603	3/4/2016 22:39	3/4/2016 22:39	UNKNOWN
...