Declaratively Map XML Files

This example shows how you can use Data Pipeline to map data from XML files using rules defined in another XML file. It enables users to define specific mapping rules and transformations for XML data, allowing for efficient data integration and manipulation.

This example can be used to map XML data to a database schema, allowing for the efficient import and storage of XML data into a relational database. This facilitates the integration of XML-based data with existing database systems, making it easier to query and analyze the data using SQL.

Input XML Files

bookstore.xml

<?xml version="1.0" encoding="ISO-8859-1"?>

<bookstore>

<book>
  <title lang="eng">Harry Potter</title>
  <price>29.99</price>
</book>

<book>
  <title lang="eng">Learning XML</title>
  <price>39.95</price>
</book>

</bookstore>

book-mapping.xml

<?xml version="1.0"?>
<data-mapping>
  <field-mappings>
    <field-mapping fieldName="title2" sourceExpression="${source.title.$text}"/>
    <field-mapping fieldName="lang2" sourceExpression="${source.title.@lang}"/>
    <field-mapping fieldName="price2" sourceExpression="${source.price}"/>
  </field-mappings>
</data-mapping>

Java Code Listing

package com.northconcepts.datapipeline.foundations.examples.datamapping;

import java.io.File;
import java.io.FileInputStream;

import com.northconcepts.datapipeline.core.DataReader;
import com.northconcepts.datapipeline.core.DataWriter;
import com.northconcepts.datapipeline.core.StreamWriter;
import com.northconcepts.datapipeline.foundations.datamapping.DataMapping;
import com.northconcepts.datapipeline.foundations.datamapping.DataMappingReader;
import com.northconcepts.datapipeline.job.Job;
import com.northconcepts.datapipeline.xml.XmlRecordReader;

public class DeclarativelyMapXmlFiles {
    
    public static void main(String[] args) throws Throwable {
//        DataReader reader = new XmlReader(new File("example/data/input/bookstore.xml"))
//                .addField("title", "//book/title/text()")
//                .addField("language", "//book/title/@lang")
//                .addField("price", "//book/price/text()")
//                .addRecordBreak("//book");
        
        DataReader reader = new XmlRecordReader(new File("example/data/input/bookstore.xml"))
                .addRecordBreak("//book");

        reader = new DataMappingReader(reader, new DataMapping()
                .fromXml(new FileInputStream(new File("example/data/input/datamapping/book-mapping.xml"))));

        DataWriter writer = StreamWriter.newSystemOutWriter();
        
        Job job = Job.run(reader, writer);
        System.out.println("Records Transferred: " + job.getRecordsTransferred());
        System.out.println("Running Time: " + job.getRunningTimeAsString());
        
    }

}

Code Walkthrough

1. XmlRecordReader is created corresponding to the input file bookstore.xml.
2. DataMapping is created to enable the field mapping rules. It imports rules from the XML file book-mapping.xml.
3. DataMappingReader instance is created to apply mapping for the input. It takes data from the reader and applies the mapping mentioned in the previous step.
4. Data is transferred from the reader to StreamWriter.newSystemOutWriter() via Job.run() method.
5. The number of records and running time are also printed in the console.

Console output

-----------------------------------------------
0 - Record (MODIFIED) {
    0:[title2]:STRING=[Harry Potter]:String
    1:[lang2]:STRING=[eng]:String
    2:[price2]:STRING=[29.99]:String
}

-----------------------------------------------
1 - Record (MODIFIED) {
    0:[title2]:STRING=[Learning XML]:String
    1:[lang2]:STRING=[eng]:String
    2:[price2]:STRING=[39.95]:String
}

-----------------------------------------------
2 records

Records Transferred: 2
Running Time: 34 Milliseconds