Map Data with Rule Based Validation

This example shows how you can use Data Pipeline to apply rule-based validations when mapping data, ensuring data integrity and compliance with predefined data schemas. It enables users to define and enforce data validation rules during the mapping process, providing robust data quality assurance.

The example can be used to validate data consistency and accuracy when migrating or synchronizing data between different systems or databases. Users can define rule-based validations to ensure that the mapped data maintains consistency and integrity across the migration or synchronization process.

Java Code Listing

package com.northconcepts.datapipeline.foundations.examples.datamapping;

import com.northconcepts.datapipeline.core.Record;
import com.northconcepts.datapipeline.filter.FieldCount;
import com.northconcepts.datapipeline.filter.FilterExpression;
import com.northconcepts.datapipeline.foundations.datamapping.DataMapping;
import com.northconcepts.datapipeline.foundations.datamapping.DataMappingResult;
import com.northconcepts.datapipeline.foundations.schema.EntityDef;
import com.northconcepts.datapipeline.foundations.schema.SchemaDef;
import com.northconcepts.datapipeline.internal.expression.DefaultExpressionContext;

public class MapDataWithRuleBasedValidation {

    public static void main(String[] args) {
        SchemaDef schema = new SchemaDef()
                .addEntity(new EntityDef("customer")
                        .addValidation(new FilterExpression("age > 12 && age < 13"))
                        .addValidation(new FieldCount(4)));
        
        EntityDef customerEntity = schema.getEntity("customer"); 

        
        DefaultExpressionContext input = new DefaultExpressionContext();
        input.setValue("fname", "John");
        input.setValue("lname", "Smith");
        input.setValue("age", 15);

        DataMapping mapping = new DataMapping()
                .setTargetValidationEntity(customerEntity)  // set optional entity definition to validate against
                .addFieldMapping("name", "source.fname + ' ' + source.lname")
                .addFieldMapping("age", "source.age");
        
        DataMappingResult result = mapping.map(input);
        Record target = result.getTarget();

        System.out.println("target = " + target);
        
        result.logExceptions();
        result.logValidationErrors();  // validation errors are separate from exceptions

    }

}

Code Walkthrough

1. A new schema with the "customer" entity is created.
2. The following validation rules are set for the Customer entity:
   1. age field should be between 12 and 13;
   2. the total number of fields should be equal to 4.
3. DefaultExpressionContext instance is used to hold the source/input data with the key-value pattern. Values are entered with setValue() method.
4. DataMapping is created to enable the field mapping rules. 
5. The Customer entity is set as a target validation entity. It means that data should meet the validation rules mentioned in step 2.
6. addFieldMapping() method accepts two String arguments:
   1. targetFieldName - the name of the field in the target record.
   2. sourceExpression - a custom expression to hold the formatting logic for a target field.
7. Fields in the source and target records can be accessed with the dot (.) operator: source.fname.
8. DataMappingResult instance is created to apply mapping for the input.
9. The target record is obtained from result object and printed in the console with System.out.println().
10. The exceptions and validation errors from the result, if exist, are also logged.

Console output

target = Record (MODIFIED) {
    0:[name]:STRING=[John Smith]:String
    1:[age]:INT=[15]:Integer
}

 --> Record failed validation rule, expected: record satisfies expression: age > 12 && age < 13
 --> Record failed validation rule, expected: record has 4 fields