1. Home
  2. Documentation
  3. Examples
  4. Validate Data using Entity Inheritance

Validate Data using Entity Inheritance

Updated: Aug 24, 2023
View on GitHub
DataPipeline Foundations Validate Schema

This example shows how to validate records based on inherited entity definitions. It allows you to define validation rules and definitions at various levels of inheritance, enabling the validation of record fields based on rules specified in both the parent and ancestor entities. This allows for comprehensive and hierarchical validation, ensuring data integrity and consistency across multiple levels of the record hierarchy.

This example of DataPipeline can be used during data modeling and transformation tasks where records have complex hierarchical structures. You can define validation rules in parent and ancestor entities to validate specific fields or relationships within the hierarchy. This ensures that the transformed or modeled data conforms to the desired structure and meets the defined validation criteria.

 

Java Code

package com.northconcepts.datapipeline.foundations.examples.schema;

import java.math.BigDecimal;
import java.time.LocalDateTime;

import com.northconcepts.datapipeline.core.FieldType;
import com.northconcepts.datapipeline.core.Record;
import com.northconcepts.datapipeline.foundations.schema.EntityDef;
import com.northconcepts.datapipeline.foundations.schema.NumericFieldDef;
import com.northconcepts.datapipeline.foundations.schema.SchemaDef;
import com.northconcepts.datapipeline.foundations.schema.TemporalFieldDef;
import com.northconcepts.datapipeline.foundations.schema.TextFieldDef;
import com.northconcepts.datapipeline.foundations.schema.ValidationMessage;
import com.northconcepts.datapipeline.foundations.schema.ValidationResult;

public class ValidateDataUsingEntityInheritance {

    public static void main(String[] args) {
        SchemaDef schema = new SchemaDef("human_resources");

        schema.addEntity(new EntityDef().setName("auditable")
                .addField(new TextFieldDef("created_by", FieldType.STRING).setRequired(true))
                .addField(new TemporalFieldDef("created_on", FieldType.DATETIME).setRequired(true))
                .addField(new TextFieldDef("modified_by", FieldType.STRING))
                .addField(new TemporalFieldDef("modified_on", FieldType.DATETIME)));

        schema.addEntity(new EntityDef().setName("user")
                .setSuperEntityName("auditable")
                .addField(new NumericFieldDef("id", FieldType.INT).setRequired(true))
                .addField(new TextFieldDef("email", FieldType.STRING))
                .addField(new TemporalFieldDef("created", FieldType.DATETIME)));

        schema.addEntity(new EntityDef().setName("employee")
                .setSuperEntityName("user")
                .addField(new NumericFieldDef("id", FieldType.LONG))
                .addField(new TextFieldDef("email", FieldType.STRING).setMaximumLength(4096).setRequired(true))
                .addField(new NumericFieldDef("salary", FieldType.BIG_DECIMAL).setMaximum(10_000_000))
                .addField(new TextFieldDef("department", FieldType.STRING)));

        Record record = new Record();
        record.addField("id", 123456L);
        record.addField("email", "john_doe@example.com");
        record.addField("salary", BigDecimal.valueOf(200000L));
        record.addField("department", "IT");
        record.addField("budget", 100_000);
        record.addField("created_by", null); // Required field
        record.addField("created_on", LocalDateTime.of(2022, 8, 19, 15, 18));
        record.addField("created", "dummy"); // wrong value type, it should be DATETIME.
        record.addField("modified_by", "test_user");

        System.out.println("==========================Input Record====================================");
        System.out.println(record);
        System.out.println("=========================================================================================");

        ValidationResult validationResult = schema.getEntity("employee").validateRecord(record);
        System.out.println(validationResult);

        System.out.println("=========================================================================================");
        for (int i = 0; i < validationResult.getErrors().size(); i++) {
            ValidationMessage validationMessage = validationResult.getErrors().get(i);
            System.out.println(i + " - " + validationMessage.getMessage());
        }
    }
}

 

Code Walkthrough

  1. SchemaDef instance is created.
  2. A new entity called auditable is defined and four fields are added to it: created_by, created_on, modified_by, modified_on.
  3. Then, three fields with corresponding data types are assigned to another new user entity definition. This entity is the child entity of auditable.
  4. A child entity of user is created with the name employee. In addition to its parent entity fields, this entity holds four additional fields with some constraints such as maximum value and length, optional or mandatory.
  5. Next, a new Record instance is created and filled with data for all fields of the corresponding employee entity. This record is printed in the console.
  6. The record instance is then validated based on the employee entity definition and the result is stored in ValidationResult.
  7. Finally, validationResult and each of its validation messages are printed in the console.

 

Console Output

==========================Input Record====================================
Record (MODIFIED) {
    0:[id]:LONG=[123456]:Long
    1:[email]:STRING=[john_doe@example.com]:String
    2:[salary]:BIG_DECIMAL=[200000]:BigDecimal
    3:[department]:STRING=[IT]:String
    4:[budget]:INT=[100000]:Integer
    5:[created_by]:UNDEFINED=[null]
    6:[created_on]:DATETIME=[Fri Aug 19 15:18:00 UZT 2022]:Date
    7:[created]:STRING=[dummy]:String
    8:[modified_by]:STRING=[test_user]:String
}

=========================================================================================
{
  "errors" : [ {
    "entity" : {
      "addMissingOptionalFields" : false,
      "allowExtraFieldsInMapping" : true,
      "allowExtraFieldsInValidation" : true,
      "attributes" : { },
      "fields" : [ {
        "allowBlank" : true,
        "array" : false,
        "attributes" : { },
        "entityQualifiedName" : "auditable.created_by",
        "limitToAllowedValues" : false,
        "name" : "created_by",
        "primaryKey" : false,
        "qualifiedName" : "human_resources.auditable.created_by",
        "required" : true,
        "strictArrays" : true,
        "tags" : { },
        "type" : "STRING"
      }, {
        "array" : false,
        "attributes" : { },
        "entityQualifiedName" : "auditable.created_on",
        "lenientPattern" : true,
        "limitToAllowedValues" : false,
        "name" : "created_on",
        "primaryKey" : false,
        "qualifiedName" : "human_resources.auditable.created_on",
        "required" : true,
        "strictArrays" : true,
        "tags" : { },
        "type" : "DATETIME"
      }, {
        "allowBlank" : true,
        "array" : false,
        "attributes" : { },
        "entityQualifiedName" : "auditable.modified_by",
        "limitToAllowedValues" : false,
        "name" : "modified_by",
        "primaryKey" : false,
        "qualifiedName" : "human_resources.auditable.modified_by",
        "required" : false,
        "strictArrays" : true,
        "tags" : { },
        "type" : "STRING"
      }, {
        "array" : false,
        "attributes" : { },
        "entityQualifiedName" : "auditable.modified_on",
        "lenientPattern" : true,
        "limitToAllowedValues" : false,
        "name" : "modified_on",
        "primaryKey" : false,
        "qualifiedName" : "human_resources.auditable.modified_on",
        "required" : false,
        "strictArrays" : true,
        "tags" : { },
        "type" : "DATETIME"
      } ],
      "name" : "auditable",
      "qualifiedName" : "human_resources.auditable",
      "tags" : { }
    },
    "field" : {
      "allowBlank" : true,
      "array" : false,
      "attributes" : { },
      "entityQualifiedName" : "auditable.created_by",
      "limitToAllowedValues" : false,
      "name" : "created_by",
      "primaryKey" : false,
      "qualifiedName" : "human_resources.auditable.created_by",
      "required" : true,
      "strictArrays" : true,
      "tags" : { },
      "type" : "STRING"
    },
    "message" : "Required created_by is null"
  }, {
    "entity" : {
      "addMissingOptionalFields" : false,
      "allowExtraFieldsInMapping" : true,
      "allowExtraFieldsInValidation" : true,
      "attributes" : { },
      "fields" : [ {
        "array" : false,
        "attributes" : { },
        "entityQualifiedName" : "user.id",
        "limitToAllowedValues" : false,
        "name" : "id",
        "primaryKey" : false,
        "qualifiedName" : "human_resources.user.id",
        "required" : true,
        "strictArrays" : true,
        "tags" : { },
        "type" : "INT"
      }, {
        "allowBlank" : true,
        "array" : false,
        "attributes" : { },
        "entityQualifiedName" : "user.email",
        "limitToAllowedValues" : false,
        "name" : "email",
        "primaryKey" : false,
        "qualifiedName" : "human_resources.user.email",
        "required" : false,
        "strictArrays" : true,
        "tags" : { },
        "type" : "STRING"
      }, {
        "array" : false,
        "attributes" : { },
        "entityQualifiedName" : "user.created",
        "lenientPattern" : true,
        "limitToAllowedValues" : false,
        "name" : "created",
        "primaryKey" : false,
        "qualifiedName" : "human_resources.user.created",
        "required" : false,
        "strictArrays" : true,
        "tags" : { },
        "type" : "DATETIME"
      } ],
      "name" : "user",
      "qualifiedName" : "human_resources.user",
      "superEntity" : {
        "addMissingOptionalFields" : false,
        "allowExtraFieldsInMapping" : true,
        "allowExtraFieldsInValidation" : true,
        "attributes" : { },
        "fields" : [ {
          "allowBlank" : true,
          "array" : false,
          "attributes" : { },
          "entityQualifiedName" : "auditable.created_by",
          "limitToAllowedValues" : false,
          "name" : "created_by",
          "primaryKey" : false,
          "qualifiedName" : "human_resources.auditable.created_by",
          "required" : true,
          "strictArrays" : true,
          "tags" : { },
          "type" : "STRING"
        }, {
          "array" : false,
          "attributes" : { },
          "entityQualifiedName" : "auditable.created_on",
          "lenientPattern" : true,
          "limitToAllowedValues" : false,
          "name" : "created_on",
          "primaryKey" : false,
          "qualifiedName" : "human_resources.auditable.created_on",
          "required" : true,
          "strictArrays" : true,
          "tags" : { },
          "type" : "DATETIME"
        }, {
          "allowBlank" : true,
          "array" : false,
          "attributes" : { },
          "entityQualifiedName" : "auditable.modified_by",
          "limitToAllowedValues" : false,
          "name" : "modified_by",
          "primaryKey" : false,
          "qualifiedName" : "human_resources.auditable.modified_by",
          "required" : false,
          "strictArrays" : true,
          "tags" : { },
          "type" : "STRING"
        }, {
          "array" : false,
          "attributes" : { },
          "entityQualifiedName" : "auditable.modified_on",
          "lenientPattern" : true,
          "limitToAllowedValues" : false,
          "name" : "modified_on",
          "primaryKey" : false,
          "qualifiedName" : "human_resources.auditable.modified_on",
          "required" : false,
          "strictArrays" : true,
          "tags" : { },
          "type" : "DATETIME"
        } ],
        "name" : "auditable",
        "qualifiedName" : "human_resources.auditable",
        "tags" : { }
      },
      "superEntityName" : "auditable",
      "tags" : { }
    },
    "field" : {
      "array" : false,
      "attributes" : { },
      "entityQualifiedName" : "user.created",
      "lenientPattern" : true,
      "limitToAllowedValues" : false,
      "name" : "created",
      "primaryKey" : false,
      "qualifiedName" : "human_resources.user.created",
      "required" : false,
      "strictArrays" : true,
      "tags" : { },
      "type" : "DATETIME"
    },
    "message" : "created is wrong type; expected DATETIME, found STRING"
  } ],
  "fieldNames" : [ "created_by", "created" ],
  "fields" : [ {
    "allowBlank" : true,
    "array" : false,
    "attributes" : { },
    "entityQualifiedName" : "auditable.created_by",
    "limitToAllowedValues" : false,
    "name" : "created_by",
    "primaryKey" : false,
    "qualifiedName" : "human_resources.auditable.created_by",
    "required" : true,
    "strictArrays" : true,
    "tags" : { },
    "type" : "STRING"
  }, {
    "array" : false,
    "attributes" : { },
    "entityQualifiedName" : "user.created",
    "lenientPattern" : true,
    "limitToAllowedValues" : false,
    "name" : "created",
    "primaryKey" : false,
    "qualifiedName" : "human_resources.user.created",
    "required" : false,
    "strictArrays" : true,
    "tags" : { },
    "type" : "DATETIME"
  } ],
  "messages" : [ "Required created_by is null", "created is wrong type; expected DATETIME, found STRING" ],
  "valid" : false
}
=========================================================================================
0 - Required created_by is null
1 - created is wrong type; expected DATETIME, found STRING
Get DataPipeline 8.3.0
All Tags | All Examples
Mobile Analytics