Schema Validation
Overview
Schemas are a declarative way to validate data in your apps. They include simple rules like length and type checks, as well as more complicated expression-based rules involving variables, lookups, and callout functions.
Like other DataPipeline Foundations components, schemas can be used on their own or plugged into a DataPipeline job.
Validate A Value
The simplest way to use schema validation is to create a FieldDef with your validation rules and apply it to a value.
This example shows how to create a basic "age" validation rule and apply it to several values. The age rule defines a required integer value between 25 and 75.
Example: ValidateAValue.java
The FielDef's validate methods return a ValidationResult containing several properties to determine success/failure and the reasons for any failures. For example, the above example produces the following output:
Validate A Field
FieldDef instances can be used to validate plain old Java values, DataPipeline Field instances, and Field instances inside a Record.
This example shows how to create an "age" validation rule and apply it to a field and a record. The rule defines a required integer value between 25 and 75.
Example: ValidateAField.java
Validate Record Fields
Schemas can do more than validate individual fields. FieldDefs can be combined together in an EntityDef to validate whole records.
This example shows how to create an entity to validate the name and age fields in a record. The validation will pass if name and age match the specified rules, even if other fields are present in the record. If extra fields in a record should cause the validation to fail, just set the EntityDef's allowExtraFieldsInValidation to false.
Example: ValidateRecordFields.java
Validate Records Using Rules
Record validation is not limited to fields. EntityDefs can accept arbitrary rules alongside (or instead of) field validations.
This example shows how to add two validation rules to an EntityDef:
- Each record must contain two fields.
- The following expression must evaluate to true:
!contains(email, '@example.com') && year(now()) >= 2019
Example: ValidateRecordsUsingRules.java
Validate Records Using Fields And Rules
Combining field validations with expressions can create a powerful duo. Fields ensure your data matches your required structure while the rules provide for deeper semantic validation.
This example shows how to combine FieldDef validations with rules in a single EntityDef. The entity is then used to validate data allowing and then not allowing extra fields. This example also shows the details of the messages returned from validation.
Example: ValidateRecordsUsingFieldsAndRules.java
Validate Records in a DataPipline Job
While schemas are useful on their own as a standalone feature, they can also be plugged into DataPipeline jobs to perform validations on-the-fly.
This example shows how to create a SchemaFilter wrapping an EntityDef to validate the structure of data coming from a CSV file.
