Build Pipelines Declaratively

Overview

In addition to the code-centric approach, DataPipeline allows you to configure pipelines and components in XML and JSON. You can start with configuration or build your components in code and export them to JSON or XML. This allows you to store your configurations in a database or file where they can be changed anytime without redeploying your app.

Features

Schema Validation

Schemas are a declarative way to validate and automatically convert data in your apps. They include simple rules like length and type checks, as well as more complicated expression-based rules.

• Automatic data type conversion
• Input validation: required, min, max, rounding, regex, value-set
• Multi-field, rule-based validations
• Default value expressions
• Call built-in and custom functions
• Define fields by name or position
• Lookups
• Apply schemas to records, fields, literals, and pipelines
• Use within or outside of a pipeline
• Convert to and from XML and JSON
• Learn more about schema validation

Data Mapping

Data mapping is a quick way to transform data from source to target using expressions and functions.

• Source-to-target data mapping using expressions
• Schema validation and automatic conversion before and after mapping
• Conditional field and record mapping
• Default value expressions
• Calculated fields
• Call built-in and custom functions
• Lookups
• Use within or outside of a pipeline
• Convert to and from XML and JSON
• Learn more about data mapping

Decision Tables

Decision tables can be used to decouple business/decision logic from application code and make them accessible to non-technical users.

• Define your business/decision logic as a decision table
• Calculated fields
• Conditions and outcomes as expressions
• Call built-in and custom functions
• Lookups
• Use within or outside of a pipeline
• Convert to and from XML and JSON
• Learn more about decision tables

Decision Trees

Decision trees allow you to express complicated logic as simple flow charts.

• Define your business/decision logic as a decision tree
• Calculated fields
• Conditions and outcomes as expressions
• Call built-in and custom functions
• Lookups
• Use within or outside of a pipeline
• Convert to and from XML and JSON
• Learn more about decision trees

Pipelines

Pipelines are a new way to create your DataPipeline jobs. Instead of working directly with readers and writers, pipelines allow you to create jobs declaratively by defining inputs, outputs, actions, and datasets.

• Build data mapping or command-based pipelines
• Define each pipeline's data source and target
• Validate and convert incoming and outgoing data
• Work with the dataset generated from a pipeline in memory or backed by a database
• Obtain stats, inferences, and metadata from datasets
• Multi-threaded
• Convert to and from XML and JSON
• Learn more about pipelines