All examples can be found on GitHub (https://github.com/NorthConcepts/DataPipeline-Examples).
- Buffer Records by Time Period or Count
- Capture Data Not Joined in a Lookup
- Compile and Run a Job
- Continue After an Error
- Convert a Single Source DataReader into Many
- Convert JSON to CSV
- Debug my Code
- Extract Bigrams, Trigrams, and Ngrams
- Generate a PDF
- Generate a Word Document
- Group Records by TimePeriod or Count
- Handle Exceptions
- Log Diagnostic Information
- Measure Data being Read and Written
- Measure Data Lineage Performance
- Measure Performance of Reader and Writer
- Obtain Statistics
- Open and Close Several Data Readers and Data Writers at Once
- Pipe a Writer to a Reader
- Profile Performance
- Read BigDecimal and BigInteger from an Excel file
- Read a Bloomberg Message File
- Read a CSV File
- Read a Fixed-width File / Fixed-length Record File
- Read a JSON Stream
- Read a Parquet File
- Read a Patient File
- Read a Simple JSON File
- Read a Simple XML File
- Read a Web Server Log
- Read an Avro File
- Read an XML File
- Read an XML File (2)
- Read and Write to an EventBus
- Read Big Decimal In JSON
- Read Emails
- Read From Amazon S3
- Read from a Database
- Read from an Excel File
- Read from Gmail
- Read from Java beans
- Read from JMS Queue
- Read from JMS Topic
- Read from Memory
- Read from MongoDB
- Read Google Analytics goal conversions
- Read Google Analytics Social Interactions
- Read Google Analytics views
- Read Google Calendar
- Read Google Contacts
- Read Google Gmail Messages
- Read JSON Records From File
- Read Selected Fields from a Parquet File
- Read using TimedReader
- Read XML Records From File
- Search for a Record
- Search Twitter for Tweets
- Serialize and Deserialize Data
- Serialize and Deserialize Records
- Throttle Data being Read
- Throttle Data being Written
- Use Multi Threading in a Single Job
- Use a Retrying Reader
- Use a Retrying Writer
- Use Data Lineage with CsvReader
- Use Data Lineage with ExcelReader
- Use Data Lineage with FixedWidthReader
- Use Data Lineage with JdbcReader
- Use Data Lineage with Lookup
- Use Data Lineage with ParquetReader
- Use Streaming Excel for Reading
- Use Streaming Excel for Writing
- Upsert Records to a Database Using Insert and Update
- Upsert Records to a Database Using Merge
- Upsert Records to MySql or MariaDB
- Upsert Records to Oracle
- Upsert Records to PostgreSql
- Upsert Records to Sybase
- Upsert Variable Field Records
- Write a CSV File to Database (1)
- Write a CSV File to Database (2)
- Write a CSV File to Fixed Width
- Write a Parquet File
- Write a Simple JSON File
- Write a Simple XML File
- Write a Sequence of Files by Record Count
- Write a Sequence of Files by Elapsed Time
- Write an Avro File
- Write an XML File Programmatically
- Write an XML File using FreeMarker Templates
- Write CSV To XML Using FreeMarker Templates
- Write HTML using FreeMarker Templates
- Write Key-Value Fields to MapWriter
- Write to Amazon S3 Using Multipart Streaming
- Write to a Database Using Custom Jdbc Insert Strategy
- Write to a Database Using Generic Upsert Strategy
- Write to a Database Using Merge Upsert Strategy
- Write to a Database Using Merge Upsert Strategy with Batch
- Write to a Database Using Multiple Connections
- Write to a Database Using Multi Row Prepared Statement Insert Strategy
- Write to a Database Using Multi Row Statement Insert Strategy
- Write to Excel
- Write to JMS Queue
- Write to JMS Topic
- Write to JSON Stream (simple)
- Write to JSON Stream Programmatically
- Write to Memory
- Write to MongoDB
- Write to Several Data Writers at Once
- Write to the Console
- Write to XML Stream (Simple)
- Add a Sequence Number Column
- Add a Sequence Number Column when Values Change
- Add a Sequence Number Column for Repeat Values
- Add Nonpersistent Data to Records and Fields
- Cache Lookup Values
- Conditionally Transform Data
- Conditionally Write to Multiple Files
- Copy a Field
- Filter Records
- Filter Twitter in Real Time
- Find The Minimum Maximum Average Sum Count
- Get a Query Param from a Url
- Handle Filtered Records
- Handle Transforming Reader Failures
- Handle Transforming Writer Failures
- Handle Validation Failures
- Log Job Progress
- Lookup Data in a Database
- Lookup Data in any Data Reader
- Lookup in a Flat File
- Manipulate Fields
- Read a Set of Files
- Remove Duplicates
- Remove/Retain Duplicate Fields
- Remove Fields
- Rename a Field
- Select and Arrange Fields
- Set a Calculated Field at Runtime
- Set a Calculated Field Programmatically
- Set a Field
- Sort Records
- Specify Lookup Values Programmatically
- Validate Incoming Data
- Write to A Data Writer using Tee Reader
- Add my own Functions
- Blacklist and Whitelist Functions in DP Expression Language
- My Data Reader
- My Data Writer
- My Filter
- My Lookup
- My Proxy Reader
- My Proxy Writer
- My Transformer
- Write my own Data Reader
- Write my own Data Writer
- Write my own Filter or Validator
- Write my own Lookup
- Write my own Transformer
- Add a Decision Table to a Pipeline
- Add a Decision Tree to a Pipeline
- Add Calculated Fields to a Decision Table
- Add Calculated Fields to a Decision Tree
- Conditionally map Data from Source to Target
- Conditionally map DataField from Source to Target
- Create Custom Pipeline Action
- Create Custom Pipeline Input
- Create Custom Pipeline Output
- Evaluate a Decision Table
- Evaluate a Decision Table with Lookup
- Evaluate a Decision Tree
- Evaluate a Decision Tree with Lookup
- Execute an Action in a Decision Table
- Execute an Action in a Decision Tree
- Filter Columns with All Null Values
- Generate Java Beans from a Database
- Map Data with Rule Based Validation
- Map Data with Schema Based Validation
- Map Data from Source to Target
- Map Data from Source to Target in a Pipeline
- Map Data from Source to Target in a Pipeline with Validation
- Map Data from Source to Target with Lookup
- Read from CSV And Writer to Excel
- Save and Load DataMapping to XML
- Save and Load DecisionTable to XML
- Save and Load DecisionTree to XML
- Load Snapshot of Dataset
- Save and Restore Pipeline from JSON
- Show Column Statistics
- Show the Columns and Tables of a Schema
- Show Unique Values in Column
- Use an EventBus
- Use an EventBus in a Pipeline
- Use SchemaFilter to Validate Records in a Pipeline
- Validate a Field
- Validate a Value
- Validate Record Fields
- Validate Records using Fields and Rules
- Validate Records using Rules
Tags
Aggregate (2)
Amazon S3 (2)
Assign Field (10)
Avro (5)
Bloomberg (1)
Cache (1)
Calculate Field (9)
Conversion (7)
Copy Field (2)
CSV (20)
Customize (14)
Data Lineage (7)
Data Mapping (9)
Debug (9)
Decision Table (6)
Decision Tree (6)
Email (3)
EventBus (3)
Exceptions (4)
Exclude Fields (1)
Filter (10)
Fixed Width (6)
Freemarker (1)
Functions (2)
Google (6)
HTML (1)
In-Memory (17)
Include Fields (1)
Java Beans (1)
JDBC (24)
JMS (4)
Job management (2)
JSON (11)
Lookup (10)
Meter (2)
MongoDB (2)
MS Excel (10)
Multi-writer (4)
Native (1)
Out-of-band data (1)
Parquet (4)
PDF (1)
Performance (4)
Pipe (1)
Pipeline (9)
Push (1)
Reactive (0)
Rename Field (2)
Retrying (2)
RTF (1)
Rules (2)
Schema (6)
Search (1)
Serialization (2)
Simple (14)
Sort (1)
Stats (2)
STDOUT (12)
Streaming data (2)
Template (5)
Threading (4)
Throttle (2)
Transform (37)
Twitter (2)
Validate (3)
Web Server Logs (1)
XML (16)
XPath (5)