Data Pipeline User Guide
Contents
- Introduction
- Getting Started
- 1. Add Maven or Gradle Dependencies
- 2. Download a license
- 3. Install License
- 4. Start Coding
- Integrations
- Configure Gradle/Maven for Compressed Parquet Support
- Additional Resources
- Running Diagnostics
- How Data Pipeline Works
- Records
- Field Names
- Field Values
- Field Types
- Data Readers and Writers
- Stream Operators
- Java Streams and Iterators
- Running Jobs
- Synchronous Jobs
- Asynchronous Jobs
- RunAsync
- Thread
- ExecutorService
- Waiting for Jobs to Start and Finish
- Managing Running Jobs
- Cancel
- Pause and Resume
- Monitoring Jobs
- Directly Access Job Properties
- Job Callback Hooks
- Job Lifecycle Listener
- Jobs Logging
- JMX Monitoring of Jobs
- Data Formats
- Filtering Data
- Data Validation
- Transforming Data
- Conditional Transformations
- Lookup and Joins
- Data Conversions
- Data Aggregation
- Simple Aggregation
- Aggregate Operators
- Sliding Window Aggregation
- Batch/Adjacent Sliding Windows
- Overlapping Sliding Windows
- Sparse/Sample Sliding Windows
- Time-Based Sliding Windows
- Built-in Window Strategies
- Custom Window Strategies
- Custom Aggregate Operators
- Detail and Summarize in a Single Pipeline
- Expression Language
- Multi-threaded Processing
- Asynchronous Reading
- Asynchronous Writing
- Reading Many-to-One Asynchronously
- Reading One-to-Many Asynchronously
- Writing One-to-Many Asynchronously
- Process Incoming Data on Multiple Threads
- Event Bus
- How the Event Bus Works
- EventListener Interface
- Starting and Stopping an Event Bus
- Using The System Event Bus
- Publish and Subscribe to Events on the Bus
- Accessing the Current Event
- Connecting Pipelines to an Event Bus
- EventBusWriter
- EventBusReader
- Event Bus Pipelines
- Monitoring Event Buses
- Directly Access Event Bus Properties
- Event Bus Lifecycle Listener
- Watching All Records on an Event Bus
- Watching All Events on an Event Bus
- Monitoring Event Buses with JMX
- Handling Exceptions in the Event Bus
- Meter and Throttle DataPipeline Jobs
- Debugging DataPipeline Jobs
- Error Handling in DataPipeline Jobs
- Creating Custom DataPipeline Components
