If instead of a slow source you're working with a slow target, Data Pipeline provides an
AsyncWriter that uses a separate thread
to continuously write data from its buffer to any downstream
Data written to the
AsyncWriter is placed in the buffer, allowing the pipeline to continue with the next record without waiting for the previous
record to be written. The only time the pipeline must wait is when the buffer is full.
The following example uses an
AsyncWriter with a 1000 record buffer to write to a
StreamWriter in a separate thread.
The StreamWriter here can be replaced with a
JdbcWriter or any other writer.