Asynchronous Writing

If instead of a slow source you're working with a slow target, Data Pipeline provides an AsyncWriter that uses a separate thread to continuously write data from its buffer to any downstream DataWriter.

Data written to the AsyncWriter is placed in the buffer, allowing the pipeline to continue with the next record without waiting for the previous record to be written. The only time the pipeline must wait is when the buffer is full.

The following example uses an AsyncWriter with a 1000 record buffer to write to a StreamWriter in a separate thread. The StreamWriter here can be replaced with a JdbcWriter or any other writer.