com.northconcepts.datapipeline.jdbc

Class JdbcMultiUpsertWriter

java.lang.Object

com.northconcepts.datapipeline.core.DataObject

com.northconcepts.datapipeline.core.Endpoint

com.northconcepts.datapipeline.core.DataEndpoint

com.northconcepts.datapipeline.core.DataWriter

com.northconcepts.datapipeline.core.ProxyWriter

com.northconcepts.datapipeline.jdbc.JdbcMultiUpsertWriter

public class JdbcMultiUpsertWriter
extends ProxyWriter

Writes records to a database table using the upsert idiom (attempt to insert, but update if duplicate key exists). It uses 1 or more connections; each connection writing asynchronously using a separate thread. The default strategy used is GenericUpsert, however several other are provided: MergeUpsert, VariableFieldsUpsert, MySqlUpsert.

Nested Class Summary

Nested classes/interfaces inherited from class com.northconcepts.datapipeline.core.DataEndpoint

DataEndpoint.State

Field Summary

Fields inherited from class com.northconcepts.datapipeline.core.DataEndpoint

lastRecord, PRODUCT, PRODUCT_VERSION, VENDOR, XML_INPUT_FACTORY_KEY

Fields inherited from class com.northconcepts.datapipeline.core.Endpoint

BUFFER_SIZE, captureElapsedTime, DEFAULT_READ_BUFFER_SIZE

Fields inherited from class com.northconcepts.datapipeline.core.DataObject

id, log, name, TIMESTAMP_FORMAT

Constructor Summary

Constructors

Constructor and Description
JdbcMultiUpsertWriter(DataSource dataSource, int connections, int maxQueuedRecordsPerConnection, String tableName, IUpsert upsert, String... keyFieldNames)
JdbcMultiUpsertWriter(DataSource dataSource, int connections, int maxQueuedRecordsPerConnection, String tableName, String... keyFieldNames)
JdbcMultiUpsertWriter(JdbcConnectionFactory factory, int connections, int maxQueuedRecordsPerConnection, String tableName, IUpsert upsert, String... keyFieldNames)
JdbcMultiUpsertWriter(JdbcConnectionFactory factory, int connections, int maxQueuedRecordsPerConnection, String tableName, String... keyFieldNames)

Method Summary

Modifier and Type	Method and Description
DataException	addExceptionProperties(DataException exception) Adds this endpoint's current state to a DataException.
protected JdbcUpsertWriter	createJdbcUpsertWriter(Connection connection, String tableName, IUpsert upsertStrategy)
AsyncWriter	getAsyncWriter(int index)
int	getBatchSize()
int	getConnections()
DataException	getException() Returns an exception thrown by one of the internal AsyncWriters if any.
List<DataException>	getExceptions() Returns all exceptions thrown by the internal AsyncWriters (the list will not be null, but may contain no elements).
Integer	getJdbcType(Class<?> type) The Types constant (or a database specific value) assigned to the specified Java class type using setJdbcType(Class, int) or null if none was set.
JdbcUpsertWriter	getJdbcWriter(int index)
FieldList	getNonUpdateFields() Returns the list of fields that must not be updated if the recored already exists (default null).
IUpsert	getUpsert()
boolean	isCommitBatch() Indicates if this writer should call commit on each connection after every batch is sent, otherwise, commit is called after the final batch update (default is false).
boolean	isDebug() Indicates if the generated SQL should be logged (default false).
void	open() Makes this endpoint ready for reading or writing.
void	rethrowAsyncException() Re-throws an exception thrown by one of the internal AsyncWriters or returns silently if no exceptions were thrown.
JdbcMultiUpsertWriter	setBatchSize(int batchSize) Sets the number of records to chunk together in batch updates.
JdbcMultiUpsertWriter	setCommitBatch(boolean commitBatch) Indicates if this writer should call commit on each connection after every batch is sent, otherwise, commit is called after the final batch update (default is false).
JdbcMultiUpsertWriter	setDebug(boolean debug) Indicates if the generated SQL should be logged (default false).
JdbcMultiUpsertWriter	setJdbcType(Class<?> type, int jdbcType) Forces field values of the specified Java class type to be sent to the database using PreparedStatement.setObject(parameterIndex, fieldValue, jdbcType) (instead of the appropriate PreparedStatement.setXXX() method for its field's type).
JdbcMultiUpsertWriter	setJdbcType(String fieldName, int jdbcType) The given Java object will be converted to the given targetSqlType before being sent to the database.
JdbcMultiUpsertWriter	setNonUpdateFields(FieldList nonUpdateFields) Assigns the list of fields that must not be updated if the recored already exists (default null).
JdbcMultiUpsertWriter	setNonUpdateFields(String... nonUpdateFields) Assigns the list of fields that must not be updated if the recored already exists (default null).
protected void	writeImpl(Record record) Overridden by subclasses to write the specified record to this DataWriter.

Methods inherited from class com.northconcepts.datapipeline.core.ProxyWriter

available, close, getNestedWriter, interceptRecord, map, map, setNestedDataWriter, setNestedDataWriter

Methods inherited from class com.northconcepts.datapipeline.core.DataWriter

getNestedEndpoint, getRootEndpoint, getRootWriter, getWriter, write

Methods inherited from class com.northconcepts.datapipeline.core.DataEndpoint

decrementRecordCount, enableJmx, getLastRecord, getRecordCount, getRecordCountAsBigInteger, getRecordCountAsString, incrementRecordCount, isRecordCountBigInteger, resetRecordCount, toString

Methods inherited from class com.northconcepts.datapipeline.core.Endpoint

addElapsedtime, assertClosed, assertNotOpened, assertOpened, finalize, getClosedOn, getDescription, getElapsedTime, getElapsedTimeAsString, getOpenedOn, getOpenElapsedTime, getOpenElapsedTimeAsString, getSelfTime, getSelfTimeAsString, getState, isCaptureElapsedTime, isClosed, isOpen, setCaptureElapsedTime, setDescription

Methods inherited from class com.northconcepts.datapipeline.core.DataObject

exception, exception, exception, getId, getName, resetID

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

JdbcMultiUpsertWriter

public JdbcMultiUpsertWriter(DataSource dataSource,
                             int connections,
                             int maxQueuedRecordsPerConnection,
                             String tableName,
                             IUpsert upsert,
                             String... keyFieldNames)

JdbcMultiUpsertWriter

public JdbcMultiUpsertWriter(DataSource dataSource,
                             int connections,
                             int maxQueuedRecordsPerConnection,
                             String tableName,
                             String... keyFieldNames)

JdbcMultiUpsertWriter

public JdbcMultiUpsertWriter(JdbcConnectionFactory factory,
                             int connections,
                             int maxQueuedRecordsPerConnection,
                             String tableName,
                             IUpsert upsert,
                             String... keyFieldNames)
                      throws DataException

Throws:

DataException

JdbcMultiUpsertWriter

public JdbcMultiUpsertWriter(JdbcConnectionFactory factory,
                             int connections,
                             int maxQueuedRecordsPerConnection,
                             String tableName,
                             String... keyFieldNames)
                      throws DataException

Throws:

DataException

Method Detail

getConnections

public int getConnections()

getJdbcWriter

public JdbcUpsertWriter getJdbcWriter(int index)

getAsyncWriter

public AsyncWriter getAsyncWriter(int index)

getBatchSize

public int getBatchSize()

setBatchSize

public JdbcMultiUpsertWriter setBatchSize(int batchSize)

Sets the number of records to chunk together in batch updates. Updates are not batched if this value is 1 or less. JdbcWriter defaults to non batching updates.

isCommitBatch

public boolean isCommitBatch()

Indicates if this writer should call commit on each connection after every batch is sent, otherwise, commit is called after the final batch update (default is false).

setCommitBatch

public JdbcMultiUpsertWriter setCommitBatch(boolean commitBatch)

Indicates if this writer should call commit on each connection after every batch is sent, otherwise, commit is called after the final batch update (default is false).

getNonUpdateFields

public FieldList getNonUpdateFields()

Returns the list of fields that must not be updated if the recored already exists (default null). This method may return null or a list of empty fields.

setNonUpdateFields

public JdbcMultiUpsertWriter setNonUpdateFields(FieldList nonUpdateFields)

Assigns the list of fields that must not be updated if the recored already exists (default null).

setNonUpdateFields

public JdbcMultiUpsertWriter setNonUpdateFields(String... nonUpdateFields)

Assigns the list of fields that must not be updated if the recored already exists (default null).

getUpsert

public IUpsert getUpsert()

setJdbcType

public JdbcMultiUpsertWriter setJdbcType(String fieldName,
                                         int jdbcType)

The given Java object will be converted to the given targetSqlType before being sent to the database.

Note that this method may be used to pass database-specific abstract data types.

Parameters:

fieldName -

jdbcType - the object containing the JDBC Type parameter value

setJdbcType

public JdbcMultiUpsertWriter setJdbcType(Class<?> type,
                                         int jdbcType)

Forces field values of the specified Java class type to be sent to the database using PreparedStatement.setObject(parameterIndex, fieldValue, jdbcType) (instead of the appropriate PreparedStatement.setXXX() method for its field's type). This method may be used to pass database-specific abstract data types.

Parameters:

type - the Java class type to override.

jdbcType - one of the Types constants or a database specific value.

getJdbcType

public Integer getJdbcType(Class<?> type)

The Types constant (or a database specific value) assigned to the specified Java class type using setJdbcType(Class, int) or null if none was set.

Returns:

one of the Types constants or a database specific value.

isDebug

public boolean isDebug()

Indicates if the generated SQL should be logged (default false).

setDebug

public JdbcMultiUpsertWriter setDebug(boolean debug)

Indicates if the generated SQL should be logged (default false).

open

public void open()
          throws DataException

Description copied from class: DataEndpoint

Makes this endpoint ready for reading or writing.

Overrides:

open in class ProxyWriter

Throws:

DataException

createJdbcUpsertWriter

protected JdbcUpsertWriter createJdbcUpsertWriter(Connection connection,
                                                  String tableName,
                                                  IUpsert upsertStrategy)

writeImpl

protected void writeImpl(Record record)
                  throws Throwable

Description copied from class: DataWriter

Overridden by subclasses to write the specified record to this DataWriter.

Overrides:

writeImpl in class ProxyWriter

Throws:

Throwable

getException

public DataException getException()

Returns an exception thrown by one of the internal AsyncWriters if any.

getExceptions

public List<DataException> getExceptions()

Returns all exceptions thrown by the internal AsyncWriters (the list will not be null, but may contain no elements).

rethrowAsyncException

public void rethrowAsyncException()
                           throws DataException

Re-throws an exception thrown by one of the internal AsyncWriters or returns silently if no exceptions were thrown.

Throws:

DataException

addExceptionProperties

public DataException addExceptionProperties(DataException exception)

Description copied from class: Endpoint

Adds this endpoint's current state to a DataException. Since this method is called whenever an exception is thrown, subclasses should override it to add their specific information.

Overrides:

addExceptionProperties in class ProxyWriter