com.northconcepts.datapipeline.foundations.pipeline.dataset

Class JdbcDataset

java.lang.Object

com.northconcepts.datapipeline.foundations.core.Bean

com.northconcepts.datapipeline.foundations.core.FoundationObject

com.northconcepts.datapipeline.foundations.pipeline.dataset.Dataset

com.northconcepts.datapipeline.foundations.pipeline.dataset.JdbcDataset

All Implemented Interfaces:

DataExceptionContributor, JsonSerializable, RecordSerializable, XmlSerializable, Closeable, Serializable, AutoCloseable, Iterable<Record>

public class JdbcDataset
extends Dataset

Caches the dataset's records on database.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	JdbcDataset.DatabaseProvider Enumeration of supported database providers for JDBC datasets.

Nested classes/interfaces inherited from class com.northconcepts.datapipeline.foundations.pipeline.dataset.Dataset

Dataset.ColumnsDataReader

Field Summary

Fields inherited from class com.northconcepts.datapipeline.foundations.core.FoundationObject

internalId, internalName, log, TIMESTAMP_FORMAT

Fields inherited from interface com.northconcepts.datapipeline.core.RecordSerializable

SERIALIZED_CLASS_NAME, TYPE

Fields inherited from interface com.northconcepts.datapipeline.core.XmlSerializable

XML_SERIALIZED_CLASS_NAME

Method Summary

Modifier and Type	Method and Description
DataException	addExceptionProperties(DataException exception) Adds this object's current state to a DataException.
protected void	afterColumnStatsLoaded() Called during the data loading process after all the column stats have been loaded.
protected void	afterLoad() Called at the end of the data loading process after all the records and column stats have been loaded.
protected void	beforeLoad() Called at the start of the data loading process, but before any records or column stats have been loaded.
void	close()
static JdbcDataset	createDataset(JdbcDataset.DatabaseProvider databaseProvider, JdbcConnectionFactory factory, String tableNamePrefix, AbstractPipeline pipeline) Creates a persistent dataset stored in a database determined by the provided JdbcDataset.DatabaseProvider.
static JdbcDataset	createDataset(JdbcDataset.DatabaseProvider databaseProvider, JdbcConnectionFactory factory, String tableNamePrefix, DataReaderFactory dataReaderFactory) Creates a persistent dataset stored in a database determined by the provided JdbcDataset.DatabaseProvider.
protected DataWriter	createDataWriter() Writes records to this dataset's cache after clearing it.
Column	getColumn(int index)
Column	getColumn(String name)
long	getColumnCount()
List<String>	getColumnNames()
List<Column>	getColumns()
String	getDataTableName() Gets the name of the data table in the database.
protected Column	getOrCreateColumn(String name, int index)
Record	getRecord(long index)
long	getRecordCount()
RecordList	getRecordList(long offset, int count) Get a subset of the records cached in this dataset.
String	getStatsTableName() Gets the name of the statistics table in the database.
boolean	isAutoCloseConnection() Returns whether the database connection will be automatically closed when this dataset is closed.
boolean	isDropTablesOnClose() Returns whether the database tables will be dropped when this dataset is closed.
static JdbcDataset	openDataset(JdbcDataset.DatabaseProvider databaseProvider, JdbcConnectionFactory factory, String tableNamePrefix) Loads an existing persistent dataset stored in a database determined by the provided JdbcDataset.DatabaseProvider.
JdbcDataset	setAutoCloseConnection(boolean autoCloseConnection) Sets whether to automatically close the database connection when this dataset is closed.
JdbcDataset	setDropTablesOnClose(boolean dropTablesOnClose) Sets whether to drop the database tables when this dataset is closed.

Methods inherited from class com.northconcepts.datapipeline.foundations.pipeline.dataset.Dataset

addField, afterRecordsLoaded, cancelLoad, createColumnsDataReader, createDataReader, createDataReader, finalize, forEach, fromRecord, getColumnStatsException, getColumnStatsReaderThreads, getDataLoadException, getJob, getJobExecutor, getMaxColumnStatsRecords, getMaxColumnsToAnalyze, getMaxRecordsToLoad, getPipeline, isCollectUniqueValues, isColumnStatsLoaded, isDataLoaded, isDataLoading, isDetectBigNumberValues, isDetectBooleanValues, isDetectNumericValues, isDetectTemporalValues, isDetectUuidValues, isInferStringTypes, isRecordsLoaded, iterator, load, load, load, setCollectUniqueValues, setColumnStatsLoaded, setColumnStatsReaderThreads, setDetectBigNumberValues, setDetectBooleanValues, setDetectNumericValues, setDetectTemporalValues, setDetectUuidValues, setInferStringTypes, setJobExecutor, setMaxColumnStatsRecords, setMaxColumnsToAnalyze, setPipeline, setRecordsLoaded, stream, toRecord, updateColumns, waitForColumnStatsToLoad, waitForColumnStatsToLoad, waitForRecordsToLoad, waitForRecordsToLoad, waitUntilJobFinished

Methods inherited from class com.northconcepts.datapipeline.foundations.core.FoundationObject

assertValid, assertValid, clone, exception, exception, exception, getInternalId, getInternalName, resetInternalId

Methods inherited from class com.northconcepts.datapipeline.foundations.core.Bean

toString

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface java.lang.Iterable

spliterator

Methods inherited from interface com.northconcepts.datapipeline.core.RecordSerializable

fromArrayValue, fromArrayValue, fromArrayValue, fromJson, fromJson, newInstanceFromRecord, newInstanceFromRecord, toArrayValue, toJson, toJson, toJson, toRecord

Methods inherited from interface com.northconcepts.datapipeline.core.JsonSerializable

newInstanceFromJson

Methods inherited from interface com.northconcepts.datapipeline.core.XmlSerializable

forEachGrandChildElement, fromXml, fromXml, fromXmlElement, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getChildElement, getChildElements, newInstanceFromXml, newInstanceFromXml, setAttribute, toXml, toXml, toXml, toXml, toXml, toXmlElement, writeXml, writeXml

Methods inherited from interface com.northconcepts.datapipeline.core.DataExceptionContributor

addExceptionProperties, contributeExceptionProperties, contributeExceptionProperties

Method Detail

createDataset

public static JdbcDataset createDataset(JdbcDataset.DatabaseProvider databaseProvider,
                                        JdbcConnectionFactory factory,
                                        String tableNamePrefix,
                                        AbstractPipeline pipeline)

Creates a persistent dataset stored in a database determined by the provided JdbcDataset.DatabaseProvider.

The data and statistics tables will remain on the database even after the dataset is closed, unless setDropTablesOnClose(boolean) is set to true.

Parameters:

databaseProvider - the database provider defining database-specific SQL syntax

factory - the JDBC connection factory for creating database connections

tableNamePrefix - the prefix for the data and statistics table names

pipeline - the pipeline to use for loading data into the dataset

Returns:

a new JdbcDataset instance

createDataset

public static JdbcDataset createDataset(JdbcDataset.DatabaseProvider databaseProvider,
                                        JdbcConnectionFactory factory,
                                        String tableNamePrefix,
                                        DataReaderFactory dataReaderFactory)

Creates a persistent dataset stored in a database determined by the provided JdbcDataset.DatabaseProvider.

The data and statistics tables will remain on the database even after the dataset is closed, unless setDropTablesOnClose(boolean) is set to true.

Parameters:

databaseProvider - the database provider defining database-specific SQL syntax

factory - the JDBC connection factory for creating database connections

tableNamePrefix - the prefix for the data and statistics table names

dataReaderFactory - the factory for creating the data reader to populate the dataset

Returns:

a new JdbcDataset instance

openDataset

public static JdbcDataset openDataset(JdbcDataset.DatabaseProvider databaseProvider,
                                      JdbcConnectionFactory factory,
                                      String tableNamePrefix)

Loads an existing persistent dataset stored in a database determined by the provided JdbcDataset.DatabaseProvider.

This method opens a dataset that was previously created with createDataset(com.northconcepts.datapipeline.foundations.pipeline.dataset.JdbcDataset.DatabaseProvider, com.northconcepts.datapipeline.jdbc.JdbcConnectionFactory, java.lang.String, com.northconcepts.datapipeline.foundations.pipeline.AbstractPipeline) and reads the existing data and statistics from the database tables.

Parameters:

databaseProvider - the database provider defining database-specific SQL syntax

factory - the JDBC connection factory for creating database connections

tableNamePrefix - the prefix for the data and statistics table names

Returns:

a JdbcDataset instance containing the existing dataset

getDataTableName

public String getDataTableName()

Gets the name of the data table in the database.

Returns:

the data table name

getStatsTableName

public String getStatsTableName()

Gets the name of the statistics table in the database.

Returns:

the statistics table name

setDropTablesOnClose

public JdbcDataset setDropTablesOnClose(boolean dropTablesOnClose)

Sets whether to drop the database tables when this dataset is closed.

If true, both the data and statistics tables will be dropped from the database when close() is called. If false (default), the tables will remain in the database.

Parameters:

dropTablesOnClose - true to drop tables on close, false otherwise

Returns:

this instance for method chaining

isDropTablesOnClose

public boolean isDropTablesOnClose()

Returns whether the database tables will be dropped when this dataset is closed.

Returns:

true if tables will be dropped on close, false otherwise

setAutoCloseConnection

public JdbcDataset setAutoCloseConnection(boolean autoCloseConnection)

Sets whether to automatically close the database connection when this dataset is closed.

If true (default), the database connection will be closed when close() is called. If false, the connection will remain open and must be closed manually.

Parameters:

autoCloseConnection - true to auto-close the connection, false otherwise

Returns:

this instance for method chaining

isAutoCloseConnection

public boolean isAutoCloseConnection()

Returns whether the database connection will be automatically closed when this dataset is closed.

Returns:

true if the connection will be auto-closed, false otherwise

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class Dataset

getRecordCount

public long getRecordCount()

Specified by:

getRecordCount in class Dataset

getRecord

public Record getRecord(long index)

Specified by:

getRecord in class Dataset

getRecordList

public RecordList getRecordList(long offset,
                                int count)

Description copied from class: Dataset

Get a subset of the records cached in this dataset.

Overrides:

getRecordList in class Dataset

getColumnCount

public long getColumnCount()

Specified by:

getColumnCount in class Dataset

getColumnNames

public List<String> getColumnNames()

Specified by:

getColumnNames in class Dataset

getColumn

public Column getColumn(int index)

Specified by:

getColumn in class Dataset

getColumn

public Column getColumn(String name)

Specified by:

getColumn in class Dataset

getOrCreateColumn

protected Column getOrCreateColumn(String name,
                                   int index)

Specified by:

getOrCreateColumn in class Dataset

getColumns

public List<Column> getColumns()

Specified by:

getColumns in class Dataset

beforeLoad

protected void beforeLoad()

Description copied from class: Dataset

Called at the start of the data loading process, but before any records or column stats have been loaded.

Specified by:

beforeLoad in class Dataset

afterLoad

protected void afterLoad()

Description copied from class: Dataset

Called at the end of the data loading process after all the records and column stats have been loaded.

Specified by:

afterLoad in class Dataset

afterColumnStatsLoaded

protected void afterColumnStatsLoaded()

Description copied from class: Dataset

Called during the data loading process after all the column stats have been loaded. The records would have already been loaded when this method is called since column stats require additional processing.

Overrides:

afterColumnStatsLoaded in class Dataset

createDataWriter

protected DataWriter createDataWriter()

Description copied from class: Dataset

Writes records to this dataset's cache after clearing it.

Specified by:

createDataWriter in class Dataset

addExceptionProperties

public DataException addExceptionProperties(DataException exception)

Description copied from class: FoundationObject

Adds this object's current state to a DataException. Since this method is called whenever an exception is thrown, subclasses should override it to add their specific information.

Specified by:

addExceptionProperties in interface DataExceptionContributor

Overrides:

addExceptionProperties in class FoundationObject