com.northconcepts.datapipeline.foundations.pipeline.dataset

Class MvStoreDataset

java.lang.Object

com.northconcepts.datapipeline.foundations.core.Bean

com.northconcepts.datapipeline.foundations.core.FoundationObject

com.northconcepts.datapipeline.foundations.pipeline.dataset.Dataset

com.northconcepts.datapipeline.foundations.pipeline.dataset.MvStoreDataset

All Implemented Interfaces:

DataExceptionContributor, JsonSerializable, RecordSerializable, XmlSerializable, Closeable, Serializable, AutoCloseable, Iterable<Record>

public class MvStoreDataset
extends Dataset

Caches the dataset's records on disk using MVStore.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class com.northconcepts.datapipeline.foundations.pipeline.dataset.Dataset

Dataset.ColumnsDataReader

Field Summary

Fields inherited from class com.northconcepts.datapipeline.foundations.core.FoundationObject

internalId, internalName, log, TIMESTAMP_FORMAT

Fields inherited from interface com.northconcepts.datapipeline.core.RecordSerializable

SERIALIZED_CLASS_NAME, TYPE

Fields inherited from interface com.northconcepts.datapipeline.core.XmlSerializable

XML_SERIALIZED_CLASS_NAME

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	MvStoreDataset(File databaseFile)
protected	MvStoreDataset(File databaseFile, AbstractPipeline pipeline)

Method Summary

Modifier and Type	Method and Description
protected Column	addField(Record record, Field field, int fieldIndex)
protected void	afterLoad() Called at the end of the data loading process after all the records and column stats have been loaded.
protected void	beforeLoad() Called at the start of the data loading process, but before any records or column stats have been loaded.
void	close()
static MvStoreDataset	createDataset(File databaseFile, AbstractPipeline pipeline) Creates a persistent dataset on disk in the specified databaseFile.
protected DataWriter	createDataWriter() Writes records to this dataset's cache after clearing it.
static MvStoreDataset	createTempDataset(AbstractPipeline pipeline) Creates a temporary dataset on disk that will be deleted when the dataset is closed or when the JVM exits.
static MvStoreDataset	createTempDataset(File databaseFolder, AbstractPipeline pipeline) Creates a temporary dataset on disk that will be deleted when the dataset is closed or when the JVM exits.
Column	getColumn(int index)
Column	getColumn(String name)
long	getColumnCount()
List<String>	getColumnNames()
List<Column>	getColumns()
File	getDatabaseFile()
protected Column	getOrCreateColumn(String name, int index)
Record	getRecord(long index)
long	getRecordCount()
boolean	isDeleteDatabaseFileOnClose()
Dataset	load(Integer maxRecords, JobCallback<DataReader,DataWriter> callback) Starts the asynchronous loading of records from the pipeline into this dataset.
static MvStoreDataset	openDataset(File databaseFile) Loads an existing dataset from the specified databaseFile.
MvStoreDataset	setDeleteDatabaseFileOnClose(boolean deleteDatabaseFileOnClose)
MvStoreDataset	setPipeline(AbstractPipeline pipeline)

Methods inherited from class com.northconcepts.datapipeline.foundations.pipeline.dataset.Dataset

afterColumnStatsLoaded, afterRecordsLoaded, cancelLoad, createColumnsDataReader, createDataReader, createDataReader, finalize, forEach, fromRecord, getColumnStatsException, getColumnStatsReaderThreads, getDataLoadException, getJob, getMaxColumnStatsRecords, getMaxRecordsToLoad, getPipeline, getRecordList, isCollectUniqueValues, isColumnStatsLoaded, isDataLoaded, isDataLoading, isDetectBigNumberValues, isDetectBooleanValues, isDetectNumericValues, isDetectTemporalValues, isDetectUuidValues, isInferStringTypes, isRecordsLoaded, iterator, load, load, setCollectUniqueValues, setColumnStatsLoaded, setColumnStatsReaderThreads, setDetectBigNumberValues, setDetectBooleanValues, setDetectNumericValues, setDetectTemporalValues, setDetectUuidValues, setInferStringTypes, setMaxColumnStatsRecords, setRecordsLoaded, stream, toRecord, updateColumns, waitForColumnStatsToLoad, waitForColumnStatsToLoad, waitForRecordsToLoad, waitForRecordsToLoad, waitUntilJobFinished

Methods inherited from class com.northconcepts.datapipeline.foundations.core.FoundationObject

addExceptionProperties, assertValid, assertValid, clone, exception, exception, exception, getInternalId, getInternalName, resetInternalId

Methods inherited from class com.northconcepts.datapipeline.foundations.core.Bean

toString

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface java.lang.Iterable

spliterator

Methods inherited from interface com.northconcepts.datapipeline.core.RecordSerializable

fromArrayValue, fromArrayValue, fromArrayValue, fromJson, fromJson, newInstanceFromRecord, newInstanceFromRecord, toArrayValue, toJson, toJson, toJson, toRecord

Methods inherited from interface com.northconcepts.datapipeline.core.JsonSerializable

newInstanceFromJson

Methods inherited from interface com.northconcepts.datapipeline.core.XmlSerializable

forEachGrandChildElement, fromXml, fromXml, fromXmlElement, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getChildElement, getChildElements, newInstanceFromXml, newInstanceFromXml, setAttribute, toXml, toXml, toXml, toXml, toXml, toXmlElement, writeXml, writeXml

Methods inherited from interface com.northconcepts.datapipeline.core.DataExceptionContributor

addExceptionProperties, contributeExceptionProperties, contributeExceptionProperties

Constructor Detail

MvStoreDataset

protected MvStoreDataset(File databaseFile,
                         AbstractPipeline pipeline)

MvStoreDataset

protected MvStoreDataset(File databaseFile)

Method Detail

createTempDataset

public static MvStoreDataset createTempDataset(AbstractPipeline pipeline)

Creates a temporary dataset on disk that will be deleted when the dataset is closed or when the JVM exits. The temporary file will be created in the default temporary-file directory, using the prefix "dataset-" and suffix ".mvstore" to generate its name.

Parameters:

pipeline - the source of the dataset's data

createTempDataset

public static MvStoreDataset createTempDataset(File databaseFolder,
                                               AbstractPipeline pipeline)

Creates a temporary dataset on disk that will be deleted when the dataset is closed or when the JVM exits. The temporary file will be created in the specified directory (or the default temporary-file directory if null), using the prefix "dataset-" and suffix ".mvstore" to generate its name.

Parameters:

databaseFolder - the directory to store the temporary file or null to use the default temporary-file directory.

pipeline - the source of the dataset's data

createDataset

public static MvStoreDataset createDataset(File databaseFile,
                                           AbstractPipeline pipeline)

Creates a persistent dataset on disk in the specified databaseFile. The file will remain on disk even after the dataset is closed and the JVM exits.

Parameters:

databaseFile - the file to store the dataset's contents.

pipeline - the source of the dataset's data

openDataset

public static MvStoreDataset openDataset(File databaseFile)

Loads an existing dataset from the specified databaseFile.

Parameters:

databaseFile - the existing file to containing the dataset's contents to load.

setPipeline

public MvStoreDataset setPipeline(AbstractPipeline pipeline)

Overrides:

setPipeline in class Dataset

getDatabaseFile

public File getDatabaseFile()

isDeleteDatabaseFileOnClose

public boolean isDeleteDatabaseFileOnClose()

setDeleteDatabaseFileOnClose

public MvStoreDataset setDeleteDatabaseFileOnClose(boolean deleteDatabaseFileOnClose)

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class Dataset

getRecordCount

public long getRecordCount()

Specified by:

getRecordCount in class Dataset

getRecord

public Record getRecord(long index)

Specified by:

getRecord in class Dataset

getColumnCount

public long getColumnCount()

Specified by:

getColumnCount in class Dataset

getColumnNames

public List<String> getColumnNames()

Specified by:

getColumnNames in class Dataset

getColumn

public Column getColumn(int index)

Specified by:

getColumn in class Dataset

getColumn

public Column getColumn(String name)

Specified by:

getColumn in class Dataset

getOrCreateColumn

protected Column getOrCreateColumn(String name,
                                   int index)

Specified by:

getOrCreateColumn in class Dataset

getColumns

public List<Column> getColumns()

Specified by:

getColumns in class Dataset

addField

protected Column addField(Record record,
                          Field field,
                          int fieldIndex)

Overrides:

addField in class Dataset

beforeLoad

protected void beforeLoad()

Description copied from class: Dataset

Called at the start of the data loading process, but before any records or column stats have been loaded.

Specified by:

beforeLoad in class Dataset

afterLoad

protected void afterLoad()

Description copied from class: Dataset

Called at the end of the data loading process after all the records and column stats have been loaded.

Specified by:

afterLoad in class Dataset

createDataWriter

protected DataWriter createDataWriter()

Description copied from class: Dataset

Writes records to this dataset's cache after clearing it.

Specified by:

createDataWriter in class Dataset

load

public Dataset load(Integer maxRecords,
                    JobCallback<DataReader,DataWriter> callback)

Description copied from class: Dataset

Starts the asynchronous loading of records from the pipeline into this dataset. This method returns immediately and does not wait for loading to complete. See Dataset.waitForRecordsToLoad() and Dataset.waitForRecordsToLoad(long, long).

Overrides:

load in class Dataset

Parameters:

maxRecords - the maximum records to load or null to load all records.

callback - the object to notify as data is being loaded.