com.northconcepts.datapipeline.foundations.pipeline.dataset

Class LocalFileDataset

java.lang.Object

com.northconcepts.datapipeline.foundations.core.Bean

com.northconcepts.datapipeline.foundations.core.FoundationObject

com.northconcepts.datapipeline.foundations.pipeline.dataset.Dataset

com.northconcepts.datapipeline.foundations.pipeline.dataset.LocalFileDataset

All Implemented Interfaces:

DataExceptionContributor, JsonSerializable, RecordSerializable, XmlSerializable, Closeable, Serializable, AutoCloseable, Iterable<Record>

public class LocalFileDataset
extends Dataset

Caches the dataset's records on disk as binary data.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class com.northconcepts.datapipeline.foundations.pipeline.dataset.Dataset

Dataset.ColumnsDataReader

Field Summary

Fields inherited from class com.northconcepts.datapipeline.foundations.core.FoundationObject

internalId, internalName, log, TIMESTAMP_FORMAT

Fields inherited from interface com.northconcepts.datapipeline.core.RecordSerializable

SERIALIZED_CLASS_NAME, TYPE

Fields inherited from interface com.northconcepts.datapipeline.core.XmlSerializable

XML_SERIALIZED_CLASS_NAME

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	LocalFileDataset(String rootFolder, String fileNamePrefix)
protected	LocalFileDataset(String rootFolder, String fileNamePrefix, AbstractPipeline pipeline)

Method Summary

Modifier and Type	Method and Description
DataException	addExceptionProperties(DataException exception) Adds this object's current state to a DataException.
protected void	afterColumnStatsLoaded() Called during the data loading process after all the column stats have been loaded.
protected void	afterLoad() Called at the end of the data loading process after all the records and column stats have been loaded.
protected void	beforeLoad() Called at the start of the data loading process, but before any records or column stats have been loaded.
void	close()
static LocalFileDataset	createDataset(String rootFolder, String fileNamePrefix, AbstractPipeline pipeline) Creates a persistent dataset on disk in the specified rootFolder directory.
static LocalFileDataset	createDataset(String rootFolder, String fileNamePrefix, DataReaderFactory dataReaderFactory) Creates a persistent dataset on disk in the specified rootFolder directory.
protected DataWriter	createDataWriter() Writes records to this dataset's cache after clearing it.
static LocalFileDataset	createTempDataset(String rootFolder, String fileNamePrefix, AbstractPipeline pipeline) Creates a temporary dataset on disk that will be deleted when the dataset is closed or when the JVM exits.
static LocalFileDataset	createTempDataset(String rootFolder, String fileNamePrefix, DataReaderFactory dataReaderFactory) Creates a temporary dataset on disk that will be deleted when the dataset is closed or when the JVM exits.
Column	getColumn(int index)
Column	getColumn(String name)
long	getColumnCount()
List<String>	getColumnNames()
List<Column>	getColumns()
protected File	getDataFile(long fileIndex)
protected File	getMetadataFile()
protected Column	getOrCreateColumn(String name, int index)
Record	getRecord(long index)
long	getRecordCount()
RecordList	getRecordList(long offset, int count) Get a subset of the records cached in this dataset.
int	getRecordsPerFile()
protected File	getStatsFile()
boolean	isDeleteFilesOnClose()
LocalFileDataset	load(Integer maxRecords, JobCallback<DataReader,DataWriter> callback) Starts the asynchronous loading of records from the pipeline into this dataset.
static LocalFileDataset	openDataset(String rootFolder, String fileNamePrefix) Loads an existing dataset from the specified rootFolder & fileNamePrefix.
LocalFileDataset	setDeleteFilesOnClose(boolean deleteFilesOnClose)
LocalFileDataset	setPipeline(AbstractPipeline pipeline)
LocalFileDataset	setRecordsPerFile(int recordsPerFile)

Methods inherited from class com.northconcepts.datapipeline.foundations.pipeline.dataset.Dataset

addField, afterRecordsLoaded, cancelLoad, createColumnsDataReader, createDataReader, createDataReader, finalize, forEach, fromRecord, getColumnStatsException, getColumnStatsReaderThreads, getDataLoadException, getJob, getMaxColumnStatsRecords, getMaxRecordsToLoad, getPipeline, isCollectUniqueValues, isColumnStatsLoaded, isDataLoaded, isDataLoading, isDetectBigNumberValues, isDetectBooleanValues, isDetectNumericValues, isDetectTemporalValues, isDetectUuidValues, isInferStringTypes, isRecordsLoaded, iterator, load, load, setCollectUniqueValues, setColumnStatsLoaded, setColumnStatsReaderThreads, setDetectBigNumberValues, setDetectBooleanValues, setDetectNumericValues, setDetectTemporalValues, setDetectUuidValues, setInferStringTypes, setMaxColumnStatsRecords, setRecordsLoaded, stream, toRecord, updateColumns, waitForColumnStatsToLoad, waitForColumnStatsToLoad, waitForRecordsToLoad, waitForRecordsToLoad, waitUntilJobFinished

Methods inherited from class com.northconcepts.datapipeline.foundations.core.FoundationObject

assertValid, assertValid, clone, exception, exception, exception, getInternalId, getInternalName, resetInternalId

Methods inherited from class com.northconcepts.datapipeline.foundations.core.Bean

toString

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface java.lang.Iterable

spliterator

Methods inherited from interface com.northconcepts.datapipeline.core.RecordSerializable

fromArrayValue, fromArrayValue, fromArrayValue, fromJson, fromJson, newInstanceFromRecord, newInstanceFromRecord, toArrayValue, toJson, toJson, toJson, toRecord

Methods inherited from interface com.northconcepts.datapipeline.core.JsonSerializable

newInstanceFromJson

Methods inherited from interface com.northconcepts.datapipeline.core.XmlSerializable

forEachGrandChildElement, fromXml, fromXml, fromXmlElement, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getChildElement, getChildElements, newInstanceFromXml, newInstanceFromXml, setAttribute, toXml, toXml, toXml, toXml, toXml, toXmlElement, writeXml, writeXml

Methods inherited from interface com.northconcepts.datapipeline.core.DataExceptionContributor

addExceptionProperties, contributeExceptionProperties, contributeExceptionProperties

Constructor Detail

LocalFileDataset

protected LocalFileDataset(String rootFolder,
                           String fileNamePrefix,
                           AbstractPipeline pipeline)

LocalFileDataset

protected LocalFileDataset(String rootFolder,
                           String fileNamePrefix)

Method Detail

createTempDataset

public static LocalFileDataset createTempDataset(String rootFolder,
                                                 String fileNamePrefix,
                                                 DataReaderFactory dataReaderFactory)

Creates a temporary dataset on disk that will be deleted when the dataset is closed or when the JVM exits. The temporary file will be created in the specified directory (or the default temporary-file directory if null), using the prefix "dataset-" and suffix ".mvstore" to generate its name.

Parameters:

rootFolder - the directory to store the temporary file or null to use the default temporary-file directory.

fileNamePrefix - the base file name of data & stats.

dataReaderFactory - the source of the dataset's data

createTempDataset

public static LocalFileDataset createTempDataset(String rootFolder,
                                                 String fileNamePrefix,
                                                 AbstractPipeline pipeline)

Creates a temporary dataset on disk that will be deleted when the dataset is closed or when the JVM exits. The temporary files will be created in the default temporary-file or specified directory, using the specified prefix.

Parameters:

rootFolder - the directory to store the temporary file or null to use the default temporary-file directory.

fileNamePrefix - the base file name of data & stats.

pipeline - the source of the dataset's data

createDataset

public static LocalFileDataset createDataset(String rootFolder,
                                             String fileNamePrefix,
                                             DataReaderFactory dataReaderFactory)

Creates a persistent dataset on disk in the specified rootFolder directory. The data & stats files will remain on disk even after the dataset is closed and the JVM exits.

Parameters:

rootFolder - the root directory where data & stats files will be saved.

fileNamePrefix - the base file name of data & stats.

dataReaderFactory - the source of the dataset's data

createDataset

public static LocalFileDataset createDataset(String rootFolder,
                                             String fileNamePrefix,
                                             AbstractPipeline pipeline)

Creates a persistent dataset on disk in the specified rootFolder directory. The data & stats files will remain on disk even after the dataset is closed and the JVM exits.

Parameters:

rootFolder - the root directory where data & stats files will be saved.

fileNamePrefix - the base file name of data & stats.

pipeline - the source of the dataset's data

openDataset

public static LocalFileDataset openDataset(String rootFolder,
                                           String fileNamePrefix)

Loads an existing dataset from the specified rootFolder & fileNamePrefix.

Parameters:

rootFolder - the root directory where data & stats files are available.

fileNamePrefix - the base file name of data & stats.

getDataFile

protected File getDataFile(long fileIndex)

getStatsFile

protected File getStatsFile()

getMetadataFile

protected File getMetadataFile()

setPipeline

public LocalFileDataset setPipeline(AbstractPipeline pipeline)

Overrides:

setPipeline in class Dataset

isDeleteFilesOnClose

public boolean isDeleteFilesOnClose()

setDeleteFilesOnClose

public LocalFileDataset setDeleteFilesOnClose(boolean deleteFilesOnClose)

getRecordsPerFile

public int getRecordsPerFile()

setRecordsPerFile

public LocalFileDataset setRecordsPerFile(int recordsPerFile)

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class Dataset

getRecordCount

public long getRecordCount()

Specified by:

getRecordCount in class Dataset

getRecord

public Record getRecord(long index)

Specified by:

getRecord in class Dataset

getRecordList

public RecordList getRecordList(long offset,
                                int count)

Description copied from class: Dataset

Get a subset of the records cached in this dataset.

Overrides:

getRecordList in class Dataset

getColumnCount

public long getColumnCount()

Specified by:

getColumnCount in class Dataset

getColumnNames

public List<String> getColumnNames()

Specified by:

getColumnNames in class Dataset

getColumn

public Column getColumn(int index)

Specified by:

getColumn in class Dataset

getColumn

public Column getColumn(String name)

Specified by:

getColumn in class Dataset

getOrCreateColumn

protected Column getOrCreateColumn(String name,
                                   int index)

Specified by:

getOrCreateColumn in class Dataset

getColumns

public List<Column> getColumns()

Specified by:

getColumns in class Dataset

beforeLoad

protected void beforeLoad()

Description copied from class: Dataset

Called at the start of the data loading process, but before any records or column stats have been loaded.

Specified by:

beforeLoad in class Dataset

afterLoad

protected void afterLoad()

Description copied from class: Dataset

Called at the end of the data loading process after all the records and column stats have been loaded.

Specified by:

afterLoad in class Dataset

afterColumnStatsLoaded

protected void afterColumnStatsLoaded()

Description copied from class: Dataset

Called during the data loading process after all the column stats have been loaded. The records would have already been loaded when this method is called since column stats require additional processing.

Overrides:

afterColumnStatsLoaded in class Dataset

createDataWriter

protected DataWriter createDataWriter()

Description copied from class: Dataset

Writes records to this dataset's cache after clearing it.

Specified by:

createDataWriter in class Dataset

load

public LocalFileDataset load(Integer maxRecords,
                             JobCallback<DataReader,DataWriter> callback)

Description copied from class: Dataset

Starts the asynchronous loading of records from the pipeline into this dataset. This method returns immediately and does not wait for loading to complete. See Dataset.waitForRecordsToLoad() and Dataset.waitForRecordsToLoad(long, long).

Overrides:

load in class Dataset

Parameters:

maxRecords - the maximum records to load or null to load all records.

callback - the object to notify as data is being loaded.

addExceptionProperties

public DataException addExceptionProperties(DataException exception)

Description copied from class: FoundationObject

Adds this object's current state to a DataException. Since this method is called whenever an exception is thrown, subclasses should override it to add their specific information.

Specified by:

addExceptionProperties in interface DataExceptionContributor

Overrides:

addExceptionProperties in class FoundationObject