com.northconcepts.datapipeline.foundations.pipeline.dataset

Class MemoryDataset

java.lang.Object

com.northconcepts.datapipeline.foundations.core.Bean

com.northconcepts.datapipeline.foundations.core.FoundationObject

com.northconcepts.datapipeline.foundations.pipeline.dataset.Dataset

com.northconcepts.datapipeline.foundations.pipeline.dataset.MemoryDataset

All Implemented Interfaces:

DataExceptionContributor, JsonSerializable, RecordSerializable, XmlSerializable, Closeable, Serializable, AutoCloseable, Iterable<Record>

public class MemoryDataset
extends Dataset

Caches the dataset's records in RAM.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class com.northconcepts.datapipeline.foundations.pipeline.dataset.Dataset

Dataset.ColumnsDataReader

Field Summary

Fields inherited from class com.northconcepts.datapipeline.foundations.core.FoundationObject

internalId, internalName, log, TIMESTAMP_FORMAT

Fields inherited from interface com.northconcepts.datapipeline.core.RecordSerializable

SERIALIZED_CLASS_NAME, TYPE

Fields inherited from interface com.northconcepts.datapipeline.core.XmlSerializable

XML_SERIALIZED_CLASS_NAME

Constructor Summary

Constructors

Constructor and Description
MemoryDataset(AbstractPipeline pipeline)

Method Summary

Modifier and Type	Method and Description
protected void	afterLoad() Called at the end of the data loading process after all the records and column stats have been loaded.
protected void	beforeLoad() Called at the start of the data loading process, but before any records or column stats have been loaded.
void	close()
protected DataWriter	createDataWriter() Writes records to this dataset's cache after clearing it.
Column	getColumn(int index)
Column	getColumn(String name)
long	getColumnCount()
List<String>	getColumnNames()
List<Column>	getColumns()
protected Column	getOrCreateColumn(String name, int index)
Record	getRecord(long index)
long	getRecordCount()
MemoryDataset	setPipeline(AbstractPipeline pipeline)

Methods inherited from class com.northconcepts.datapipeline.foundations.pipeline.dataset.Dataset

addField, afterColumnStatsLoaded, afterRecordsLoaded, cancelLoad, createColumnsDataReader, createDataReader, createDataReader, finalize, forEach, fromRecord, getColumnStatsException, getColumnStatsReaderThreads, getDataLoadException, getJob, getJobExecutor, getMaxColumnStatsRecords, getMaxColumnsToAnalyze, getMaxRecordsToLoad, getPipeline, getRecordList, isCollectUniqueValues, isColumnStatsLoaded, isDataLoaded, isDataLoading, isDetectBigNumberValues, isDetectBooleanValues, isDetectNumericValues, isDetectTemporalValues, isDetectUuidValues, isInferStringTypes, isRecordsLoaded, iterator, load, load, load, setCollectUniqueValues, setColumnStatsLoaded, setColumnStatsReaderThreads, setDetectBigNumberValues, setDetectBooleanValues, setDetectNumericValues, setDetectTemporalValues, setDetectUuidValues, setInferStringTypes, setJobExecutor, setMaxColumnStatsRecords, setMaxColumnsToAnalyze, setRecordsLoaded, stream, toRecord, updateColumns, waitForColumnStatsToLoad, waitForColumnStatsToLoad, waitForRecordsToLoad, waitForRecordsToLoad, waitUntilJobFinished

Methods inherited from class com.northconcepts.datapipeline.foundations.core.FoundationObject

addExceptionProperties, assertValid, assertValid, clone, exception, exception, exception, getInternalId, getInternalName, resetInternalId

Methods inherited from class com.northconcepts.datapipeline.foundations.core.Bean

toString

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface java.lang.Iterable

spliterator

Methods inherited from interface com.northconcepts.datapipeline.core.RecordSerializable

fromArrayValue, fromArrayValue, fromArrayValue, fromJson, fromJson, newInstanceFromRecord, newInstanceFromRecord, toArrayValue, toJson, toJson, toJson, toRecord

Methods inherited from interface com.northconcepts.datapipeline.core.JsonSerializable

newInstanceFromJson

Methods inherited from interface com.northconcepts.datapipeline.core.XmlSerializable

forEachGrandChildElement, fromXml, fromXml, fromXmlElement, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getChildElement, getChildElements, newInstanceFromXml, newInstanceFromXml, setAttribute, toXml, toXml, toXml, toXml, toXml, toXmlElement, writeXml, writeXml

Methods inherited from interface com.northconcepts.datapipeline.core.DataExceptionContributor

addExceptionProperties, contributeExceptionProperties, contributeExceptionProperties

Constructor Detail

MemoryDataset

public MemoryDataset(AbstractPipeline pipeline)

Method Detail

setPipeline

public MemoryDataset setPipeline(AbstractPipeline pipeline)

Overrides:

setPipeline in class Dataset

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class Dataset

getRecordCount

public long getRecordCount()

Specified by:

getRecordCount in class Dataset

getRecord

public Record getRecord(long index)

Specified by:

getRecord in class Dataset

getColumnCount

public long getColumnCount()

Specified by:

getColumnCount in class Dataset

getColumnNames

public List<String> getColumnNames()

Specified by:

getColumnNames in class Dataset

getColumn

public Column getColumn(int index)

Specified by:

getColumn in class Dataset

getColumn

public Column getColumn(String name)

Specified by:

getColumn in class Dataset

getOrCreateColumn

protected Column getOrCreateColumn(String name,
                                   int index)

Specified by:

getOrCreateColumn in class Dataset

getColumns

public List<Column> getColumns()

Specified by:

getColumns in class Dataset

beforeLoad

protected void beforeLoad()

Description copied from class: Dataset

Called at the start of the data loading process, but before any records or column stats have been loaded.

Specified by:

beforeLoad in class Dataset

afterLoad

protected void afterLoad()

Description copied from class: Dataset

Called at the end of the data loading process after all the records and column stats have been loaded.

Specified by:

afterLoad in class Dataset

createDataWriter

protected DataWriter createDataWriter()

Description copied from class: Dataset

Writes records to this dataset's cache after clearing it.

Specified by:

createDataWriter in class Dataset