com.northconcepts.datapipeline.parquet.input

Class ParquetPipelineInput

java.lang.Object

com.northconcepts.datapipeline.foundations.core.Bean

com.northconcepts.datapipeline.foundations.core.FoundationObject

com.northconcepts.datapipeline.foundations.pipeline.PipelineObject

com.northconcepts.datapipeline.foundations.pipeline.PipelineInput

com.northconcepts.datapipeline.foundations.pipeline.input.FileSourcePipelineInput

com.northconcepts.datapipeline.parquet.input.ParquetPipelineInput

All Implemented Interfaces:

DataExceptionContributor, JsonSerializable, RecordSerializable, XmlSerializable, DataReaderFactory, JavaCodeGenerator, Serializable

public class ParquetPipelineInput
extends FileSourcePipelineInput

See Also:

Serialized Form

Field Summary

Fields inherited from class com.northconcepts.datapipeline.foundations.core.FoundationObject

internalId, internalName, log, TIMESTAMP_FORMAT

Fields inherited from interface com.northconcepts.datapipeline.core.RecordSerializable

SERIALIZED_CLASS_NAME, TYPE

Fields inherited from interface com.northconcepts.datapipeline.core.XmlSerializable

XML_SERIALIZED_CLASS_NAME

Constructor Summary

Constructors

Constructor and Description
ParquetPipelineInput()

Method Summary

Modifier and Type	Method and Description
DataReader	createDataReader()
ParquetPipelineInput	fromRecord(Record source) Loads this instance's state from a record and returns this (for fluid API call chaining).
ParquetPipelineInput	fromXmlElement(Element pipelineInputElement)
void	generateJavaCode(JavaCodeBuilder code)
Map<String,String>	getConfiguration() Returns the Parquet configuration parameters.
FileSource	getFileSource()
MetadataFilter	getFilter()
String	getName()
MessageType	getSchema() Indicates the schema used to read the file.
boolean	isDebug() Indicates if debugging is enabled to print log statements.
boolean	isLineageSupported()
boolean	isMakeOptionalFieldsRequired()
boolean	isMakeRequiredFieldsOptional()
boolean	isRemoveFieldsWithoutColumnMetadata()
boolean	isRemoveFieldsWithoutValues()
ParquetPipelineInput	setConfiguration(Map<String,String> configuration) Sets the Parquet configuration parameters.
ParquetPipelineInput	setConfiguration(String key, String value)
ParquetPipelineInput	setDebug(boolean debug) Indicates if debugging is enabled to print log statements.
ParquetPipelineInput	setFileSource(FileSource fileSource)
ParquetPipelineInput	setFilter(MetadataFilter filter)
ParquetPipelineInput	setMakeOptionalFieldsRequired(boolean makeOptionalFieldsRequired)
ParquetPipelineInput	setMakeRequiredFieldsOptional(boolean makeRequiredFieldsOptional)
ParquetPipelineInput	setRemoveFieldsWithoutColumnMetadata(boolean removeFieldsWithoutColumnMetadata)
ParquetPipelineInput	setRemoveFieldsWithoutValues(boolean removeFieldsWithoutValues)
ParquetPipelineInput	setSchema(MessageType schema) Indicates the schema used to read the file.
Record	toRecord()
Element	toXmlElement(Document document)

Methods inherited from class com.northconcepts.datapipeline.foundations.pipeline.PipelineInput

getNestedPipelineInput, getRootPipelineInput, isSaveLineage, setSaveLineage

Methods inherited from class com.northconcepts.datapipeline.foundations.core.FoundationObject

addExceptionProperties, assertValid, assertValid, clone, exception, exception, exception, getInternalId, getInternalName, resetInternalId

Methods inherited from class com.northconcepts.datapipeline.foundations.core.Bean

toString

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface com.northconcepts.datapipeline.core.RecordSerializable

fromArrayValue, fromArrayValue, fromArrayValue, fromJson, fromJson, newInstanceFromRecord, newInstanceFromRecord, toArrayValue, toJson, toJson, toJson, toRecord

Methods inherited from interface com.northconcepts.datapipeline.core.JsonSerializable

newInstanceFromJson

Methods inherited from interface com.northconcepts.datapipeline.core.XmlSerializable

forEachGrandChildElement, fromXml, fromXml, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getAttribute, getChildElement, getChildElements, newInstanceFromXml, newInstanceFromXml, setAttribute, toXml, toXml, toXml, toXml, toXml, writeXml, writeXml

Methods inherited from interface com.northconcepts.datapipeline.core.DataExceptionContributor

addExceptionProperties, contributeExceptionProperties, contributeExceptionProperties

Constructor Detail

ParquetPipelineInput

public ParquetPipelineInput()

Method Detail

getName

public String getName()

Overrides:

getName in class FileSourcePipelineInput

getFileSource

public FileSource getFileSource()

Specified by:

getFileSource in class FileSourcePipelineInput

setFileSource

public ParquetPipelineInput setFileSource(FileSource fileSource)

Specified by:

setFileSource in class FileSourcePipelineInput

createDataReader

public DataReader createDataReader()

generateJavaCode

public void generateJavaCode(JavaCodeBuilder code)

isLineageSupported

public boolean isLineageSupported()

Overrides:

isLineageSupported in class PipelineInput

isDebug

public boolean isDebug()

Indicates if debugging is enabled to print log statements.

setDebug

public ParquetPipelineInput setDebug(boolean debug)

Indicates if debugging is enabled to print log statements.

getConfiguration

public Map<String,String> getConfiguration()

Returns the Parquet configuration parameters.

setConfiguration

public ParquetPipelineInput setConfiguration(Map<String,String> configuration)

Sets the Parquet configuration parameters.

setConfiguration

public ParquetPipelineInput setConfiguration(String key,
                                             String value)

getFilter

public MetadataFilter getFilter()

setFilter

public ParquetPipelineInput setFilter(MetadataFilter filter)

getSchema

public MessageType getSchema()

Indicates the schema used to read the file.

setSchema

public ParquetPipelineInput setSchema(MessageType schema)

Indicates the schema used to read the file.

isMakeRequiredFieldsOptional

public boolean isMakeRequiredFieldsOptional()

setMakeRequiredFieldsOptional

public ParquetPipelineInput setMakeRequiredFieldsOptional(boolean makeRequiredFieldsOptional)

isMakeOptionalFieldsRequired

public boolean isMakeOptionalFieldsRequired()

setMakeOptionalFieldsRequired

public ParquetPipelineInput setMakeOptionalFieldsRequired(boolean makeOptionalFieldsRequired)

isRemoveFieldsWithoutColumnMetadata

public boolean isRemoveFieldsWithoutColumnMetadata()

setRemoveFieldsWithoutColumnMetadata

public ParquetPipelineInput setRemoveFieldsWithoutColumnMetadata(boolean removeFieldsWithoutColumnMetadata)

isRemoveFieldsWithoutValues

public boolean isRemoveFieldsWithoutValues()

setRemoveFieldsWithoutValues

public ParquetPipelineInput setRemoveFieldsWithoutValues(boolean removeFieldsWithoutValues)

toRecord

public Record toRecord()

Specified by:

toRecord in interface RecordSerializable

Overrides:

toRecord in class PipelineInput

fromRecord

public ParquetPipelineInput fromRecord(Record source)

Description copied from interface: RecordSerializable

Loads this instance's state from a record and returns this (for fluid API call chaining). For fluid API call chaining, the overridden method should change the declared return type to its class.

Specified by:

fromRecord in interface RecordSerializable

Overrides:

fromRecord in class PipelineInput

Returns:

this instance.

toXmlElement

public Element toXmlElement(Document document)

Specified by:

toXmlElement in interface XmlSerializable

Overrides:

toXmlElement in class PipelineInput

fromXmlElement

public ParquetPipelineInput fromXmlElement(Element pipelineInputElement)

Specified by:

fromXmlElement in interface XmlSerializable

Overrides:

fromXmlElement in class PipelineInput