com.northconcepts.datapipeline.avro

Class AvroWriter

java.lang.Object

com.northconcepts.datapipeline.core.DataObject

com.northconcepts.datapipeline.core.Endpoint

com.northconcepts.datapipeline.core.DataEndpoint

com.northconcepts.datapipeline.core.DataWriter

com.northconcepts.datapipeline.internal.lang.IntegrationWriter

com.northconcepts.datapipeline.avro.AvroWriter

public class AvroWriter
extends IntegrationWriter

Write Records to an Apache Avro file.

Nested Class Summary

Nested classes/interfaces inherited from class com.northconcepts.datapipeline.core.DataEndpoint

DataEndpoint.State

Field Summary

Fields inherited from class com.northconcepts.datapipeline.core.DataEndpoint

lastRecord, PRODUCT, PRODUCT_VERSION, VENDOR, XML_INPUT_FACTORY_KEY

Fields inherited from class com.northconcepts.datapipeline.core.Endpoint

BUFFER_SIZE, captureElapsedTime, DEFAULT_READ_BUFFER_SIZE

Fields inherited from class com.northconcepts.datapipeline.core.DataObject

id, log, name, TIMESTAMP_FORMAT

Constructor Summary

Constructors

Constructor and Description
AvroWriter(File file, DataWriter discardWriter) Write Avro data to the specified file.
AvroWriter(OutputStream outputStream, DataWriter discardWriter) Write Avro data to an OutputStream.

Method Summary

Modifier and Type	Method and Description
DataException	addExceptionProperties(DataException exception) Adds this endpoint's current state to a DataException.
void	close() Indicates that this endpoint has finished reading or writing.
int	getDefaultBigDecimalScale() Returns the default scale used when writing BigDecimal values (default 5).
int	getDefaultBigNumberPrecision() Returns the default precision used when writing BigDecimal & BigInteger values (default 25).
long	getDiscardedRecordCount() Returns the number of discarded Records
boolean	isRemoveUnsupportedChars() Returns whether unsupported characters are removed from field names.
boolean	isWriteUuidAsString() Indicates whether UUID values are written as strings or bytes.
void	open() Makes this endpoint ready for reading or writing.
AvroWriter	setDefaultBigDecimalScale(int defaultBigDecimalScale) Sets the default scale used when writing BigDecimal values (default 5).
AvroWriter	setDefaultBigNumberPrecision(int defaultBigNumberPrecision) Sets the default precision used when writing BigDecimal & BigInteger values (default 25).
AvroWriter	setRemoveUnsupportedChars(boolean removeUnsupportedChars) Sets whether unsupported characters are removed from field names.
AvroWriter	setWriteUuidAsString(boolean writeUuidAsString) Indicates whether UUID values are written as strings or bytes.
protected void	writeImpl(Record record) Overridden by subclasses to write the specified record to this DataWriter.

Methods inherited from class com.northconcepts.datapipeline.core.DataWriter

available, getNestedEndpoint, getNestedWriter, getRootEndpoint, getRootWriter, getWriter, write

Methods inherited from class com.northconcepts.datapipeline.core.DataEndpoint

decrementRecordCount, enableJmx, getLastRecord, getRecordCount, getRecordCountAsBigInteger, getRecordCountAsString, incrementRecordCount, isRecordCountBigInteger, resetRecordCount, toString

Methods inherited from class com.northconcepts.datapipeline.core.Endpoint

addElapsedtime, assertClosed, assertNotOpened, assertOpened, finalize, getClosedOn, getDescription, getElapsedTime, getElapsedTimeAsString, getOpenedOn, getOpenElapsedTime, getOpenElapsedTimeAsString, getSelfTime, getSelfTimeAsString, getState, isCaptureElapsedTime, isClosed, isOpen, setCaptureElapsedTime, setDescription

Methods inherited from class com.northconcepts.datapipeline.core.DataObject

exception, exception, exception, getId, getName, resetID

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

AvroWriter

public AvroWriter(File file,
                  DataWriter discardWriter)

Write Avro data to the specified file.

Parameters:

file - the file to write to

discardWriter - discard sink for records not conforming to the Avro schema

AvroWriter

public AvroWriter(OutputStream outputStream,
                  DataWriter discardWriter)

Write Avro data to an OutputStream.

Parameters:

outputStream - the target stream

discardWriter - discard sink for records not conforming to the Avro schema

Method Detail

getDiscardedRecordCount

public long getDiscardedRecordCount()

Returns the number of discarded Records

Returns:

long number of discarded records

isWriteUuidAsString

public boolean isWriteUuidAsString()

Indicates whether UUID values are written as strings or bytes. If set to false, UUID values will be written as bytes.

Default value is true.

setWriteUuidAsString

public AvroWriter setWriteUuidAsString(boolean writeUuidAsString)

Indicates whether UUID values are written as strings or bytes. If set to false, UUID values will be written as bytes.

Default value is true.

getDefaultBigDecimalScale

public int getDefaultBigDecimalScale()

Returns the default scale used when writing BigDecimal values (default 5).

setDefaultBigDecimalScale

public AvroWriter setDefaultBigDecimalScale(int defaultBigDecimalScale)

Sets the default scale used when writing BigDecimal values (default 5).

getDefaultBigNumberPrecision

public int getDefaultBigNumberPrecision()

Returns the default precision used when writing BigDecimal & BigInteger values (default 25).

setDefaultBigNumberPrecision

public AvroWriter setDefaultBigNumberPrecision(int defaultBigNumberPrecision)

Sets the default precision used when writing BigDecimal & BigInteger values (default 25).

isRemoveUnsupportedChars

public boolean isRemoveUnsupportedChars()

Returns whether unsupported characters are removed from field names.

setRemoveUnsupportedChars

public AvroWriter setRemoveUnsupportedChars(boolean removeUnsupportedChars)

Sets whether unsupported characters are removed from field names.

open

public void open()
          throws DataException

Description copied from class: DataEndpoint

Makes this endpoint ready for reading or writing.

Overrides:

open in class IntegrationWriter

Throws:

DataException

close

public void close()
           throws DataException

Description copied from class: DataEndpoint

Indicates that this endpoint has finished reading or writing.

Overrides:

close in class DataEndpoint

Throws:

DataException

writeImpl

protected void writeImpl(Record record)
                  throws Throwable

Description copied from class: DataWriter

Overridden by subclasses to write the specified record to this DataWriter.

Specified by:

writeImpl in class DataWriter

Throws:

Throwable

addExceptionProperties

public DataException addExceptionProperties(DataException exception)

Description copied from class: Endpoint

Adds this endpoint's current state to a DataException. Since this method is called whenever an exception is thrown, subclasses should override it to add their specific information.

Overrides:

addExceptionProperties in class DataWriter