Category Archives: News

Data Pipeline 2.2.8 Now Available

  • added TemplateWriter for writing text streams using FreeMarker templates
  • added new examples for writing XML and HTML files using TemplateWriter
  • BUGFIX: XmlWriter’s (XmlTemplate, File) constructor now calls setFieldNamesInFirstRow(false) by default
  • BUGFIX: The JxlProvider now converts intervals and user-defined types to string when generating Excel files
  • Intervals are no longer converted to strings when added to a field/record
  • BasicFieldTransformer can now convert numbers to intervals (seconds, months, days, minutes, etc.)
  • JdbcWriter now has public accessors for connection, tableName, batchMode, and jdbcTypes
  • individual fields can now be removed from a FieldList
  • FieldList can now accept collections of strings
  • updated Apache POI to v3.8

Data Pipeline 2.2.7 Now Available

  • added JdbcMultiWriter for multi-threaded writing to one or more database connections concurrently
  • added multi-threaded AsyncWriter to compliment AsyncReader
  • data writers now have an available() method to indicate the number of records that can probably be written without blocking
  • MultiWriter now supports configurable write strategies (ReplicateWriteStrategy, RoundRobinWriteStrategy, AvailableCapacityWriteStrategy, and user defined)
  • added support for CLOB fields (see JdbcValueReader.DEFAULT)
  • Field and Record’s toString() methods now limit displayed strings to the first 128 characters
  • RecordMeter is now public and returned by MeteredReader and MeteredWriter’s getMeter() method
  • BUGFIX: record count is no longer off by 1 in some cases

Data Pipeline 2.2.6 Now Available

  • performance improvements in CSV and fixed width handling
  • untyped expression evaluation is now based on the value’s type, instead of the field’s declared type
  • BUGFIX: now handles untyped expressions between primitive and object values
  • float expressions are now upgraded to doubles during evaluation
  • all non doubles and floats numbers are now upgraded to longs during evaluation
  • expressions can now reference Java beans, not just primitive values
  • method call expression now finds the most appropriate method based on the runtime argument types (
  • improved handling for collections and arrays in DataException properties
  • Apache PoiProvider can now distinguish between date, time, and datetimes fields in Excel

Data Pipeline 2.2.5 Now Available

  • Added JavaBeanReader whice uses XPath expressions to identify field values and break records (see the Read from Java beans example)
  • AbstractReader’s setStartingRow and setLastRow no return this
  • Filter rule IsInstanceOfJavaType now returns false for null values
  • Added number-to-date methods to BasicFieldTransformer (numberToDate(), minutesToDate(), hoursToDate(), and daysToDate())
  • BasicFieldTransformer.Operation and BasicFieldTransformer.StringOperation are now public classes
  • BasicFieldTransformer.add(Operation … operation) is now public
  • ConditionalTransformer is now private (use TransformingReader.filter instead)
  • TransformingReader now contains an optional Filter, allowing any transformer to be conditionally applied
  • Removed TransformingReader.add(Filter filter, Transformer … transformer) method

Data Pipeline 2.2.1 Now Available

Data Pipeline 2.2.1 is now available with the following enhancements:

  • added batch execution to JdbcWriter (see JdbcWriter.setBatchSize)
  • added callback mechanism to track job progress (see JobTemplate.transfer(R reader, W writer, boolean async, JobCallback callback))
  • early access to DeMuxReader

Data Pipeline 2.2 Released

We are happy to announce that Data Pipeline 2.2 is now released. We have added the enhancements below:

  • added an XPath-based XmlReader
  • updates so that Excel now defaults to the Apache POI instead of JXL
  • updates so that the following classes use java.util.List instead of java.util.ArrayList in their public APIs: CompositeValue, FieldList, Lookup, LookupTransformer, Record, RecordList

Data Pipeline 2.1 Released

We are happy to announce that Data Pipeline 2.1 is now released. We have added a great deal of enhancements below:

  • support for Excel 2003 XLS files
  • support for Excel XLSX (XML format) files
  • added FixedWidthReader.setLastFieldConsumesRemaining(boolean lastFieldConsumesRemaining) functionality
  • added ExcelReader.setUseSheetColumnCount(boolean useSheetColumnCount) functionality
  • added more string utils to BasicFieldTransformer
  • added ConditionalTransformer class
  • added TransformingReader.add(Filter filter, Transformer ... transformer) functionality
  • setField now has type-specific constructors
  • added Eclipse project files
  • added Ant build project
  • bug-fix: whitespace (like tab) can now be used as the field separator in CSVReader
  • bug-fix: handle null variable names in expressions