This blog will show you how to pull selected columns from a CSV file containing IP geolocation data and save them into a second CSV file using our Data Pipeline Java library. As part of the transformation, you’ll also have the option to rearrange the order of the resulting columns.
Import CSV and Excel to Java Web Apps with Data Pipeline
This blog will demonstrate how to upload Excel and CSV files into a database while using Data Pipeline to handle the differences in format and structure of the individual files. Continue reading
Data Pipeline 2.3 Now Available
- added streaming JSON reading and writing (simple and template based)
- added SimpleXmlWriter
- improved handling of recursive XML-to-records
- added user-definable demux strategies
- DeMuxReader is no longer a public class since it should not be reference directly
- improved exception handling in JdbcReader
- BUGFIX: JavaBeanReader now handles xpath for recursive text children
- updated Apache POI to v3.9
- IncludeFields & ExcludeFields now accept a collection of field names in their constructor and add method
- added JdbcReader.useColumnLabel property to allow fields to be named using the column labels (or aliases) instead of the underlying, real column names
- added Excel 2007 provider (POI_XSSF)
Excel handling now defaults to the Apache POI_XSSF (Excel 2007) provider, instead of POI (Excel 2003) - added FixedWidthField.align to allow left-filled (right aligned) fields
- added FixedWidthField.fillChar to allow fields to specify a different filler from their reader/writer
- reduced memory overhead for fields and records
- CSV performance improvements
- exception property values now truncated to 256 chars
- using StringBuilder (instead of StringBuffer) internally to improve performance
Generate Data Migration Java Code with Data Pipeline Builder
Data Pipeline Builder – our new web GUI – is now available in early access. DPB generates Java code for Data Pipeline by letting you configure your inputs, outputs, and transformations.
Continue reading
6 Tips to Improve Your Exception Handling
Getting exception handling right can save you hours (or even days) of troubleshooting. Unexpected production issues can ruin your dinner and weekend plans. They can even affect your reputation if not resolved quickly. Having a clear policy on how to manage exceptions will save you time diagnosing, reproducing, and correcting issues. Here are 6 tips to improve your exception handling.
Continue reading
Help us Design a Data Conversion and Transformation UI for Java Apps
We’re finally tackling a UI for Data Pipeline. Over the next few weeks, we’ll be drawing up plans and working on the first iteration/MVP (Minimum Viable Product) of Data Pipeline Builder, our online code generator. Continue reading
Export CSV and Excel from Java web apps With Data Pipeline
Data Pipeline is typically used in non-GUI applications to move data from one place and/or format to another. However, Data Pipeline can just as easily be plugged into your web, mobile, and desktop applications. Its small size and Java-centric approach makes it perfect for handling your data conversion and data manipulation use-cases.
This article will demonstrate how to download CSV (comma-separated values) and Excel data from Java web applications using Data Pipeline. It will also show one way to plug Data Pipeline into your JSPs.
Continue reading
Data Pipeline 2.2.8 Now Available
- added TemplateWriter for writing text streams using FreeMarker templates
- added new examples for writing XML and HTML files using TemplateWriter
- BUGFIX: XmlWriter’s (XmlTemplate, File) constructor now calls setFieldNamesInFirstRow(false) by default
- BUGFIX: The JxlProvider now converts intervals and user-defined types to string when generating Excel files
- Intervals are no longer converted to strings when added to a field/record
- BasicFieldTransformer can now convert numbers to intervals (seconds, months, days, minutes, etc.)
- JdbcWriter now has public accessors for connection, tableName, batchMode, and jdbcTypes
- individual fields can now be removed from a FieldList
- FieldList can now accept collections of strings
- updated Apache POI to v3.8
Data Pipeline 2.2.7 Now Available
- added JdbcMultiWriter for multi-threaded writing to one or more database connections concurrently
- added multi-threaded AsyncWriter to compliment AsyncReader
- data writers now have an available() method to indicate the number of records that can probably be written without blocking
- MultiWriter now supports configurable write strategies (ReplicateWriteStrategy, RoundRobinWriteStrategy, AvailableCapacityWriteStrategy, and user defined)
- added support for CLOB fields (see JdbcValueReader.DEFAULT)
- Field and Record’s toString() methods now limit displayed strings to the first 128 characters
- RecordMeter is now public and returned by MeteredReader and MeteredWriter’s getMeter() method
- BUGFIX: record count is no longer off by 1 in some cases
Data Pipeline 2.2.6 Now Available
- performance improvements in CSV and fixed width handling
- untyped expression evaluation is now based on the value’s type, instead of the field’s declared type
- BUGFIX: now handles untyped expressions between primitive and object values
- float expressions are now upgraded to doubles during evaluation
- all non doubles and floats numbers are now upgraded to longs during evaluation
- expressions can now reference Java beans, not just primitive values
- method call expression now finds the most appropriate method based on the runtime argument types (http://en.wikipedia.org/wiki/Multiple_dispatch)
- improved handling for collections and arrays in DataException properties
- Apache PoiProvider can now distinguish between date, time, and datetimes fields in Excel