Data Pipeline Builder – our new web GUI – is now available in early access. DPB generates Java code for Data Pipeline by letting you configure your inputs, outputs, and transformations.
Continue reading
6 Tips to Improve Your Exception Handling
Updated: July 2021
Proper exception handling can save you days in troubleshooting. Unexpected production issues can ruin anyone’s dinner and weekend plans, at any time. Furthermore, your reputation is on the line if you can’t resolve them quickly. A clear policy on exception management will save you diagnosis, reproduction, and correction time. What’s most important, it will give you peace of mind (and some hours back!).
Here are 6 tips on how you too can improve your exception handling.
Continue reading
Help us Design a Data Conversion and Transformation UI for Java Apps
We’re finally tackling a UI for Data Pipeline. Over the next few weeks, we’ll be drawing up plans and working on the first iteration/MVP (Minimum Viable Product) of Data Pipeline Builder, our online code generator. Continue reading
Export CSV and Excel from Java web apps With Data Pipeline
Updated: July 2021
This article will demonstrate how to download CSV (comma-separated values) and Excel data from Java web applications using Data Pipeline. It will also show one way to plug Data Pipeline into your JSPs.
Data Pipeline is often used in non-GUI applications to move data from one place/format to another. However, Data Pipeline can just as easily be plugged into your web, mobile, and desktop applications. Being as lightweight and Java-centric as it is, it is perfect for handling your data conversion and manipulation use-cases.
Data Pipeline 2.2.8 Now Available
- added TemplateWriter for writing text streams using FreeMarker templates
- added new examples for writing XML and HTML files using TemplateWriter
- BUGFIX: XmlWriter’s (XmlTemplate, File) constructor now calls setFieldNamesInFirstRow(false) by default
- BUGFIX: The JxlProvider now converts intervals and user-defined types to string when generating Excel files
- Intervals are no longer converted to strings when added to a field/record
- BasicFieldTransformer can now convert numbers to intervals (seconds, months, days, minutes, etc.)
- JdbcWriter now has public accessors for connection, tableName, batchMode, and jdbcTypes
- individual fields can now be removed from a FieldList
- FieldList can now accept collections of strings
- updated Apache POI to v3.8
Data Pipeline 2.2.7 Now Available
- added JdbcMultiWriter for multi-threaded writing to one or more database connections concurrently
- added multi-threaded AsyncWriter to compliment AsyncReader
- data writers now have an available() method to indicate the number of records that can probably be written without blocking
- MultiWriter now supports configurable write strategies (ReplicateWriteStrategy, RoundRobinWriteStrategy, AvailableCapacityWriteStrategy, and user defined)
- added support for CLOB fields (see JdbcValueReader.DEFAULT)
- Field and Record’s toString() methods now limit displayed strings to the first 128 characters
- RecordMeter is now public and returned by MeteredReader and MeteredWriter’s getMeter() method
- BUGFIX: record count is no longer off by 1 in some cases
Data Pipeline 2.2.6 Now Available
- performance improvements in CSV and fixed width handling
- untyped expression evaluation is now based on the value’s type, instead of the field’s declared type
- BUGFIX: now handles untyped expressions between primitive and object values
- float expressions are now upgraded to doubles during evaluation
- all non doubles and floats numbers are now upgraded to longs during evaluation
- expressions can now reference Java beans, not just primitive values
- method call expression now finds the most appropriate method based on the runtime argument types (http://en.wikipedia.org/wiki/Multiple_dispatch)
- improved handling for collections and arrays in DataException properties
- Apache PoiProvider can now distinguish between date, time, and datetimes fields in Excel
How to build a simple GWT event bus using Generators
In his Google I/O session Best Practices For Architecting Your GWT App, Ray Ryan discusses the benefits of using an event bus in GWT (Google Web Toolkit) applications. Inspired by this talk, I decided to try my hand at building a simple GWT event bus modeled after our pure java event bus.
Continue reading
Data Pipeline 2.2.5 Now Available
- Added JavaBeanReader whice uses XPath expressions to identify field values and break records (see the Read from Java beans example)
- AbstractReader’s setStartingRow and setLastRow no return this
- Filter rule IsInstanceOfJavaType now returns false for null values
- Added number-to-date methods to BasicFieldTransformer (numberToDate(), minutesToDate(), hoursToDate(), and daysToDate())
- BasicFieldTransformer.Operation and BasicFieldTransformer.StringOperation are now public classes
- BasicFieldTransformer.add(Operation … operation) is now public
- ConditionalTransformer is now private (use TransformingReader.filter instead)
- TransformingReader now contains an optional Filter, allowing any transformer to be conditionally applied
- Removed TransformingReader.add(Filter filter, Transformer … transformer) method
How To Manage Your Application Properties using WebDAV
Managing the configuration of an application is a consistent pain-point for developers, administrators, and business analysts.
Often in production environments, configuration is isolated as files on the local disk, limiting easy access by all but administrators. Another common approach is to store configuration in a database or LDAP. While this have benefits of a local disk file, it lacks the ability to manage properties as naturally as a file.
The solution proposed uses a database and WebDAV to help resolve many negatives to do with application configuration in both a simple and lightweight way.
Continue reading