We’re pleased to announce the release of version 3.0 of our Data Pipeline engine.
This release includes the new Sliding Window Aggregations feature to perform continuous SQL group-by operations on streaming data.
We’ve improved the performance of the XPath based readers (JsonReader, XmlReader, and JavaBeanReader), included new conveniences to reduce your code size, and added several new transformers and filters.
We’re also now offering a free 30-day trial for you to take the premium and enterprise features out for a test drive.
We’re hiring a new Java developer and decided to start by asking them to write code instead of the usual Q&A.
Recently we needed to add an hourly scheduler to our sliding window data aggregator and decided this would be a good test to see how people think and code.
A new release of Data Pipeline is now available for download: https://northconcepts.com/downloads/. This release includes a new Twitter search reader, custom aggregate operations, and much more.
Data Pipeline’s query engine allows you to use XPath to query XML, JSON, and Java objects. This walkthrough will show you how to query Java objects using XPath and save the results to a CSV file. While the reading and writing will be done with the JavaBeanReader and CSVWriter classes, you can swap out the CSVWriter for any other endpoint or transformation that Data Pipeline supports. Continue reading
This blog will show you how to pull selected columns from a CSV file containing IP geolocation data and save them into a second CSV file using our Data Pipeline Java library. As part of the transformation, you’ll also have the option to rearrange the order of the resulting columns.
This blog will demonstrate how to upload Excel and CSV files into a database while using Data Pipeline to handle the differences in format and structure of the individual files. Continue reading
Data Pipeline Builder – our new web GUI – is now available in early access. DPB generates Java code for Data Pipeline by letting you configure your inputs, outputs, and transformations.
Updated: July 2021
Proper exception handling can save you days in troubleshooting. Unexpected production issues can ruin anyone’s dinner and weekend plans, at any time. Furthermore, your reputation is on the line if you can’t resolve them quickly. A clear policy on exception management will save you diagnosis, reproduction, and correction time. What’s most important, it will give you peace of mind (and some hours back!).
Here are 6 tips on how you too can improve your exception handling.
We’re finally tackling a UI for Data Pipeline. Over the next few weeks, we’ll be drawing up plans and working on the first iteration/MVP (Minimum Viable Product) of Data Pipeline Builder, our online code generator. Continue reading