Data Pipeline v3.1.4 is now available for download. This release includes support for MySQL upserts, lower JSON and XML memory usage, bug fixes, and more.
Continue reading
How To Aggregate Twitter Searches Without A Database
One feature of Data Pipeline is its ability to aggregate data without a database. This feature allows you to apply SQL “group by” operations to JSON, CSV, XML, Java beans, and other formats on-the-fly — in real-time. This quick tutorial will show you how to use the GroupByReader class to aggregate Twitter search results.
Data Pipeline 3.1 Now Available
Data Pipeline 3.1 is now available for download. This is a milestone release that adds native support for hierarchical data (nested records and multidimensional arrays).
How to read data in parallel using AsyncMultiReader
Data Pipeline now includes a new AsyncMultiReader endpoint that lets you read from multiple DataReaders in parallel. Here’s how it works.
How to convert XML to Excel (2023)
Data Pipeline makes it easy to read, transform, and write XML and Excel files. This post shows you how you too can load data from an on-disk XML file, apply transformations on the fly, and save the result to an Excel file.
How to create multiple sheets in a single Excel file
Data Pipeline lets you read, write, and convert Excel files using a very simple API. This post will show you how to create Excel files containing more than one work sheet or tab.
Data Pipeline 3.0 Now Available
We’re pleased to announce the release of version 3.0 of our Data Pipeline engine.
This release includes the new Sliding Window Aggregations feature to perform continuous SQL group-by operations on streaming data.
We’ve improved the performance of the XPath based readers (JsonReader, XmlReader, and JavaBeanReader), included new conveniences to reduce your code size, and added several new transformers and filters.
We’re also now offering a free 30-day trial for you to take the premium and enterprise features out for a test drive.
How would you implement this Java class?
We’re hiring a new Java developer and decided to start by asking them to write code instead of the usual Q&A.
Recently we needed to add an hourly scheduler to our sliding window data aggregator and decided this would be a good test to see how people think and code.
Data Pipeline 2.3.4 Now Available
A new release of Data Pipeline is now available for download: https://northconcepts.com/downloads/. This release includes a new Twitter search reader, custom aggregate operations, and much more.
How to Query Java Objects with XPath
Data Pipeline’s query engine allows you to use XPath to query XML, JSON, and Java objects. This walkthrough will show you how to query Java objects using XPath and save the results to a CSV file. While the reading and writing will be done with the JavaBeanReader and CSVWriter classes, you can swap out the CSVWriter for any other endpoint or transformation that Data Pipeline supports. Continue reading