Category Archives: Data Pipeline

How to Convert Tabular Data to Trees Using Aggregation

How to Convert Tabular Data to Trees Using Aggregation

We recently received an email from a Java developer asking how to convert records in a table (like you get in a relational database, CSV, or Excel file) to a composite tree structure.  Normally, we’d point to one of Data Pipeline’s XML or JSON data writers, but for good reasons those options didn’t apply here.  The developer emailing us needed the hierarchical structures in object form for use in his API calls.

Since we didn’t have a general purpose, table-tree mapper, we built one.  We looked at several options, but ultimately decided to add a new operator to the GroupByReader.  This not only answered the immediate mapping question, but also allowed him to use the new operator with sliding window aggregation if the need ever arose.

The rest of this blog will walk you through the implementation in case you ever need to add your own custom aggregate operator to Data Pipeline.

Continue reading

24 ETL Tools for Java Developers

ETL Tools for Java Developers

ETL is a process for performing data extraction, transformation and loading. The process extracts data from a variety of sources and formats, transforms it into a standard structure, and loads it into a database, file, web service, or other system for analysis, visualization, machine learning, etc.

ETL tools come in a wide variety of shapes.  Some run on your desktop or on-premise servers, while others run as SaaS in the cloud.  Some are code-based, built on standard programming languages that many developers already know.  Others are built on a custom DSL (domain specific language) in an attempt to be more intentional and require less code.  Others still are completely graphical, only offering programming interfaces for complex transformations.

What follows is a list of ETL tools for developers already familiar with Java and the JVM (Java Virtual Machine) to clean, validate, filter, and prepare your data for use.

Continue reading

How to Export Emails from Gmail to Excel with Data Pipeline

Export emails from Gmail and G Suite to Excel

Have you ever wanted to pull emails into Excel for analysis?  Maybe you need to find the top companies contacting you for your sales team.  Maybe you need to perform text or sentiment analysis on the contents of your messages.  Or maybe you’re creating visualizations to better understand who’s emailing you.  This quick guide will show you how to use Data Pipeline to search and read emails from Gmail or G Suite (formerly Google Apps), process them any way you like, and store them in Excel.

Continue reading

Spring Batch vs Data Pipeline – ETL Job Example

Data Pipeline vs Spring Batch

I was reading a blog at Java Code Geeks on how to create a Spring Batch ETL Job.  What struck me about the example was the amount of code required by the framework for such a routine task.  In this blog, you’ll see how to accomplish the same task of summarize a million stock trades to find the open, close, high, and low prices for each symbol using our Data Pipeline framework.

Continue reading