11 Java Data Integration Libraries (2023)

11 Java Data Integration Libraries for 2022

Updated: May 2023

With data being produced from many sources in a variety of formats businesses must have a sane way to gain useful insight. Data integration is the process of transforming data from one or more sources into a form that can be loaded into a target system or used for analysis and business intelligence.

Data integration libraries take some programming burden from the shoulders of developers by abstracting data processing and transformation tasks and allowing the developer to focus on tasks that are directly related to the application logic.

Continue reading →

25 Machine Learning and Artificial Intelligence Conferences

52 Machine Learning and Artificial Intelligence Conferences in 2017 and 2018

Machine learning and artificial intelligence in general are two of today’s hottest skills.  AI and ML conferences provide a place for you to improve your skills, discuss trends, and exchange ideas with other data scientists, developers, and entrepreneurs.  Whether you’re new to the world of machine learning, trying to stay up-to-date, or just looking to network, there’s a conference happening for you.  This article lists over 50 conferences taking place around the world for you to consider attending.

Continue reading →

How to Convert Tabular Data to Trees Using Aggregation

How to Convert Tabular Data to Trees Using Aggregation

We recently received an email from a Java developer asking how to convert records in a table (like you get in a relational database, CSV, or Excel file) to a composite tree structure.  Normally, we’d point to one of Data Pipeline’s XML or JSON data writers, but for good reasons those options didn’t apply here.  The developer emailing us needed the hierarchical structures in object form for use in his API calls.

Since we didn’t have a general purpose, table-tree mapper, we built one.  We looked at several options, but ultimately decided to add a new operator to the GroupByReader.  This not only answered the immediate mapping question, but also allowed him to use the new operator with sliding window aggregation if the need ever arose.

The rest of this blog will walk you through the implementation in case you ever need to add your own custom aggregate operator to Data Pipeline.

Continue reading →

18 ETL Tools for Java Developers (Updated 2023)

ETL Tools for Java Developers

Updated: May 2023

ETL is a process for performing data extraction, transformation and loading. The process extracts data from a variety of sources and formats, transforms it into a standard structure, and loads it into a database, file, web service, or other system for analysis, visualization, machine learning, etc.

ETL tools come in a wide variety of shapes.  Some run on your desktop or on-premises servers, while others run as SaaS in the cloud.  Some are code-based, built on standard programming languages that many developers already know.  Others are built on a custom DSL (domain specific language) in an attempt to be more intentional and require less code.  Others still are completely graphical, only offering programming interfaces for complex transformations.

What follows is a list of ETL tools for developers already familiar with Java and the JVM (Java Virtual Machine) to clean, validate, filter, and prepare your data for use.

Continue reading →

How to Export Emails from Gmail to Excel with Data Pipeline

Export emails from Gmail and G Suite to Excel

Updated: July 2021

If you have ever tried to export emails to Excel for analysis, you know it is not exactly straightforward.  Maybe you need to find the top companies contacting you and your sales team.  Maybe you need to perform text or sentiment analysis on the contents of your messages.  Or maybe you’re creating visualizations to better understand who’s emailing you.  This east guide will show you how you can use Data Pipeline to search and read emails from Gmail or G Suite, process them any way you like, and store them in Excel.

Continue reading →

25 Conferences Data Scientists Should Attend in 2022 and 2023

 

20 Conferences Data Scientists Should Attend

Updated: June 2022

Being a data scientist means dedication to continuous learning.  One great way to keep learning, improve your network, and get exposed to different views is to attend conferences.

Since 2020 organizers have been opting for online virtual conferences instead of in-person conferences. In 2021 and 2022 the same trend continues although some conferences are also being scheduled to be attended in person since the second half of the year 2021.

Data science conferences are one of the best ways to learn, develop new skills, meet and discuss ideas and discover how others are applying AI, analytics and machine learning in their work.

Here are several conferences for data scientists you should consider attending.

Continue reading →