Skip navigation links

Data Pipeline

Core 
Package Description
com.northconcepts.datapipeline.buffer
Organize incoming data into groups of sequential records without aggregation or transformation.
com.northconcepts.datapipeline.core
Core DataPipeline classes for reading, writing, and processing data.
com.northconcepts.datapipeline.csv
Read and write comma separated values (CSV) and other delimited data.
com.northconcepts.datapipeline.diagnostic
Troubleshoot DataPipeline's setup and environment.
com.northconcepts.datapipeline.eventbus
In-memory, publish-subscribe hub using simple Java interfaces or loosely-coupled readers and writers.
com.northconcepts.datapipeline.eventbus.filter  
com.northconcepts.datapipeline.excel
Read and write Microsoft Excel documents.
com.northconcepts.datapipeline.expression
com.northconcepts.datapipeline.file
Read and write records using DataPipeline's native binary format.
com.northconcepts.datapipeline.filter
Filter or exclude data by criteria.
com.northconcepts.datapipeline.filter.rule  
com.northconcepts.datapipeline.fixedwidth
Read and write fixed-width / fixed-length records.
com.northconcepts.datapipeline.group
Summarize batch and streaming data similar to "group by" in SQL.
com.northconcepts.datapipeline.group.operation  
com.northconcepts.datapipeline.internal.lang  
com.northconcepts.datapipeline.io  
com.northconcepts.datapipeline.javabean
Read Java Beans as a job's data source.
com.northconcepts.datapipeline.jdbc
Database reading and writing via JDBC.
com.northconcepts.datapipeline.jdbc.insert
Insert strategies for JdbcWriter and JdbcMultiWriter.
com.northconcepts.datapipeline.jdbc.sql
Common SQL generation classes.
com.northconcepts.datapipeline.jdbc.sql.conflict  
com.northconcepts.datapipeline.jdbc.sql.delete
SQL generation classes for DELETE.
com.northconcepts.datapipeline.jdbc.sql.insert
SQL generation classes for INSERT.
com.northconcepts.datapipeline.jdbc.sql.merge
SQL generation classes for MERGE.
com.northconcepts.datapipeline.jdbc.sql.select
SQL generation classes for SELECT.
com.northconcepts.datapipeline.jdbc.sql.update
SQL generation classes for UPDATE.
com.northconcepts.datapipeline.jdbc.upsert
Upsert strategies for JdbcUpsertWriter and JdbcMultiUpsertWriter.
com.northconcepts.datapipeline.job
Running, tracking, and managing data transfer pipelines.
com.northconcepts.datapipeline.json
JSON reading and writing.
com.northconcepts.datapipeline.json.builder
Programmatic JSON creation.
com.northconcepts.datapipeline.lineage
Extract data lineage from records and fields.
com.northconcepts.datapipeline.memory
Reading and writing data to and from in-memory buffers.
com.northconcepts.datapipeline.meter
Measures the rate (bytes/second or records/second) at which data is read or written.
com.northconcepts.datapipeline.multiplex
Strategies to convert a single source DataReader into many downstream readers.
com.northconcepts.datapipeline.retry
Classes to retry and continue reading or writing data on failure.
com.northconcepts.datapipeline.schedule
Common task scheduling.
com.northconcepts.datapipeline.sourcecode
Code generation classes.
com.northconcepts.datapipeline.throttle
Limits the rate (bytes/second or records/second) at which data is read or written.
com.northconcepts.datapipeline.transform
Restructure data using a variety of built-in and customizable operations.
com.northconcepts.datapipeline.transform.format  
com.northconcepts.datapipeline.transform.lookup
Enrich data by with data from another source.
com.northconcepts.datapipeline.transform.net  
com.northconcepts.datapipeline.validate
Validate data while reading.
com.northconcepts.datapipeline.weblog
Read web server logs.
com.northconcepts.datapipeline.xml
XML reading and writing.
com.northconcepts.datapipeline.xml.builder
Programmatic XML creation.
Foundations 
Package Description
com.northconcepts.datapipeline.foundations.core  
com.northconcepts.datapipeline.foundations.datamapping
Model and execute source-target data mapping using expressions and functions.
com.northconcepts.datapipeline.foundations.decisiontable
Decision table rule engine.
com.northconcepts.datapipeline.foundations.decisiontree
Decision tree rule engine.
com.northconcepts.datapipeline.foundations.difference
Common classes for comparing objects to detect elements that were added, removed, renamed, and changed.
com.northconcepts.datapipeline.foundations.expression  
com.northconcepts.datapipeline.foundations.file  
com.northconcepts.datapipeline.foundations.i18n  
com.northconcepts.datapipeline.foundations.internal  
com.northconcepts.datapipeline.foundations.jdbc
Database metadata model and code generation.
com.northconcepts.datapipeline.foundations.number
Detect number patterns in strings.
com.northconcepts.datapipeline.foundations.pipeline
Declarative pipeline modeling and execution.
com.northconcepts.datapipeline.foundations.pipeline.action  
com.northconcepts.datapipeline.foundations.pipeline.action.aggregate  
com.northconcepts.datapipeline.foundations.pipeline.action.convert  
com.northconcepts.datapipeline.foundations.pipeline.action.datamapping  
com.northconcepts.datapipeline.foundations.pipeline.action.decision  
com.northconcepts.datapipeline.foundations.pipeline.action.filter  
com.northconcepts.datapipeline.foundations.pipeline.action.transform  
com.northconcepts.datapipeline.foundations.pipeline.action.validate  
com.northconcepts.datapipeline.foundations.pipeline.dataset  
com.northconcepts.datapipeline.foundations.pipeline.input  
com.northconcepts.datapipeline.foundations.pipeline.output  
com.northconcepts.datapipeline.foundations.schema
Schema models for validation, mapping, and code generation.
com.northconcepts.datapipeline.foundations.schema.diff
Compare schema models to detect elements that were added, removed, renamed, and changed.
com.northconcepts.datapipeline.foundations.time
Detect date-time patterns in strings.
com.northconcepts.datapipeline.foundations.tools
DataPipeline Foundation tools for generating code and models.
Integrations 
Package Description
com.northconcepts.datapipeline.avro
Read and write Apache Avro files.
com.northconcepts.datapipeline.avro.pipeline.input  
com.northconcepts.datapipeline.avro.pipeline.output  
com.northconcepts.datapipeline.bloomberg
Read Bloomberg Message data.
com.northconcepts.datapipeline.email
Read emails.
com.northconcepts.datapipeline.google  
com.northconcepts.datapipeline.google.analytics
Read Google Analytics Report.
com.northconcepts.datapipeline.google.calendar
Read Google Calendar Events.
com.northconcepts.datapipeline.google.contacts
Read Google Contacts.
com.northconcepts.datapipeline.google.gmail
Read Gmail messages.
com.northconcepts.datapipeline.instagram
Search and read Instagram media and user data.
com.northconcepts.datapipeline.jira
Read Jira project, sprint and issue data.
com.northconcepts.datapipeline.jira.client
RESTful client and Java wrapper for Jira.
com.northconcepts.datapipeline.jms
Read and write Java Message Service data.
com.northconcepts.datapipeline.kafka
Read and write Apache Kafka distributed messaging system data.
com.northconcepts.datapipeline.mailchimp
Read MailChimp lists.
com.northconcepts.datapipeline.mailchimp.list  
com.northconcepts.datapipeline.mongodb
Read and write data in MongoDB.
com.northconcepts.datapipeline.orc
Read and write Apache ORC columnar files.
com.northconcepts.datapipeline.parquet
Read and Write Apache Parquet columnar files.
com.northconcepts.datapipeline.parquet.file  
com.northconcepts.datapipeline.parquet.input  
com.northconcepts.datapipeline.parquet.output  
com.northconcepts.datapipeline.parquet.tools  
com.northconcepts.datapipeline.pdf
Write data to a PDF file.
com.northconcepts.datapipeline.rtf
Writes records to an RTF document stream.
com.northconcepts.datapipeline.security
Encrypt and decrypt data.
com.northconcepts.datapipeline.shopify
Read data from Shopify.
com.northconcepts.datapipeline.shopify.client
REST client to read and write Shopify store data.
com.northconcepts.datapipeline.sql.mysql
Generate SQL code for the MySQL database.
com.northconcepts.datapipeline.sql.postgresql
Generate SQL code for the PostgreSQL database.
com.northconcepts.datapipeline.template
Write data to a text stream using FreeMarker template.
com.northconcepts.datapipeline.trello
Read and write data to a Trello board and cards.
com.northconcepts.datapipeline.twitter
Search and read Twitter users and tweets.
com.northconcepts.datapipeline.twitter2
Search and read Twitter users and tweets using Twitter v2 Api.
FileSystems 
Package Description
com.northconcepts.datapipeline.amazons3
Read and write to Amazon-S3.
com.northconcepts.datapipeline.filesystem.dropbox
Read and write to Dropbox.
com.northconcepts.datapipeline.filesystem.ftp
Read and write files to an FTP Server.
com.northconcepts.datapipeline.filesystem.hdfs
Read and write files to Hadoop Distributed File System.
com.northconcepts.datapipeline.filesystem.sftp
Read and write files to an SFTP Server.
com.northconcepts.datapipeline.filesystem.zip
Read and write data by treating Zips as a file system.
com.northconcepts.datapipeline.googledrive
Read and write files to Google Drive.
Skip navigation links

Copyright (c) 2006-2023 North Concepts Inc. All Rights Reserved.