1. Home
  2. Documentation
  3. Examples

Examples

All examples can be found on GitHub (https://github.com/NorthConcepts/DataPipeline-Examples).

Get DataPipeline 6.0.0
  1. Buffer Records by Time Period or Count
  2. Capture Data Not Joined in a Lookup
  3. Compile and Run a Job
  4. Continue After an Error
  5. Convert a Single Source DataReader into Many
  6. Convert JSON to CSV
  7. Debug my Code
  8. Extract Bigrams, Trigrams, and Ngrams
  9. Generate a PDF
  10. Generate a Word Document
  11. Group Records by TimePeriod or Count
  12. Handle Exceptions
  13. Log Diagnostic Information
  14. Measure Data being Read and Written
  15. Measure Data Lineage Performance
  16. Measure Performance of Reader and Writer
  17. Obtain Statistics
  18. Open and Close Several Data Readers and Data Writers at Once
  19. Pipe a Writer to a Reader
  20. Profile Performance
  21. Read BigDecimal and BigInteger from an Excel file
  22. Read a Bloomberg Message File
  23. Read a CSV File
  24. Read a Fixed-width File / Fixed-length Record File
  25. Read a JSON Stream
  26. Read a Parquet File
  27. Read a Patient File
  28. Read a Simple JSON File
  29. Read a Simple XML File
  30. Read a Web Server Log
  31. Read an Avro File
  32. Read an XML File
  33. Read an XML File (2)
  34. Read and Write to an EventBus
  35. Read Big Decimal In JSON
  36. Read Emails
  37. Read From Amazon S3
  38. Read from a Database
  39. Read from an Excel File
  40. Read from Gmail
  41. Read from Java beans
  42. Read from JMS Queue
  43. Read from JMS Topic
  44. Read from Memory
  45. Read from MongoDB
  46. Read Google Analytics goal conversions
  47. Read Google Analytics Social Interactions
  48. Read Google Analytics views
  49. Read Google Calendar
  50. Read Google Contacts
  51. Read Google Gmail Messages
  52. Read JSON Records From File
  53. Read Selected Fields from a Parquet File
  54. Read using TimedReader
  55. Read XML Records From File
  56. Search for a Record
  57. Search Twitter for Tweets
  58. Serialize and Deserialize Data
  59. Serialize and Deserialize Records
  60. Throttle Data being Read
  61. Throttle Data being Written
  62. Use Multi Threading in a Single Job
  63. Use a Retrying Reader
  64. Use a Retrying Writer
  65. Use Data Lineage with CsvReader
  66. Use Data Lineage with ExcelReader
  67. Use Data Lineage with FixedWidthReader
  68. Use Data Lineage with JdbcReader
  69. Use Data Lineage with Lookup
  70. Use Data Lineage with ParquetReader
  71. Use Streaming Excel for Reading
  72. Use Streaming Excel for Writing
  73. Upsert Records to a Database Using Insert and Update
  74. Upsert Records to a Database Using Merge
  75. Upsert Records to MySql or MariaDB
  76. Upsert Records to Oracle
  77. Upsert Records to PostgreSql
  78. Upsert Records to Sybase
  79. Upsert Variable Field Records
  80. Write a CSV File to Database (1)
  81. Write a CSV File to Database (2)
  82. Write a CSV File to Fixed Width
  83. Write a Parquet File
  84. Write a Simple JSON File
  85. Write a Simple XML File
  86. Write a Sequence of Files by Record Count
  87. Write a Sequence of Files by Elapsed Time
  88. Write an Avro File
  89. Write an XML File Programmatically
  90. Write an XML File using FreeMarker Templates
  91. Write CSV To XML Using FreeMarker Templates
  92. Write HTML using FreeMarker Templates
  93. Write Key-Value Fields to MapWriter
  94. Write to Amazon S3 Using Multipart Streaming
  95. Write to a Database Using Custom Jdbc Insert Strategy
  96. Write to a Database Using Generic Upsert Strategy
  97. Write to a Database Using Merge Upsert Strategy
  98. Write to a Database Using Merge Upsert Strategy with Batch
  99. Write to a Database Using Multiple Connections
  100. Write to a Database Using Multi Row Prepared Statement Insert Strategy
  101. Write to a Database Using Multi Row Statement Insert Strategy
  102. Write to Excel
  103. Write to JMS Queue
  104. Write to JMS Topic
  105. Write to JSON Stream (simple)
  106. Write to JSON Stream Programmatically
  107. Write to Memory
  108. Write to MongoDB
  109. Write to Several Data Writers at Once
  110. Write to the Console
  111. Write to XML Stream (Simple)
  1. Add a Sequence Number Column
  2. Add a Sequence Number Column when Values Change
  3. Add a Sequence Number Column for Repeat Values
  4. Add Nonpersistent Data to Records and Fields
  5. Cache Lookup Values
  6. Conditionally Transform Data
  7. Conditionally Write to Multiple Files
  8. Copy a Field
  9. Filter Records
  10. Filter Twitter in Real Time
  11. Find The Minimum Maximum Average Sum Count
  12. Get a Query Param from a Url
  13. Handle Filtered Records
  14. Handle Transforming Reader Failures
  15. Handle Transforming Writer Failures
  16. Handle Validation Failures
  17. Log Job Progress
  18. Lookup Data in a Database
  19. Lookup Data in any Data Reader
  20. Lookup in a Flat File
  21. Manipulate Fields
  22. Read a Set of Files
  23. Remove Duplicates
  24. Remove/Retain Duplicate Fields
  25. Remove Fields
  26. Rename a Field
  27. Select and Arrange Fields
  28. Set a Calculated Field at Runtime
  29. Set a Calculated Field Programmatically
  30. Set a Field
  31. Sort Records
  32. Specify Lookup Values Programmatically
  33. Validate Incoming Data
  34. Write to A Data Writer using Tee Reader
  1. Add my own Functions
  2. Blacklist and Whitelist Functions in DP Expression Language
  3. My Data Reader
  4. My Data Writer
  5. My Filter
  6. My Lookup
  7. My Proxy Reader
  8. My Proxy Writer
  9. My Transformer
  10. Write my own Data Reader
  11. Write my own Data Writer
  12. Write my own Filter or Validator
  13. Write my own Lookup
  14. Write my own Transformer
  1. Add a Decision Table to a Pipeline
  2. Add a Decision Tree to a Pipeline
  3. Add Calculated Fields to a Decision Table
  4. Add Calculated Fields to a Decision Tree
  5. Conditionally map Data from Source to Target
  6. Conditionally map DataField from Source to Target
  7. Create Custom Pipeline Action
  8. Create Custom Pipeline Input
  9. Create Custom Pipeline Output
  10. Evaluate a Decision Table
  11. Evaluate a Decision Table with Lookup
  12. Evaluate a Decision Tree
  13. Evaluate a Decision Tree with Lookup
  14. Execute an Action in a Decision Table
  15. Execute an Action in a Decision Tree
  16. Filter Columns with All Null Values
  17. Generate Java Beans from a Database
  18. Map Data with Rule Based Validation
  19. Map Data with Schema Based Validation
  20. Map Data from Source to Target
  21. Map Data from Source to Target in a Pipeline
  22. Map Data from Source to Target in a Pipeline with Validation
  23. Map Data from Source to Target with Lookup
  24. Read from CSV And Writer to Excel
  25. Save and Load DataMapping to XML
  26. Save and Load DecisionTable to XML
  27. Save and Load DecisionTree to XML
  28. Load Snapshot of Dataset
  29. Save and Restore Pipeline from JSON
  30. Show Column Statistics
  31. Show the Columns and Tables of a Schema
  32. Show Unique Values in Column
  33. Use an EventBus
  34. Use an EventBus in a Pipeline
  35. Use SchemaFilter to Validate Records in a Pipeline
  36. Validate a Field
  37. Validate a Value
  38. Validate Record Fields
  39. Validate Records using Fields and Rules
  40. Validate Records using Rules

Tags

Avro (5)
Cache (1)
CSV (20)
Debug (9)
Email (3)
Filter (10)
Google (6)
HTML (1)
JDBC (24)
JMS (4)
JSON (11)
Lookup (10)
Meter (2)
Native (1)
PDF (1)
Pipe (1)
Push (1)
RTF (1)
Rules (2)
Schema (6)
Search (1)
Simple (14)
Sort (1)
Stats (2)
STDOUT (12)
XML (16)
XPath (5)
Get DataPipeline 6.0.0
Mobile Analytics