Last month we released version 8.2.0 of DataPipline. Here’s what you can expect.
- SQL builder classes now support unions and sub-queries.
- Added FieldList.contains(List<String> fieldNames), containedWithin(String … fieldNames), and containedWithin(List<String> fieldNames).
- The expression languages can now produce expressions even when they contain syntax errors. This is used in the Foundations library in DataMapping, DecisionTable, DecisionTree, and others to allow work-in-progress to be saved and restored from XML and JSON.
- The expression languages improves ClassCastException messages by adding the candidate’s value and type.
- MergeUpsert now automatically terminates with a semicolon for Microsoft SQL Server and SAP ASE (Sybase) databases.
- SybaseUpsert now terminates with a semicolon by default.
- Added JdbcConnectionFactory.close() for scenarios where the factory needs to handle lifecycle termination. For example, when the factory holds a single, pre-connected database connection.
- Job now emits less debug logs
- DataMapping and FieldMapping now return any invalid expressions with syntax errors as problems. See how to retrieve DataMapping problems.
- Added Dataset.isDataLoading(), getDataLoadException(), getMaxRecordsToLoad(), and afterLoad() to better support real-time, interactive, and UI-based use-cases.
- Added DataMappingEditor for interactive, UI-based use-cases (in SimpleDataHub) that involve data mapping, sorting, and pagination of a source dataset.
- Added DataMappingReader/DataMappingWriter.onFailure(Record record, DataException exception, DataMappingResult result) to allow error handling to be intercepted/overridden.
- Added DataMappingResult.getDataMappingResult(Record record) and setDataMappingResult(Record record, DataMappingResult result) to attach DataMappingResult to Record as session properties.
- Added DataMappingValidator.checkValidExpression() to look for expressions with syntax errors.
- Added Column.getInferredTextualValueCount(), getTextual() to count text values in columns.
- Improved the type detection algorithm used in Column.getInferredFieldType().
- BUGFIX: DatasetReader.readImpl() will now wait until the dataset produces records or closes instead of returning null when no records are immediately available.
- BUGFIX: SchemaValidator.checkEntityRelationshipCardinality now correctly looks for invalid one-to-many relationships to return in the problems list.
- Added initial release of DetectPrimaryKeysInDataset tool.
- Added JiraService.close() to explicitly and eagerly close the REST endpoint.
- Added AmazonS3FileSystem.getEndpointConfiguration(), setEndpointConfiguration(EndpointConfiguration endpointConfiguration), and setClient(AmazonS3 client).
- getClient() now returns AmazonS3 instead of AmazonS3Client.
- BUGFIX: AmazonS3FileSystem can now write empty files to S3.
See the CHANGELOG for the full set of updates in DP 8.2.0.