Blacklist and Whitelist Functions in DP Expression Language

This example enables you to manage and control the functions available within DataPipeline through blacklisting and whitelisting. By selectively permitting or restricting certain functions, you can fine-tune the behavior of the library to align with your data processing security and compliance needs. In practical applications, this capability ensures that only approved functions are used, enhancing data integrity and reducing the risk of unauthorized operations within your processing pipelines.

You can reference this documentation for the complete list of blacklisted methods in DataPipeline.

Input CSV File

Account,LastName,FirstName,Balance,CreditLimit,AccountCreated,Rating
101,Reeves,Keanu,9315.45,10000.00,1/17/1998,A
312,Butler,Gerard,90.00,1000.00,8/6/2003,B
868,Hewitt,Jennifer Love,0,17000.00,5/25/1985,B
761,Pinkett-Smith,Jada,49654.87,100000.00,12/5/2006,A
317,Murray,Bill,789.65,5000.00,2/5/2007,C

Java Code Listing

package com.northconcepts.datapipeline.examples.cookbook.customization;

import java.io.File;

import com.northconcepts.datapipeline.core.DataReader;
import com.northconcepts.datapipeline.core.Functions;
import com.northconcepts.datapipeline.core.StreamWriter;
import com.northconcepts.datapipeline.csv.CSVReader;
import com.northconcepts.datapipeline.job.Job;
import com.northconcepts.datapipeline.transform.SetCalculatedField;
import com.northconcepts.datapipeline.transform.TransformingReader;

public class BlacklistAndWhitelistFunctionsInExpressionLanguage {

    public static void main(String[] args) {
        callingBlacklistedFunctions();
        whitelistingBlacklistedFunctions();
        blacklistingCustomFunctions();
    }

    private static void callingBlacklistedFunctions() {
        System.out.println("================================Calling Blacklisted Functions================================--");
        try {
            DataReader reader = new CSVReader(new File("example/data/input/credit-balance-01.csv"))
                    .setFieldNamesInFirstRow(true);

            TransformingReader transformingReader = new TransformingReader(reader);

            // adding new field which should exit the program immediately..
            // but java.lang package is blacklisted by default, so this will throw an exception
            transformingReader.add(new SetCalculatedField("exitProgram", "java.lang.System.exit(0)"));

            Job.run(transformingReader, new StreamWriter(System.out));
        } catch (Throwable e) {
            e.printStackTrace();
        }
    }

    private static void whitelistingBlacklistedFunctions() {
        System.out.println("\n\n================================Whitelist Blacklisted Functions================================--");
        DataReader reader = new CSVReader(new File("example/data/input/credit-balance-01.csv"))
                .setFieldNamesInFirstRow(true);

        // Adding java.lang as whitelist.
        Functions.addWhitelistPrefix("java.lang.System.currentTimeMillis");

        TransformingReader transformingReader = new TransformingReader(reader);
        // adding new field which will display current time in milliseconds.
        transformingReader.add(new SetCalculatedField("currentTime", "java.lang.System.currentTimeMillis()"));

        Job.run(transformingReader, new StreamWriter(System.out));
    }

    private static void blacklistingCustomFunctions() {
        System.out.println("\n\n================================Blacklisting Custom Functions================================--");

        try {
            DataReader reader = new CSVReader(new File("example/data/input/credit-balance-01.csv"))
                    .setFieldNamesInFirstRow(true);

            // Adding custom blacklist function
            Functions.addBlacklistPrefix("com.northconcepts.datapipeline.examples.cookbook.BlacklistAndWhitelistFunctionsInExpressionLanguage");

            TransformingReader transformingReader = new TransformingReader(reader);
            // This will throw an exception as package is defined as blacklisted.
            transformingReader.add(new SetCalculatedField("currentTime", "com.northconcepts.datapipeline.examples.cookbook.BlacklistAndWhitelistFunctionsInExpressionLanguage.getCurrentTime()"));

            Job.run(transformingReader, new StreamWriter(System.out));
        } catch (Throwable e) {
            e.printStackTrace();
        }
    }

    public static long getCurrentTime() {
        return System.currentTimeMillis();
    }
}

Code Walkthrough

1. Inside the main method, three methods are called sequentially: callingBlacklistedFunctions(), whitelistingBlacklistedFunctions(), and blacklistingCustomFunctions().
2. In all three methods, CSVReader is created corresponding to the input file credit-balance-01.csv and TransformingReader is used to apply transformations to records from the reader.
3. Inside callingBlacklistedFunctions(), java.lang.System.exit(0) is called within SetCalculatedField. However, as java.lang package is blacklisted by default, it throws an exception.
4. In the next method whitelistingBlacklistedFunctions(),  java.lang.System.currentTimeMillis() is added to the white list using Functions.addWhitelistPrefix() and called successfully within SetCalculatedField.
5. In the final method, how to blacklist custom functions is shown. Functions.addBlacklistPrefix adds the current class to the blacklist and therefore, getCurrentTime() method throws an exception when invoked within SetCalculatedField.
6. In all methods, Job.run() is used to transfer the data from transformingReader to StreamWriter(System.out). See how to compile and run data pipeline jobs. 

Console Output

================================Calling Blacklisted Functions================================--
18:31:55,201 DEBUG [main] datapipeline:37 - DataPipeline v8.2.0 by North Concepts Inc.


================================Whitelist Blacklisted Functions================================--
-----------------------------------------------
0 - Record (MODIFIED) {
    0:[Account]:STRING=[101]:String
    1:[LastName]:STRING=[Reeves]:String
    2:[FirstName]:STRING=[Keanu]:String
    3:[Balance]:STRING=[9315.45]:String
    4:[CreditLimit]:STRING=[10000.00]:String
    5:[AccountCreated]:STRING=[1/17/1998]:String
    6:[Rating]:STRING=[A]:String
    7:[currentTime]:LONG=[1691933515420]:Long
}

-----------------------------------------------
1 - Record (MODIFIED) {
    0:[Account]:STRING=[312]:String
    1:[LastName]:STRING=[Butler]:String
    2:[FirstName]:STRING=[Gerard]:String
    3:[Balance]:STRING=[90.00]:String
    4:[CreditLimit]:STRING=[1000.00]:String
    5:[AccountCreated]:STRING=[8/6/2003]:String
    6:[Rating]:STRING=[B]:String
    7:[currentTime]:LONG=[1691933515420]:Long
}

-----------------------------------------------
2 - Record (MODIFIED) {
    0:[Account]:STRING=[868]:String
    1:[LastName]:STRING=[Hewitt]:String
    2:[FirstName]:STRING=[Jennifer Love]:String
    3:[Balance]:STRING=[0]:String
    4:[CreditLimit]:STRING=[17000.00]:String
    5:[AccountCreated]:STRING=[5/25/1985]:String
    6:[Rating]:STRING=[B]:String
    7:[currentTime]:LONG=[1691933515421]:Long
}

-----------------------------------------------
3 - Record (MODIFIED) {
    0:[Account]:STRING=[761]:String
    1:[LastName]:STRING=[Pinkett-Smith]:String
    2:[FirstName]:STRING=[Jada]:String
    3:[Balance]:STRING=[49654.87]:String
    4:[CreditLimit]:STRING=[100000.00]:String
    5:[AccountCreated]:STRING=[12/5/2006]:String
    6:[Rating]:STRING=[A]:String
    7:[currentTime]:LONG=[1691933515421]:Long
}

-----------------------------------------------
4 - Record (MODIFIED) {
    0:[Account]:STRING=[317]:String
    1:[LastName]:STRING=[Murray]:String
    2:[FirstName]:STRING=[Bill]:String
    3:[Balance]:STRING=[789.65]:String
    4:[CreditLimit]:STRING=[5000.00]:String
    5:[AccountCreated]:STRING=[2/5/2007]:String
    6:[Rating]:STRING=[C]:String
    7:[currentTime]:LONG=[1691933515421]:Long
}

-----------------------------------------------
5 records


================================Blacklisting Custom Functions================================--

BUILD SUCCESSFUL in 4s
3 actionable tasks: 1 executed, 2 up-to-date
com.northconcepts.datapipeline.core.DataException:  method java.lang.System.exit() is blacklisted
-------------------------------
expression=[java.lang.System.exit(0)]
expression.MethodCallExpression.alias=[java.lang.System.exit]
expression.MethodCallExpression.methodSignature=[java.lang.System.exit]

com.northconcepts.datapipeline.core.DataException:  method com.northconcepts.datapipeline.examples.cookbook.BlacklistAndWhitelistFunctionsInExpressionLanguage.getCurrentTime() is blacklisted
-------------------------------
expression=[com.northconcepts.datapipeline.examples.cookbook.BlacklistAndWhitelistFunctionsInExpressionLanguage.getCurrentTime()]
expression.MethodCallExpression.alias=[com.northconcepts.datapipeline.examples.cookbook.BlacklistAndWhitelistFunctionsInExpressionLanguage.getCurrentTime]
expression.MethodCallExpression.methodSignature=[com.northconcepts.datapipeline.examples.cookbook.BlacklistAndWhitelistFunctionsInExpressionLanguage.getCurrentTime]