Write to Google Drive

This example shows how to write a file to Google Drive with DataPipeline. It reads a local CSV file and writes it back to the root of your Google Drive as a pretty-printed JSON file named credit-balance-01.json. Any DataPipeline writer can target Drive by writing to the OutputStream returned by the GoogleDriveFileSystem.

Authentication & Setup

GoogleDriveFileSystem is constructed from four values: an APPLICATION_NAME, a USER_NAME, an InputStream of the OAuth 2.0 client-secrets file (example/data/input/client_id.json, downloaded from the Google Cloud console), and a CREDENTIALS_DIR (example/credentials) where the authorized credential is stored. On the first run Google prompts you to authorize access in a browser; once a credential has been stored, Google will not prompt again unless it is deleted.

Java Code Listing

package com.northconcepts.datapipeline.examples.google.drive;

import java.io.File;
import java.io.FileInputStream;
import java.io.OutputStreamWriter;

import com.northconcepts.datapipeline.core.DataReader;
import com.northconcepts.datapipeline.core.DataWriter;
import com.northconcepts.datapipeline.csv.CSVReader;
import com.northconcepts.datapipeline.googledrive.GoogleDriveFileSystem;
import com.northconcepts.datapipeline.job.Job;
import com.northconcepts.datapipeline.json.SimpleJsonWriter;

public class WriteToGoogleDrive {

    private static final String APPLICATION_NAME = "datapipeline";
    private static final String USER_NAME = "YOUR GOOGLE DRIVE USERNAME";
    private static final File CLIENT_SECRETS = new File("example/data/input/client_id.json");
    private static final File CREDENTIALS_DIR = new File("example", "credentials");

    // On first run you will be prompted by Google to authorize access in a browser.
    // Once a credential has been stored in CREDENTIALS_DIR, Google will not prompt again
    // unless the stored credential is deleted.
    public static void main(String[] args) throws Throwable {
        GoogleDriveFileSystem fileSystem = new GoogleDriveFileSystem(APPLICATION_NAME, USER_NAME,
                new FileInputStream(CLIENT_SECRETS), CREDENTIALS_DIR);
        fileSystem.open();

        try {
            DataReader reader = new CSVReader(new File("example/data/input/credit-balance-01.csv"))
                    .setFieldNamesInFirstRow(true);

            // Writes "credit-balance-01.json" to the root of your Google Drive.
            OutputStreamWriter osw = new OutputStreamWriter(fileSystem.writeFile("credit-balance-01.json"));
            DataWriter writer = new SimpleJsonWriter(osw)
                    .setPretty(true);

            Job.run(reader, writer);

            System.out.println("Records written: " + writer.getRecordCount());
        } finally {
            fileSystem.close();
        }
    }

}

Code Walkthrough

• A CSVReader reads the local file example/data/input/credit-balance-01.csv, using its first row as field names.
• writeFile("credit-balance-01.json") returns an OutputStream for a new file in the Drive root; it is wrapped in an OutputStreamWriter and handed to a SimpleJsonWriter. setPretty(true) produces indented, human-readable JSON.
• Job.run() streams the records from the CSV file into the JSON file on Drive, and the number of records written is printed.
• fileSystem.close() runs in the finally block.