Read from Google Drive

This example shows how to read a file from Google Drive with DataPipeline. DataPipeline exposes Google Drive as a virtual file system through the GoogleDriveFileSystem class, so any DataPipeline reader can consume a file stored in Drive by reading from the stream it returns. Here it lists the files in the root of your Google Drive, then reads a CSV file named cached-lookup.csv from the root and prints its records to the console.

Authentication & Setup

GoogleDriveFileSystem is constructed from four values: an APPLICATION_NAME, a USER_NAME, an InputStream of the OAuth 2.0 client-secrets file (example/data/input/client_id.json, downloaded from the Google Cloud console), and a CREDENTIALS_DIR (example/credentials) where the authorized credential is stored. On the first run Google prompts you to authorize access in a browser; once a credential has been stored, Google will not prompt again unless it is deleted.

Java Code Listing

package com.northconcepts.datapipeline.examples.google.drive;

import java.io.File;
import java.io.FileInputStream;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.List;

import com.northconcepts.datapipeline.core.DataReader;
import com.northconcepts.datapipeline.core.DataWriter;
import com.northconcepts.datapipeline.core.StreamWriter;
import com.northconcepts.datapipeline.csv.CSVReader;
import com.northconcepts.datapipeline.googledrive.GoogleDriveFileSystem;
import com.northconcepts.datapipeline.job.Job;

public class ReadFromGoogleDrive {

    private static final String APPLICATION_NAME = "datapipeline";
    private static final String USER_NAME = "YOUR GOOGLE DRIVE USERNAME";
    private static final File CLIENT_SECRETS = new File("example/data/input/client_id.json");
    private static final File CREDENTIALS_DIR = new File("example", "credentials");

    // On first run you will be prompted by Google to authorize access in a browser.
    // Once a credential has been stored in CREDENTIALS_DIR, Google will not prompt again
    // unless the stored credential is deleted.
    public static void main(String[] args) throws Throwable {
        GoogleDriveFileSystem fileSystem = new GoogleDriveFileSystem(APPLICATION_NAME, USER_NAME,
                new FileInputStream(CLIENT_SECRETS), CREDENTIALS_DIR);
        fileSystem.open();

        try {
            List files = fileSystem.listRootFolder();
            if (files == null || files.size() == 0) {
                System.out.println("No files found.");
            } else {
                System.out.println("Files:");
                for (com.google.api.services.drive.model.File file : files) {
                    System.out.printf("%s (%s)%n", file.getName(), file.getId());
                }
            }

            // "cached-lookup.csv" must already exist in the root of your Google Drive.
            InputStream inputStream = fileSystem.readFile("cached-lookup.csv");

            DataReader reader = new CSVReader(new InputStreamReader(inputStream))
                    .setFieldNamesInFirstRow(true);
            DataWriter writer = StreamWriter.newSystemOutWriter();
            Job.run(reader, writer);

            System.out.println("Records read: " + writer.getRecordCount());
        } finally {
            fileSystem.close();
        }
    }

}

Code Walkthrough

• A GoogleDriveFileSystem is created from the application name, user name, the client-secrets stream, and the credentials directory, then connected with open().
• listRootFolder() returns the files in the Drive root as a list of Google Drive File objects; each file's name and id are printed.
• readFile("cached-lookup.csv") returns an InputStream for the file in the Drive root, which is wrapped in an InputStreamReader and read by a CSVReader. setFieldNamesInFirstRow(true) treats the first CSV row as field names.
• Records are transferred to a StreamWriter bound to System.out via Job.run(), and the record count is printed.
• fileSystem.close() runs in the finally block to release the connection regardless of success or failure.