Read from a Google Drive Folder

This example shows how to read a file from a folder in Google Drive with DataPipeline. It lists the contents of a subfolder named First SubFolder, then reads an Excel workbook stored inside that folder and prints the balance sheet to the console. Files are located by folder and name using the two-argument form of readFile() on the GoogleDriveFileSystem.

Authentication & Setup

GoogleDriveFileSystem is constructed from four values: an APPLICATION_NAME, a USER_NAME, an InputStream of the OAuth 2.0 client-secrets file (example/data/input/client_id.json, downloaded from the Google Cloud console), and a CREDENTIALS_DIR (example/credentials) where the authorized credential is stored. On the first run Google prompts you to authorize access in a browser; once a credential has been stored, Google will not prompt again unless it is deleted.

Java Code Listing

package com.northconcepts.datapipeline.examples.google.drive;

import java.io.File;
import java.io.FileInputStream;
import java.util.List;

import com.northconcepts.datapipeline.core.DataReader;
import com.northconcepts.datapipeline.core.DataWriter;
import com.northconcepts.datapipeline.core.StreamWriter;
import com.northconcepts.datapipeline.excel.ExcelDocument;
import com.northconcepts.datapipeline.excel.ExcelReader;
import com.northconcepts.datapipeline.googledrive.GoogleDriveFileSystem;
import com.northconcepts.datapipeline.job.Job;

public class ReadFromAGoogleDriveFolder {

    private static final String APPLICATION_NAME = "datapipeline";
    private static final String USER_NAME = "YOUR GOOGLE DRIVE USERNAME";
    private static final File CLIENT_SECRETS = new File("example/data/input/client_id.json");
    private static final File CREDENTIALS_DIR = new File("example", "credentials");

    // On first run you will be prompted by Google to authorize access in a browser.
    // Once a credential has been stored in CREDENTIALS_DIR, Google will not prompt again
    // unless the stored credential is deleted.
    public static void main(String[] args) throws Throwable {
        GoogleDriveFileSystem fileSystem = new GoogleDriveFileSystem(APPLICATION_NAME, USER_NAME,
                new FileInputStream(CLIENT_SECRETS), CREDENTIALS_DIR);
        fileSystem.open();

        try {
            // "First SubFolder" must already exist in your Google Drive.
            List files = fileSystem.listFolder("First SubFolder");
            if (files == null || files.size() == 0) {
                System.out.println("No files found.");
            } else {
                System.out.println("Files in First SubFolder:");
                for (com.google.api.services.drive.model.File file : files) {
                    System.out.printf("%s (%s)%n", file.getName(), file.getId());
                }
            }

            // "call-center-inbound-call-2.xls" must already exist inside "First SubFolder".
            ExcelDocument document = new ExcelDocument()
                    .open(fileSystem.readFile("First SubFolder", "call-center-inbound-call-2.xls"));
            DataReader reader = new ExcelReader(document)
                    .setSheetName("balance")
                    .setFieldNamesInFirstRow(true);
            DataWriter writer = StreamWriter.newSystemOutWriter();
            Job.run(reader, writer);

            System.out.println("Records read: " + writer.getRecordCount());
        } finally {
            fileSystem.close();
        }
    }

}

Code Walkthrough

• listFolder("First SubFolder") returns the files contained in the named subfolder rather than the Drive root; each file's name and id are printed.
• readFile("First SubFolder", "call-center-inbound-call-2.xls") returns an InputStream for a file located inside that folder.
• The stream is opened as an ExcelDocument and read by an ExcelReader. setSheetName("balance") selects the worksheet and setFieldNamesInFirstRow(true) treats its first row as field names.
• Records are transferred to a StreamWriter bound to System.out via Job.run(), and the record count is printed.
• fileSystem.close() runs in the finally block.