Read Google Gmail Messages

This example shows how to read and interact with Gmail messages from Google accounts. It enables you to access email content, metadata, and attachments to open up a wide range of use cases. You can use DataPipeline for email automation, email archiving, tracking customer support requests, and sentiment analysis of email communications.

Java Code Listing

package com.northconcepts.datapipeline.examples.google.gmail;

import java.io.File;
import java.io.InputStreamReader;
import java.util.Collections;
import java.util.List;

import com.google.api.client.auth.oauth2.Credential;
import com.google.api.client.extensions.java6.auth.oauth2.AuthorizationCodeInstalledApp;
import com.google.api.client.extensions.jetty.auth.oauth2.LocalServerReceiver;
import com.google.api.client.googleapis.auth.oauth2.GoogleAuthorizationCodeFlow;
import com.google.api.client.googleapis.auth.oauth2.GoogleClientSecrets;
import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.client.util.store.FileDataStoreFactory;
import com.google.api.services.gmail.GmailScopes;
import com.northconcepts.datapipeline.core.DataReader;
import com.northconcepts.datapipeline.core.DataWriter;
import com.northconcepts.datapipeline.core.LimitReader;
import com.northconcepts.datapipeline.core.StreamWriter;
import com.northconcepts.datapipeline.google.gmail.GmailMessageReader;
import com.northconcepts.datapipeline.job.Job;

/**
 * 
 * CLIENT_SECRET is your OAuth2 client ID created in the Google Developer Console. 
 * http://localhost:8888/Callback needs to be added as an authorized redirect URIs
 * where 8888 is the port number set in line 53.
 */
public class ReadGmailMessages {

    private static final int MESSAGE_COUNT = 10;
    private static final int BATCH_SIZE = 10;
    private static final String CLIENT_SECRET = "/integrations.json";
    private static final String APPLICATION_NAME = "GmailClient";
    private static final JsonFactory JSON_FACTORY = JacksonFactory.getDefaultInstance();
    private static final String TOKENS_DIRECTORY_PATH = "tokens/gmail";
    private static final List SCOPES = Collections.singletonList(GmailScopes.GMAIL_READONLY);

    /** Authorizes the installed application to access user's protected data. */
    private static Credential authorize(NetHttpTransport httpTransport) throws Throwable {

        // load client secrets
        GoogleClientSecrets clientSecrets = GoogleClientSecrets.load(JSON_FACTORY,
                new InputStreamReader(ReadGmailMessages.class.getResourceAsStream(CLIENT_SECRET)));
        // set up authorization code flow
        GoogleAuthorizationCodeFlow flow = new GoogleAuthorizationCodeFlow.Builder(
                httpTransport, JSON_FACTORY, clientSecrets, SCOPES)
                .setDataStoreFactory(new FileDataStoreFactory(new File(TOKENS_DIRECTORY_PATH)))
                .setAccessType("offline")
                .build();
        // authorize
        LocalServerReceiver receiver = new LocalServerReceiver.Builder().setPort(8888).build();

        return new AuthorizationCodeInstalledApp(flow, receiver).authorize("user");
    }

    public static void main(String[] args) throws Throwable {

        NetHttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();

        DataReader reader = new GmailMessageReader(authorize(httpTransport), httpTransport)
                .setBatchSize(BATCH_SIZE)
                .setApplicationName(APPLICATION_NAME);
        reader = new LimitReader(reader, MESSAGE_COUNT);

        DataWriter writer = StreamWriter.newSystemOutWriter();

        Job.run(reader, writer);
    }
}

Code Walkthrough

1. authorize() method authorizes the installed application to access the user's protected data. First, the client secret (i.e. your service account credentials created in the Google Developer Console) is loaded using GoogleClientSecrets.load() method. Next, the authorization code flow is set up, and, finally, the application is authorized to retrieve data from Google API. 
2. NetHttpTransport instance is created for thread-safe HTTP low-level transport.
3. GmailMessageReader uses an API to get messages from a Google account. The constructor of the instance accepts Credential and HttpTransport. Additional parameters such as batch size and application name are specified using corresponding methods.
4. LimitReader is created to limit the number of records in a reader.
5. StreamWriter.newSystemOutWriter() method is used to create DataWriter instance.
6. Job.run() method is used to transfer the data from reader to writer to print out the records on the console. See how to compile and run data pipeline jobs.