com.northconcepts.datapipeline.amazons3

Class AmazonS3FileSystem

java.lang.Object

com.northconcepts.datapipeline.core.DataObject

com.northconcepts.datapipeline.core.Endpoint

com.northconcepts.datapipeline.internal.lang.FileSystem

com.northconcepts.datapipeline.amazons3.AmazonS3FileSystem

public class AmazonS3FileSystem
extends FileSystem

Class for accessing an Amazon S3 file system.

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_QUEUE_SIZE
static int	MINIMUM_PART_SIZE

Fields inherited from class com.northconcepts.datapipeline.core.Endpoint

BUFFER_SIZE, captureElapsedTime, DEFAULT_READ_BUFFER_SIZE

Fields inherited from class com.northconcepts.datapipeline.core.DataObject

id, log, name, TIMESTAMP_FORMAT

Constructor Summary

Constructors

Constructor and Description
AmazonS3FileSystem() Constructs an Amazon S3 file system instance.

Method Summary

Modifier and Type	Method and Description
void	close() Closes the Amazon S3 connection.
void	deleteFile(String bucket, String key) Deletes an S3 object from the specified bucket.
boolean	exists(String bucket, String key) Checks if an S3 object exists in the specified bucket.
AmazonS3	getClient()
AWSCredentials	getCredentials()
AWSCredentialsProvider	getCredentialsProvider()
EndpointConfiguration	getEndpointConfiguration()
boolean	isDebug()
List<Bucket>	listBuckets() Returns the buckets owned by the authenticated user.
ObjectListing	listFolder(String bucket, String folder) Returns the objects in the specified bucket's folder.
ObjectListing	listRootFolder(String bucket) Returns the root objects in the specified bucket.
ObjectListing	nextBatch(ObjectListing previousListing) Returns the next set of paged objects.
void	open() Opens a connection to the Amazon S3 file system.
InputStream	readFile(String bucket, String filePath) Reads the file in the S3 bucket.
AmazonS3FileSystem	setBasicAWSCredentials(String accessKey, String secretKey)
AmazonS3FileSystem	setClient(AmazonS3 client)
AmazonS3FileSystem	setCredentials(AWSCredentials credentials)
AmazonS3FileSystem	setCredentialsProvider(AWSCredentialsProvider credentialsProvider)
AmazonS3FileSystem	setDebug(boolean debug)
AmazonS3FileSystem	setEndpointConfiguration(EndpointConfiguration endpointConfiguration)
AmazonS3FileSystem	setProfileCredentialsProvider()
OutputStream	writeMultipartFile(InitiateMultipartUploadRequest request)
OutputStream	writeMultipartFile(String bucket, String filePath)

Methods inherited from class com.northconcepts.datapipeline.core.Endpoint

addElapsedtime, addExceptionProperties, assertClosed, assertNotOpened, assertOpened, finalize, getClosedOn, getDescription, getElapsedTime, getElapsedTimeAsString, getNestedEndpoint, getOpenedOn, getOpenElapsedTime, getOpenElapsedTimeAsString, getRootEndpoint, getSelfTime, getSelfTimeAsString, getState, isCaptureElapsedTime, isClosed, isOpen, setCaptureElapsedTime, setDescription, toString

Methods inherited from class com.northconcepts.datapipeline.core.DataObject

exception, exception, exception, getId, getName, resetID

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

MINIMUM_PART_SIZE

public static final int MINIMUM_PART_SIZE

See Also:

Constant Field Values

DEFAULT_QUEUE_SIZE

public static final int DEFAULT_QUEUE_SIZE

See Also:

Constant Field Values

Constructor Detail

AmazonS3FileSystem

public AmazonS3FileSystem()

Constructs an Amazon S3 file system instance.

Method Detail

getCredentialsProvider

public AWSCredentialsProvider getCredentialsProvider()

setCredentialsProvider

public AmazonS3FileSystem setCredentialsProvider(AWSCredentialsProvider credentialsProvider)

setProfileCredentialsProvider

public AmazonS3FileSystem setProfileCredentialsProvider()

getCredentials

public AWSCredentials getCredentials()

setCredentials

public AmazonS3FileSystem setCredentials(AWSCredentials credentials)

setBasicAWSCredentials

public AmazonS3FileSystem setBasicAWSCredentials(String accessKey,
                                                 String secretKey)

getEndpointConfiguration

public EndpointConfiguration getEndpointConfiguration()

setEndpointConfiguration

public AmazonS3FileSystem setEndpointConfiguration(EndpointConfiguration endpointConfiguration)

isDebug

public boolean isDebug()

setDebug

public AmazonS3FileSystem setDebug(boolean debug)

open

public void open()
          throws DataException

Opens a connection to the Amazon S3 file system.

Overrides:

open in class Endpoint

Throws:

DataException

close

public void close()
           throws DataException

Closes the Amazon S3 connection.

Overrides:

close in class Endpoint

Throws:

DataException

getClient

public AmazonS3 getClient()

setClient

public AmazonS3FileSystem setClient(AmazonS3 client)

readFile

public InputStream readFile(String bucket,
                            String filePath)

Reads the file in the S3 bucket. To specify a nested folder, use "/" as the path separator.

exists

public boolean exists(String bucket,
                      String key)
               throws DataException

Checks if an S3 object exists in the specified bucket.

Parameters:

bucket - the name of the S3 bucket

key - the key (path) of the S3 object

Returns:

true if the object exists, false otherwise

Throws:

DataException - if bucket or key is empty, or if an error occurs while checking existence

deleteFile

public void deleteFile(String bucket,
                       String key)
                throws DataException

Deletes an S3 object from the specified bucket.

Parameters:

bucket - the name of the S3 bucket

key - the key (path) of the S3 object to delete

Throws:

DataException - if bucket or key is empty, or if an error occurs while deleting the object

writeMultipartFile

public OutputStream writeMultipartFile(String bucket,
                                       String filePath)

writeMultipartFile

public OutputStream writeMultipartFile(InitiateMultipartUploadRequest request)

listBuckets

public List<Bucket> listBuckets()
                         throws Throwable

Returns the buckets owned by the authenticated user.

Throws:

Throwable

listRootFolder

public ObjectListing listRootFolder(String bucket)
                             throws Throwable

Returns the root objects in the specified bucket.

Throws:

Throwable

listFolder

public ObjectListing listFolder(String bucket,
                                String folder)
                         throws Throwable

Returns the objects in the specified bucket's folder.

Throws:

Throwable

nextBatch

public ObjectListing nextBatch(ObjectListing previousListing)
                        throws Throwable

Returns the next set of paged objects.

Throws:

Throwable