arch-evol-1

Document Storage using S3 and Dynamo Workshop

In this workshop, we will write Java code to save objects to S3 and metadata about those objects to DynamoDB. The workshop assumes that you have an AWS account with the required credentials for interacting with S3 and DynamoDB, and that those credentials are configured on your workstation.

Problem Statement

Our solution must meet the following requirements/specifications:

1. We intend to store documents in S3 that contain a message with the current time in milliseconds of the format The current time in milliseconds is {current time}
2. When we store a document, we will create a new user key that will be a UUID and associate that user UUID with the object in S3
3. We need to be able to look up an S3 object key by a user UUID

Creating the S3 Bucket

First we will create an S3 bucket to store our documents

1. In the AWS console, navigate to the S3 service
2. Click "Create Bucket"
3. Create a bucket named nuvalence-workshop-{your name}
4. Leave the rest of the default settings (No encryption, no public access, no versioning)

Creating the DynamoDB Table

Next we will create a table for our object metadata

1. In the AWS console, navigate to the DynamoDB service
2. Click "Create Table"
3. Specify nuvalence-workshop-{your name} as the table name
4. Specify associatedUser as the partition key with type String
5. Click the checkbox to add a sort key
6. Specify objectKey as the sort key with type String
7. Uncheck the box for "Use default settings"
8. Under "Auto Scaling", uncheck the box for both "Read capacity" and "Write capacity"
9. Under "Provisioned capacity", set the RCU and WCU both to 1

CHECKPOINT - If you do not have an S3 bucket and DynamoDB, go back and start from the top of this workshop

Creating the DocumentStorageWorkshopClass

We are now ready to build the class that will create our object in S3 and store the metadata in DynamoDB. You can copy the example solution from the workshop directory or begin with the empty base provided and try to implement it yourself!

The base already provides a constructor for our class that does three things:

1. Instantiate an S3 client
2. Instantiate a DynamoDB client
3. Read environment variables corresponding to the names of both the bucket and table

Let's now begin building the logic.

Create the key information we want to store in S3

We need three pieces information, the current time, a random UUID and the key to be used for our object. Let's define them. Copy the following under the appropriate section in your class.

final long time = System.currentTimeMillis(); 
final UUID userGUID = UUID.randomUUID(); 
final String objectKey = "document/" + time + ".txt";

Create object metadata to be used when submitting our object to S3

The following section is necessary to create metadata about the object that we will be uploading to S3. It is different than the metadata being stored in DynamoDB (which is a business requirement).

ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentType("plain/text"); // set the content type of the file to be uploaded
metadata.addUserMetadata("x-amz-meta-title", "Hello"); // set a title for the file

Create InputStream object representing the content of our file

InputStream stream = new StringInputStream("The time is " + time);

Create a put object request and use the S3 client to put our object into S3 at the specified key

We now use the S3 client instantiated as part of the constructor of this class to upload our file to S3.

PutObjectRequest request = new PutObjectRequest(bucketName, objectKey, stream, metadata);
s3Client.putObject(request);

System.out.println("Successfully put object in S3 at key " + objectKey);

Create the object to be stored in DynamoDB

For the purpose of this exercise, we want to create a HashMap object of type <String,AttributeValue> and add three values, associatedUser, objectKey and data. We then will put this object to our DynamoDB table.

item.put("associatedUser", new AttributeValue(userGUID.toString()));
item.put("objectKey", new AttributeValue(objectKey));
item.put("date", new AttributeValue().withN(Long.toString(time)));

Upload item to DynamoDB

Once we created our metadata object, it's time to upload it to DynamoDB.

System.out.println("Adding new item...");
this.dynamo.putItem(this.tableName, item);
System.out.println("Success! Added item to Dynamo for user " + userGUID + " and object key " + objectKey);

Print out the current state of both the bucket and the table

As a last step, we want to output the contents of the bucket and the dynamo table. We can retrieve all current items and loop through them using the following snippet.

 System.out.println("Current bucket contents:");
ListObjectsV2Result result = s3Client.listObjectsV2(bucketName);
result.getObjectSummaries()
        .stream()
        .map(S3ObjectSummary::getKey)
        .forEach(System.out::println);

System.out.println("Current table contents:");
ScanResult dbresult = dynamo.scan(tableName, Arrays.asList("associatedUser", "objectKey", "date"));
dbresult.getItems().stream()
       .map(m -> String.format("%s - %s - %s", m.get("associatedUser").getS(), m.get("objectKey").getS(), m.get("date").getN()))
       .forEach(System.out::println);

Running the Application

There are a few things you need to do in order to run the solution. If you started with the empty base class, start on step 1, otherwise go to step 2.

1. Build your project using ./gradlew build at the root of your source
2. Configure the environment variables with the name of your S3 bucket and DynamoDB table (BUCKET_NAME and TABLE_NAME)
3. Run your application and remember to pass docs as an argument