Version 1.4.0

Support for dynamic routing:

* refactored codebase to be modules wrt transformers and routers
* created constants to be imported into modules
* added support for mocha tests

Author: IanMeyers

README.md

@@ -128,7 +128,7 @@ eventName
 
 For more information on DynamoDB Update Streams, please read http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html.
 
-By default, the function will append a newline character to received data so that files delivered to S3 are nicely formatted, and easy to load into Amazon Redshift. However, the function also provides a framework to write your own transformers. If you would like to modify the data after it's read from the Stream, but before it's forwarded to Firehose, then you can implement and register a new Javascript function with the following interface:
+By default, the function will append a newline character to received data so that files delivered to S3 are nicely formatted, and easy to load into Amazon Redshift. However, the function also provides a framework to write your own transformers. If you would like to modify the data after it's read from the Stream, but before it's forwarded to Firehose, then you can implement and register a new Javascript function with the following interface (ideally in `transformer.js`:
 
 ```
 function(inputData, callback(err,outputData));
@@ -139,10 +139,11 @@ callback: function to be invoked once transformation is completed, with argument
 	outputData: Buffer instance (typically 'ascii' encoded) which will be forwarded to Firehose
 ```
 
-You then register this transformer function by assigning an instance of it to the exported ```transformer``` instance:
+You then register this transformer function by assigning an instance of it to the exported ```transformer``` instance in the header of `index.js`:
 
 ```
-var transformer = myTransformerFunction.bind(undefined, <internal setup args>);
+// var useTransformer = transform.addNewlineTransformer.bind(undefined);
+var useTransformer = myTransformerFunction.bind(undefined, <internal setup args>);
 ```
 
 You can also take advantage of a built in regex-to-csv transformer, which can be used by un-commenting and configuring the following entry in the function:
@@ -152,6 +153,26 @@ You can also take advantage of a built in regex-to-csv transformer, which can be
 
 Where ```/(myregex) (.*)/``` is the regular expression that uses character classes to capture data from the input stream to export to the CSV, and ```"|"``` is the delimiter.
 
+# Delivery Stream Routing
+As stated previously, data will be forwarded on the basis of a Kinesis tag named `ForwardToFirehoseStream`, and if this isn't found, then it will fall back to a default delivery stream. DynamoDB update streams are always routed to the delivery stream with the same name as the base table.
+
+In version 1.4.0, we added the ability to do dynamic routing. For example, you might want to route to different destinations on S3 or Redshift on the basis of the actual data being received. You can now use this by overriding the default routing, and providing a map of who records should be routed. You do this by changing the `router.defaultRouting` method to be `router.routeByAttributeMapping`. When done, you need to previde an 'attribute delivery map' which tells the router which fields to look at in your data, and how to route based on their values. You do this with a configuration object - for example to route by the value of an attribute `binaryValue` that can only be `true` or `false`:
+
+```
+var attributeMap = {
+    "binaryValue" : {
+	"true" : "TestRouting-route-A",
+	"false" : "TestRouting-route-B"
+    }
+};
+```
+
+this attribute map is then used to configure the router instance:
+
+```
+var useRouter = router.routeByAttributeMapping.bind(undefined, attributeMap);
+```
+
 # Confirming Successful Execution
 
 When successfully configured, writes to your Stream will be automatically forwarded to the Firehose Delivery Stream, and you'll see data arriving in Amazon S3 and optionally Amazon Redshift. You can also view [CloudWatch Logs](http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/WhatIsCloudWatchLogs.html) for this Lambda function as it forwards stream data to Firehose

build.sh

@@ -26,7 +26,7 @@ npm install
 
 rm $filename 2>&1 >> /dev/null
 
-zip -r $filename index.js package.json node_modules/ README.md LICENSE && mv -f $filename dist/$filename
+zip -x \*node_modules/protobufjs/tests/\* -r $filename index.js router.js transformer.js constants.js lambda.json package.json node_modules/ README.md LICENSE NOTICE.txt && mv -f $filename dist/$filename
 
 if [ "$1" = "true" ]; then
   aws lambda update-function-code --function-name $functionName --zip-file fileb://dist/$filename --region $region

constants.js

@@ -0,0 +1,9 @@
+OK = 'OK';
+ERROR = 'ERROR';
+FORWARD_TO_FIREHOSE_STREAM = "ForwardToFirehoseStream";
+DDB_SERVICE_NAME = "aws:dynamodb";
+KINESIS_SERVICE_NAME = "aws:kinesis";
+FIREHOSE_MAX_BATCH_COUNT = 500;
+// firehose max PutRecordBatch size 4MB
+FIREHOSE_MAX_BATCH_BYTES = 4 * 1024 * 1024;
+targetEncoding = "utf8";