Beam Dataflow Pipelines

This repository contains a set of Apache Beam pipelines examples with advance features. Each one of the examples provides new concepts over the basic functionality of Apache Beam. These pipelines have been designed to run in Google's Dataflow service because majority of the features are related to Google Cloud ecosystem, such as Spanner or BigQuery.

Examples

All pipelines described here focus on batch processing. The collection includes:

1. SideInputSchema is a simple dummy pipeline that demonstrates how to use Side Inputs from an external file and introduces a custom Record class to process inside a PCollection.

2. SpannerIngester is a pipeline that writes an external file into Spanner. The pipeline can be reused from one table to another thanks to the custom Record. It writes into Spanner using the SpannerIO class and shows how to handle the Mutations.