This repository demonstrates how to automate spark job on GCP dataproc cluster using CloudComposer(an managed AirFlow service on GCP) and to perform CI/CD using CloudBuild.
Tools Used:
GCP Dataproc
GCP Cloud Build
GCP Cloud Composer
Unitest module
Docker container to run tests and deploy the code DAGS folder
unittest