Description

This repo contains implementations of Airflow workflows and tasks called respectively DAGs and Operators.

DAGs - Direct Acyclic Graphs - Python scripts defining workflows in a way that reflects their relationships.
Operators - Python functions which define the individual tasks that are executed as part of a DAG run.

To learn how to write DAGs and Operators read about core concepts and follow the official tutorial.

DAG

This repository contains:

website_sync: DAG to launch the Airbyte jobs for the status-website charts.
spiff_sync: DAG to synchronize Spiff workflows data
dbt: DAG to run all the dbt models,
gh_sync: DAG to synchronize data from repository (logos, waku, codex)
status_analytics: DAG to extract and clean data from the google play store and Mixpanel
shopify: DAG to extract and clean data from Shopify
budget: DAG to extract and clean data from an excel sheet where there are budget transactions and forecast data fron Finance team.

The DBT models run in some DAG are stored in dbt-models.

Changes pushed to master are automatically fetched to our Airflow instance by the airflow-webhook service.

This repos has 3 working branches:

All Airflow infrastructure is managed in the infra-bi repository.

Name		Name	Last commit message	Last commit date
Latest commit History 154 Commits
providers/airbyte		providers/airbyte
tasks		tasks
.gitignore		.gitignore
README.md		README.md
bi_reporting.py		bi_reporting.py
budget.py		budget.py
comm_extraction.py		comm_extraction.py
dbt.py		dbt.py
forums_sync.py		forums_sync.py
github_extractions.py		github_extractions.py
grafana_backup.py		grafana_backup.py
grafana_test_to_prod.py		grafana_test_to_prod.py
infra_sync.py		infra_sync.py
logos_org_map.py		logos_org_map.py
marketing_metabase_sync.py		marketing_metabase_sync.py
organisation_sync.py		organisation_sync.py
recruitment.py		recruitment.py
seed.py		seed.py
shopify.py		shopify.py
status_analytics.py		status_analytics.py
treasure_dashboard.py		treasure_dashboard.py
website_sync.py		website_sync.py