This repository contains a example environment that can be used by anyone who wants how to code pipelines in Airflow
- Python 3.6 or higher https://www.python.org/
- Docker https://docs.docker.com/get-docker/
- Docker Compose https://docs.docker.com/compose/install/
- MySql client (ubuntu:
sudo apt-get install libmysqlclient-dev, mac:brew install mysql) - ONlY on Mac - Developer tools
xcode-select --install
To start using repository just go in your terminal to folder when project was
cloned and type
python3 teach_aiflow.py
What gonna happen:
- Docker will build all needed images
- By Docker Compose a local cluster will be raised
- Example data wil be loaded into databases
- Virtual environment will be created
- All dependencies needed for Airflow will be installed inside virtual environment
Pre check
If for command docker ps you get error that cannot connect to docker engine
please try to run it with sudo. If that work all commands here should be run with sudo as well.
all should be run as python3 teach_airflow.py <argument>
--reload_data Reload data in all databases
--start Start airflow docker containers
--stop Stop airflow docker containers
--restart Restart airflow docker containers
--remove Delete all docker images, containers
--rebuild Rebuild airflow env
--add_vir_env Add virtual env
--rm_vir_env Remove virtual env
* login: airflow
* password: airflow
* database: airflow
* host: localhost
* port: 5433
* login: airflow
* password: airflow
* database: public
* host: localhost
* port: 3307
* username: airflow
* password: airflow
* host: localhost
* port: 21
* login: airflow
* password: airflow
* database: airflow
* host: postgres
* port: 5432
* login: airflow
* password: airflow
* database: public
* host: mysql
* port: 3306
* username: airflow
* password: airflow
* host: ftp
* port: 21