README.md

📚 Documentation

Read more tech docs here. Got questions?

Directory

1. Getting started
2. Tutorials
3. Design
4. Production
5. Integrations
6. Guides
7. About
8. Use cases

Getting started

• Setup and install
• Data pipeline management
• Kernels

Tutorials

• Tutorials

Design

• Core design principles
• Core abstractions
• Blocks

Production

• Deploying to cloud
  • Using Terraform
    • Amazon Web Services (AWS)
    • Microsoft Azure
    • Google Cloud Platform (GCP)
  • AWS (without Terraform)
• Version control
  • Setting up Git on cloud
• Configuring production settings
  • Compute resource
  • Runtime variables
• Monitoring and alerting
  • Status updates in Slack
  • Status updates in Email
  • Logging

Integrations

• Compute
  • Spark / PySpark
• Databases, data warehouses, data lakes, etc.
  • BigQuery
  • PostgreSQL
  • Redshift
  • S3
  • Snowflake
• Existing tools
  • Airflow
  • Prefect

Guides

• SQL blocks
• Sensors: blocks depending on external pipelines
• Triggering pipelines
  • Schedule pipelines to run periodically
  • Trigger pipeline via API request
  • Trigger pipeline from AWS Event (WIP)
• Testing
  • Data validation
• Templates

About

• Features
• Roadmap
• Changelog

---

Templates

io_config.yaml

This YAML file contains the information and credentials required to access databases, data warehouses, and data lakes.

Example