README.md

📚 Documentation

Read more tech docs here. Got questions?

Directory

1. Getting started
2. Tutorials
3. Design
4. Production
5. Integrations
6. Guides
7. About
8. Use cases
9. Best practices (coming soon)

Getting started

• Setup and install
• Data pipeline management
• Kernels

Tutorials

• Tutorials

Design

• Core design principles
• Core abstractions
• Blocks

Developing in the cloud

• Version control
  • Setting up Git on cloud
• Multi-development environment in the cloud

Production

• Deploying to cloud
  • Using Terraform
    • Amazon Web Services (AWS)
    • Microsoft Azure
    • Google Cloud Platform (GCP)
  • AWS (without Terraform)
• Configuring production settings
  • Compute resource
  • Runtime variables
• Observability
  • Alerting status updates in Slack
  • Alerting status updates in Email
  • Logging
  • Monitoring

Integrations

• Syncing data between 3rd party sources and destinations
• Compute
  • Spark / PySpark
• Databases, data warehouses, data lakes, etc.
  • BigQuery
  • PostgreSQL
  • Redshift
  • S3
  • Snowflake
• Existing tools
  • Airflow
  • Prefect

Guides

• Building pipelines
  • Batch pipeline
  • Streaming pipeline
  • Data integration pipeline
• SQL blocks
• Sensors: blocks depending on external pipelines
• Triggering pipelines
  • Schedule pipelines to run periodically
  • Trigger pipeline from event
  • Trigger pipeline via API request
  • Trigger pipeline from AWS Event (WIP)
• Testing
  • Data validation
• Templates

About

• Features
• Roadmap
• Changelog

---

Templates

io_config.yaml

This YAML file contains the information and credentials required to access databases, data warehouses, and data lakes.

Example