📚 Documentation

Read more tech docs here. Got questions?

Directory

1. Getting started
2. Tutorials
3. Design
4. Production
5. Integrations
6. Guides
7. About
8. Use cases
9. Best practices (coming soon)

Getting started

• Setup and install
• Data pipeline management
• Kernels

Tutorials

• Tutorials

Design

• Core design principles
• Core abstractions
• Blocks

Production

• Deploying to cloud
  • Using Terraform
    • Amazon Web Services (AWS)
    • Microsoft Azure
    • Google Cloud Platform (GCP)
  • AWS (without Terraform)
• Version control
  • Setting up Git on cloud
• Configuring production settings
  • Compute resource
  • Runtime variables
• Monitoring and alerting
  • Status updates in Slack
  • Status updates in Email
  • Logging

Integrations

• Compute
  • Spark / PySpark
• Databases, data warehouses, data lakes, etc.
  • BigQuery
  • PostgreSQL
  • Redshift
  • S3
  • Snowflake
• Existing tools
  • Airflow
  • Prefect

Guides

• Building pipelines
  • Batch pipeline
  • Streaming pipeline
  • Data integration pipeline (coming soon)
• SQL blocks
• Sensors: blocks depending on external pipelines
• Triggering pipelines
  • Schedule pipelines to run periodically
  • Trigger pipeline from event
  • Trigger pipeline via API request
  • Trigger pipeline from AWS Event (WIP)
• Testing
  • Data validation
• Templates

About

• Features
• Roadmap
• Changelog

---

Templates

io_config.yaml

This YAML file contains the information and credentials required to access databases, data warehouses, and data lakes.

Example