README.md

📡 Data pipeline management

Table of contents

1. Setup
2. Pipelines
   1. Creating new pipeline
3. Pipeline runs
4. Pipeline detail
5. Triggers
   1. Create trigger
   2. Trigger detail
6. Runs
   1. Retry runs (WIP)
7. Logs
8. Backfill (WIP)
9. Monitor (WIP)

Setup

If you haven’t setup a project before, check out the setup guide before starting.

Pipelines

_{http://localhost:3000/pipelines}

This page will show all the pipelines in your project.

Core abstraction: Pipeline

A pipeline contains references to all the blocks of code you want to run, charts for visualizing data, and organizes the dependency between each block of code.

_{Learn more about projects and pipelines here.}

From this page, you can also create a new pipeline by clicking the [+ New pipeline] button.

Creating new pipeline

Creating a new pipeline will take you to the Pipeline edit page; a notebook-like experience for adding blocks, creating dependencies between blocks, testing code, and visualizing data with charts.

_{Learn more about the Notebook for building data pipelines}.

Pipeline runs

_{http://localhost:3000/pipeline-runs}

View all the runs for every pipeline in your current project.

Core abstraction: Run

A run record stores information about when it was started, its status, when it was completed, any runtime variables used in the execution of the pipeline or block, etc.

Pipeline detail

_{http://localhost:3000/pipelines/[uuid]}

This page contains all the information and history for a single pipeline:

1. Triggers
2. Runs
3. Logs

Triggers

_{http://localhost:3000/pipelines/example_pipeline/triggers}

This page shows all the active and inactive triggers for a single pipeline.

Core abstraction: Trigger

A trigger is a set of instructions that determine when or how a pipeline should run.

Create trigger

_{http://localhost:3000/pipelines/[uuid]/triggers/[id]/edit}

Create a new trigger for this pipeline by clicking the [+ Create] button near the top of the page.

You can configure the trigger to run the pipeline on a schedule or when an event occurs.

Core abstraction: Schedule

A schedule type trigger will instruct the pipeline to run after a start date and on a set interval.

Core abstraction: Event

An event type trigger will instruct the pipeline to run whenever a specific event occurs.

_{Example page for creating a schedule type trigger.}

Trigger detail

On this page, you can start or pause the trigger. Starting the trigger will make it active. Pausing the trigger will prevent it from running the pipeline.

_{Note: if you have other triggers for this pipeline, pausing 1 trigger may not stop the pipeline from running since other triggers can also run the pipeline.}

You can also edit the trigger after creating it by clicking the [Edit trigger] button.

Runs

_{http://localhost:3000/pipelines/example_pipeline/runs}

View the pipeline runs and block runs for a pipeline.

Core abstraction: Run

A run record stores information about when it was started, its status, when it was completed, any runtime variables used in the execution of the pipeline or block, etc.

Retry run

WIP

Logs

_{http://localhost:3000/pipelines/example_pipeline/logs}

Browse all logs for a pipeline. You can search and filter logs by log level, block type, block UUID, and more.

Core abstraction: Log

A log is a file that contains system output information.

Backfill

WIP

Monitor

WIP