Local Dagster Orchestration of Batch Pipelines

[jkislin]

Summary

We're working on getting scheduled and orchestrated versions of CFA Azure Batch Pipelines into Dagster. PyRenew-HEW is one of our pilot projects, alongside the CFA Epinow2 Pipeline.

This PR allows users to run dagster's User Interface locally on their VAP while submitting jobs to Azure Batch.

It also provides a Makefile target that allows one to publish pyrenew-hew's codebase to the central dagster server, but central orchestration is not configured quite yet. More to come there soon!

Details

This is a large undertaking, and we are learning along the way. This PR allows users to demo dagster for PyRenew-HEW and report back on bugs and desired features before we move to a production-ready version. It includes changes to existing files used in other workflows, such as the pyproject.toml and Containerfile, so we are testing to make sure no cross-compatibility issues arise. Once fully tested and reviewed, we can merge this PR.

Currently ready for review (12/11/25):

1. You can now launch a local dagster server that can submit jobs to Azure Batch. Currently, feature parity is similar to the Azure Command Center. Also currently, the workflows are hard-coded to output to pyrenew-test-output. This can be changed.
2. The Makefile now includes helpful commands for dagster and blobfuse for PyRenew-HEW.
3. There are instructions on how to setup dagster locally for your user/VM and how to submit to our central dagster code server.
4. The Containerfile now pins to rocker/tidyverse:4.5.1 rather than latest.
5. The Containerfile now includes the dagster_defs.py workflow file and manages dependencies for it unified with the PyRenew-HEW project dependencies. It also takes a more explicit approach to bringing in project files.

Instructions for review:

To evaluate dagster, follow the steps in the README.

[codecov]

Codecov Report

❌ Patch coverage is 0% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 42.39%. Comparing base (11ef9b7) to head (8ffea8a).

Files with missing lines	Patch %	Lines
pipelines/batch/setup_job.py	0.00%	3 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #714   +/-   ##
=======================================
  Coverage   42.39%   42.39%           
=======================================
  Files          30       30           
  Lines        2835     2835           
=======================================
  Hits         1202     1202           
  Misses       1633     1633           

Flag	Coverage Δ
hewr	38.04% <ø> (ø)
pipelines	37.33% <0.00%> (ø)
pyrenew_hew	62.29% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jkislin]

My proposal:

1. After review, we can merge this PR.
2. After that PR is merged, I'll follow up with a PR that moves the now-merged dagster code from 'test' to 'prod', including the container images.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 20 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.