How to structure a project with multiple long-running Python data loaders

[kyrre]

For our data apps we want to have to have data loaders that triggers long-running operations, e.g., Databricks jobs, Snowflake queries and DuckDB running on some container somewhere.

They way we have structured the project for now is to have each data loader be a separate DAB / Python project that contains all the code and dependencies in one location, so for src/data/azure-applications we have this layout:

├── README.md
├── [service_principal_name].parquet.py
├── azure_applications
│   ├── __init__.py
│   ├── task.py
│   └── test_task.py
├── conftest.py
├── databricks.yml
├── dist
│   └── azure_applications-0.0.3-py3-none-any.whl
├── pyproject.toml
├── resources
│   └── azure_applications_job.yml
└── uv.lock

The file azure_applications/task.py is the entry point, which executes the Databricks job that runs the query and then writes the results to object storage (in Parquet format). DAB allows us to specify prod and dev configurations for the job and provides a way of building the wheel file with all dependencies, which is then synced to Databricks and used in the job.

class AzureApplications(DataLoader):
    def __init__(self, con, service_principal_name):
        self.con = con
        self.service_principal_name = service_principal_name.lower()

    def run(self) -> ibis.Table:
        cloud_app_events = (
            self.con.table(
                "cloud_app_events",
                database="old_security_logs.mde",
            )
       ...

        azure_application_events = cloud_app_events.filter(
            [
                _.ObjectName == self.service_principal_name,
                _.ActionType.isin(azure_application_action_types),
            ]
        )

        azure_application_events.to_delta(
              "abfss://...//observable/azure_applications"
        )

The data loader itself [service_principal_name].parquet.py] triggers the above job, waits for it to finish and then retrieves the Parquet files it produces before by writing them to stdout. Simplified it boils down to:

w = WorkspaceClient(profile='security')

# trigger the job above and wait 🤷🏻 
run = w.jobs.run_now_and_wait(
    job_id=169185052009282,
)

t = read_table(path="observable/azure_applications", container="playground")
write_dataframe_to_stdout(t)

However, when I do this the build automatically generates a lot of incorrect pages. For example, each README.md file in the .venv will result in a page.

We can ignore or get rid of the nested .venv somehow, but all this has got me thinking that it might be it's better to have some form of uv "monorepo" that builds multiple wheel files?

I am also wondering where in the CI/CD pipeline we should trigger these Databricks jobs? 🤔

Any advice on this would be much appreciated! 🙇🏻

[mbostock]

Perhaps we should add support for a .observablehqignore file that allows you to ignore any pages within .venv? Or perhaps we should ignore all subdirectories in src that start with . by default?

The current list of pages is generated here:

framework/src/config.ts

Lines 212 to 213 in 82412a4

	for (const file of visitFiles(root, (name) => !isParameterized(name))) {
	if (extname(file) !== ".md" || file === "index.md" || file === "404.md") continue;

The required top-level /index page is further added here:

framework/src/pager.ts

Line 60 in 82412a4

if (loaders.findPage("/index")) visit({name: title, path: "/index", pager: "main"});

[kyrre]

I like the idea of having a .observablehqignore and being explicit about what to ignore!