Make tasks container images?

[RobertIsmo]

@joellabes @bstancil

More a discussion than an issue, but would it make sense to make tasks images that we host a container registry of some form? As mentioned in #49 we will start adding larger and larger tasks as time goes on and the shared folder we ship with the repo is already quite large. I feel it may make sense to start separating the env from the code, I can get started on this if this is a suitable direction.

Side note: Are we going to continue hosting the databases off of somebodies google drive😂 No matter what solution is used in the end it will probably makes sense to put these project files there as well for building images.

[joellabes]

Side note: Are we going to continue hosting the databases off of somebodies google drive😂 No matter what solution is used in the end it will probably makes sense to put these project files there as well for building images.

Agreed! It felt weird to include that as a step in my GHA.

make tasks images

I value how easy it is to create a new task at the moment, and the fact that docker is happening under the surface but I don't have to spend any time thinking about it. Would this change that?

[RobertIsmo]

Really good point, honestly it is one of my frustrations with the general docker workflow as a whole(last month I even tried some experiments to rip docker out).

A few ideas here that all basically boil down allowing both local and remote tasks to get the best of both worlds. The some type of compile or bundle step to have your local task submitted to a task repository of some form.

@joellabes @bstancil How does this sound?

[bstancil]

@RobertIsmo I'm not entirely sure I follow. Is the primary aim of this to allow you to run tasks in containers that exist elsewhere?

[RobertIsmo]

@bstancil Hm, that would be an interesting byproduct! I was proposing simplifying the setup a bi. that if it's a container, there wouldn't need to be a separate data download, it would just be a layer of the image. Fundamentally the image would just ship the fs needed to run it.

On the point of existing elsewhere, if it is on a container registry then sharing tasks could be made easier easy and the tasks could be decoupled from the code repo itself. I would think this would make sense if we wanted a release? Open to thoughts there.

[RobertIsmo]

https://getdbt.slack.com/archives/C0A2FTCKTME/p1765817194387649

I think I made the mistake of naming this "Make tasks container images?"(I do think that itself is a good idea, maybe the better term being "Make tasks container LAYERS?") but more this proposal is about decoupling the ADE stuff like tasks from the project and data layer Basically I could containerize my dbt project, run the Ade definition onto of it, have a bunch of tasks defined and just run Ade.

Like the question being "What if we don't use duckdb or snowflake?" the agents are just terminal users, there isn't much stopping us from just dropping it into arbitrary dbt projects and running task definition on it, and from my understanding at the moment there really is no way to run your task definitions on an arbitrary dbt project.

Hopefully I am making sense! Sometimes I can get rambly.

[bstancil]

@RobertIsmo this close to what you're after? I think it's an inverted version of what you're describing, where ade-bench reads from external projects rather than running inside existing projects, but that seemed cleaner to me. But this essentially decouples the harness from tasks, data, and projects, at least in that they can all live in separate places.

#98

[RobertIsmo]

@bstancil Yep basically! Also as I mentioned in the livestream in response to Jason, I am going to be open sourcing a dbt project/data generation app soon so this will be a good testing ground for that. I might add in auto task generation for certain workflows.

[bstancil]

@RobertIsmo ok, dope. Any reason not to close this issue?