feat: initial implementation of project file format

[bitwalker]

This PR implements a new crate, miden-project, which implements the data structures for representing Miden projects, along with the AST representation of the miden-project.toml file initially sketched out in #1919. Note that the actual syntax of miden-project.toml as implemented in this PR differs from that initial sketch in a few ways. See the example project layouts in crates/project/examples for the syntax of workspace-based and standalone project files.

I'm still making a few small adjustments, and implementing tests for the parsing/validation bits, but this should be more or less reviewable as-is, at least in terms of the high-level design.

Links

• Original RFC for miden-project.toml can be found in RFC: Introduce miden-project.toml manifest file #1919
• Discussion on dependency specs and versioning can be found here

TODO

• Finish implementing tests
• Add the necessary pubgrub plumbing for package selection during dependency resolution
• Clean up pass (remove unused dependencies, make sure everything is correctly documented, remove unused APIs)

[greenhat]

From #1919:

The Rust tooling uses/reproduces much of this information in Cargo.toml - we could move most of that to miden-project.toml instead, or alternatively, simply merge the relevant sections from Cargo.toml into an in-memory miden-project.toml manifest. Much of what I've laid out here is in fact the parts we already define/maintain in Cargo.toml for Rust-based projects today

I wonder what our plan is for Rust projects?
Currently we store all Miden-related fields in the [package.metadata.miden] section of the Cargo.toml. See a P2ID example.

Do we want to require miden-project.toml and move the Miden fields currently residing in the Cargo.toml there? Multiple fields end up duplicated (name, version, etc.). Plus, the developer would have package dependencies defined in the miden-project.toml and the crate dependencies in the Cargo.toml.

I'd suggest to keep storing all Miden fields in Cargo.toml. The cargo miden build can embed generated miden-project.toml in the compiled Miden package.

[bitwalker]

I think the issue with storing everything in Cargo.toml has a few issues:

1. We will have tooling, and there will be third-party tooling, built around Miden projects, and storing project information in Cargo.toml requires those tools to understand not only miden-project.toml, but also Cargo.toml and however we choose to encode miden-projet.toml information inside of it.
2. In the future, other languages that can compile to Miden may not have a TOML configuration file (or at least one that can be piggy-backed on like Cargo.toml), in which case they would need to use miden-project.toml and would face the same dilemma of duplication between language-specific project config and Miden project config. Being consistent about the boundary between language-specific config and Miden config up front will lead to less confusion IMO.
3. It complicates IDE/LSP integration, since Cargo.toml is already used as a language marker for Rust, it would require parsing Cargo.toml to determine if the directory is a Miden/Rust project, or just a Rust project. In some editors it may not be a problem, if extensions are provided an opportunity to do this when a worktree is opened in the editor (e.g. Zed), but I don't believe that is always the case.
4. Even if we were to use Cargo.toml for Miden+Rust projects, we still have a situation where we are specifying dependencies in two places in Cargo.toml - one for normal crates, and then one for Miden packages under ([package.metadata.miden.dependencies]). IMO, it's more likely to confuse users that both sets of dependencies are specified in different places in the same file. If Miden package dependencies are in miden-project.toml and pure Rust dependencies are in Cargo.toml, the delineation between the two is a lot more clear (and easier to explain).
5. Rust crates which are compiled to Miden will not be published to crates.io - so virtually all of the metadata that would typically go in Cargo.toml can be specified in miden-project.toml instead, aside from name and version. GIven that, it isn't clear to me how much information would actually be duplicated, or how much of a pain that would be in practice. Changing the name is going to be very rare. Changing version is more frequent, so if it is considered painful enough to have to maintain it in two places, we could provide a solution there that allows specifying in miden-project.toml that the version is to be inferred from external/language-specific tooling, something like version = { external = true, language = "rust" } | { external = true, path = "VERSION" }. TBH, I don't think it is actually a significant enough pain point for that, but 🤷. Aside from version though - there just isn't overlap between Cargo.toml and miden-project.toml to combine them IMO.

I'm not saying I couldn't be persuaded on this point, but it does feel like preemptively addressing an issue that we don't actually know is going to be an issue in practice. I would want to see how much duplication there actually is (given the constraints I outlined above, e.g. defining description in Cargo.toml doesn't make any sense), and from there determine if it is a significant enough problem to warrant complicating project handling by allowing Rust projects to piggy-back on Cargo.toml.

[greenhat]

Fair points. I'm sold on the project-level miden-project.toml.

The workspace-level miden-project.toml still bothers me due to the members and profile duplication with Cargo.toml. Assuming they have to be maintained by the user.

[greenhat]

I like the version-or-digest approach and that we'll store it in the same field. Apart from my question about the workspace level in Rust projects at #2510 (comment) everything sounds good.

[bitwalker]

This is ready for review, though I expect to add a few more tests still, and write up a README doc that describes the usage and syntax of the project file.

A few design details to note during your review:

• No changes are made to existing crates in this repo for this PR - this is all greenfield code in a new crate
• The majority of this PR's content can be broken up into three categories, which can be reviewed independently to make it easier to focus on the relevant details:
  1. The format/syntax/features of the project file, the data structures that represent its AST, and the data structures that represent the loaded/validated representation which is derived from the AST.
  2. The basic serialization boilerplate, the parsing functions that deserialize the AST from TOML source code, and the code to translate from the AST to the final user-facing data structures.
  3. The dependency resolver and associated implementation details. Note that the resolver does not actually fetch/load packages, it provides only the high-level resolution and version selection based on a simple index of packages, their versions, and the dependency requirements of those versions.
• There are a handful of examples in the crate which are used in the tests, and provide a sense of the syntax and relationship between project file and directory structure/packages.
• TargetType is essentially a duplicate of PackageKind, and I expect that PackageKind will be replaced with TargetType in subsequent PRs.
• I expect to make miden_project::Package derivable from a miden_mast_package::Package, or even embed the manifest directly in the latter, in subsequent PRs. Either way, the goal would be that if you have a miden_mast_package::Package, you have access to the version of the manifest used to assemble that package. This can then be used in dependency resolution, etc.
• The dependency resolution algorithm is provided by pubgrub, and is set up to largely mimic Cargo's dependency resolution behavior, with some simplifications. Additionally, our usage of it extends its behavior to support version requirements that use content digests in addition to, or in lieu of semantic version constraints. This aspect of the PR is experimental, and likely to change/evolve with further testing, but is an essential component of downstream tooling that will build on this, so some implementation of its functionality was needed.

If you have any questions about specific details, leave a comment here or ping me to discuss, I'm happy to walk through any of this, since there is a lot of context that isn't easy to glean from just the source code or its documentation.

[bitwalker]

I like the version-or-digest approach and that we'll store it in the same field. Apart from my question about the workspace level in Rust projects at #2510 (comment) everything sounds good.

The duplication is hard to avoid at the workspace level. We could derive the Workspace for Rust projects from Cargo.toml, but that leaves the issue that I mentioned before, which is that this will break expectations for tooling that only cares about Miden projects in general, and not the language-specific details.

Regardless, we need the workspace-level project file for workspace management of Miden Assembly projects anyway, so the issue of duplication with Cargo.toml is something specific to the Rust toolchain, rather than an issue to be addressed here, IMO.