Skip to content

feat(sdk): promote the artifact digest recipe into wcm - #103

Merged
imran-siddique merged 1 commit into
mainfrom
feat/artifact-digest-recipe
Aug 27, 2026
Merged

feat(sdk): promote the artifact digest recipe into wcm#103
imran-siddique merged 1 commit into
mainfrom
feat/artifact-digest-recipe

Conversation

@imran-siddique

Copy link
Copy Markdown
Member

SPEC.md takes weights_hash as given. It says nothing about how a directory of shards, indexes and tokenizer assets collapses into one value, so every consumer invented that step.

By 2026-08-27 the same construction existed in three places outside this repository: the examples repository's open-model walkthrough, and two Marketplace integrations added in agentrust-io/integrations#142.

The failure mode of a recipe existing several times is not that one is wrong. It is that they drift, a manifest produced by one path stops verifying on another, and the mismatch surfaces as weights_hash not matching, which reads as tampered weights rather than as a tooling disagreement.

What lands

wcm.artifact_digest, exporting RECIPE_ID (wcm-artifact-digest/v1), artifact_files, artifact_digest, ArtifactDigestError. artifact_digest returns a HashValue, so the result drops into weights_hash or a serving-image measurement without restringing.

The construction, unchanged from the copies:

  • Files sorted by POSIX relative path, so ordering does not depend on readdir order or the platform separator.
  • Per file: relative path length as 8 bytes big-endian, the path bytes, the file size as 8 bytes big-endian, then contents.
  • .cache, .git, .gitattributes, .huggingface excluded, since they differ between two caches holding byte-identical weights.

Length prefixing is the part that earns its own test: without it a + bc and ab + c flatten to one byte stream, so two different layouts would produce one digest.

Convention, not specification

RECIPE_ID exists so a mismatch can be attributed to the recipe rather than the bytes, and so a future construction becomes /v2 instead of silently replacing this one.

A deployment computing weights_hash some other way is not non-conforming. It simply must not expect this function to agree. Nothing normative moves in this PR.

Two things the copies did not do

Symlinks are refused by default. A link is a name resolving somewhere else, so following one lets a digest cover bytes outside the artifact, and lets the covered bytes change without anything in the artifact changing. test_symlink_target_change_is_what_the_refusal_prevents demonstrates exactly that: it rewrites a link target and watches the digest move while the tree does not. follow_symlinks=True accepts it deliberately.

An inventory emptied by exclusion raises, rather than returning a well-formed digest over nothing, which a directory containing only .cache would otherwise produce.

Verification

Cross-checked byte-identical against both existing integration copies, over a tree with nested directories, multiple shards and a .cache sibling, on the plain and the include= paths:

sdk      sha256:6e05c85c239dd32b10ab0073fb4166d35ba90c896512f29b455fd2a94c508d51
hf gate  sha256:6e05c85c239dd32b10ab0073fb4166d35ba90c896512f29b455fd2a94c508d51
triton   sha256:6e05c85c239dd32b10ab0073fb4166d35ba90c896512f29b455fd2a94c508d51

25 new tests, 3 skipped on Windows where creating a symlink needs privilege. Full suite 599 passed, 3 skipped. mypy --strict clean.

Follow-up, not in this PR

Once this ships on PyPI, the two integration copies and the examples-repo copy switch to importing it and their duplicates are deleted. Until then the cross-check test in integrations/wcm-triton keeps the two in step.

🤖 Generated with Claude Code

https://claude.ai/code/session_014NL8o3PXq6kfs2SdmBv6ak

SPEC.md takes weights_hash as given. It says nothing about how a directory of
shards, indexes and tokenizer assets collapses into one value, so every
consumer invented that step, and by 2026-08-27 the same construction existed in
three places outside this repository: the examples repository's open-model
walkthrough, and two Marketplace integrations.

The failure mode of a recipe existing several times is not that one of them is
wrong. It is that they drift, a manifest produced by one path stops verifying
on another, and the mismatch surfaces as weights_hash not matching, which reads
as tampered weights rather than as a tooling disagreement. There is now one
implementation to point at.

wcm.artifact_digest exports RECIPE_ID ("wcm-artifact-digest/v1"),
artifact_files, artifact_digest and ArtifactDigestError. artifact_digest
returns a HashValue, so the result drops into weights_hash without restringing.

Named, and a convention rather than specification. RECIPE_ID exists so a
mismatch can be attributed to the recipe rather than to the bytes, and so a
future construction becomes /v2 instead of silently replacing this one. A
deployment computing weights_hash another way is not non-conforming; it simply
must not expect this function to agree.

Two things this does that the copies did not:

Symlinks are refused by default. A link is a name resolving somewhere else, so
following one lets a digest cover bytes outside the artifact, and lets the
covered bytes change without anything in the artifact changing. There is a test
that demonstrates exactly that, by rewriting a link target and watching the
digest move while the tree does not. follow_symlinks=True accepts it
deliberately.

An inventory emptied by exclusion raises rather than returning a digest over
nothing, which a directory containing only .cache would otherwise produce.

Verified byte-identical to both existing integration copies over a tree with
nested directories, multiple shards and a .cache sibling, on the plain and the
include= paths. 25 new tests, 3 skipped on Windows where creating a symlink
needs privilege. Full suite 599 passed; mypy --strict clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014NL8o3PXq6kfs2SdmBv6ak
@imran-siddique
imran-siddique merged commit 5ab394a into main Aug 27, 2026
6 checks passed
@imran-siddique
imran-siddique deleted the feat/artifact-digest-recipe branch August 27, 2026 15:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant