Repository navigation
Conversation
…e DownloadInfo failure actionable
…check SnapshotDir now requires a concrete revision. Download was still handing it opts.Model unresolved for unpinned repositories, which made existingFile silently treat every file as missing on a second run.
…ME polish - infoshim.go: remove the unreachable ErrServerClosed filter in Close and the dead conditional body in the Serve goroutine (both provably unreachable per net/http's Server.Close semantics); drop the now-unused "errors" import. - pin.go: use errors.New instead of fmt.Errorf with no format verbs, to match the identical guard in download.go and cache.go. - .gitignore: also ignore cmd/codamigo/codamigo (a `go build ./cmd/codamigo/` run from inside that directory leaves the binary there, outside the root-anchored /codamigo pattern that already covers `make build`'s output). Anchored deliberately so it cannot also match the cmd/codamigo source directory. - README.md: minor doc updates for the pinned-revision behavior.
…modules.json declares
standardManifest only ever fetched the plain Transformer+Pooling pair
(7 files). A raw repository id such as google/embeddinggemma-300m adds
a Dense projection module ("2_Dense") and a Normalize step, each with
its own files, that standardManifest has no idea exist.
Download would therefore report success while transformer.LoadModel
still needed a file it never fetched, and MissingFiles would report
the model ready when it was not — the failure only ever surfaced late,
as a loopback-shim 404 during New, i.e. "model not downloaded" for a
model whose declared manifest was in fact fully present on disk.
expandManifest (localembed/model.go) closes the gap for an unpinned
repository id (registry models keep their hand-verified manifest
untouched):
- Download fetches modules.json ahead of the main loop, parses it, and
folds in any extra module's files using the sizes/hashes already
present in the repository info, before the loop decides what counts
as "downloaded".
- ResolvePin does the load-time equivalent with no network access,
reading modules.json off the resolved snapshot directory, so
MissingFiles catches a genuinely incomplete download before New ever
starts talking to go-huggingface.
Also updates the design doc's "Known limitations" entry, which called
this out as deliberately deferred.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a pin file (
codamigo-pin.json) recording the resolved upstreamcommit for a downloaded local embedding model, and a loopback info shim
so a pinned model loads without any network access — go-huggingface has
no offline mode and otherwise makes one mandatory revision-lookup call per
load.
download-modelresolves and pins the exact commit, verifying everyfile against a per-file checksum for built-in models.
pin file via a
127.0.0.1shim, alive only for the span of the load.never redirect it, and a mismatch is reported rather than silently
resolved.
the revision from go-huggingface's own cached info, no re-download.
doctoranddownload-modelreport the resolved revision, realembedding dimensions, and superseded snapshot sizes (never deleted
automatically).
google/embeddinggemma-300m) nowdiscovers its own extra module files instead of silently missing them.
standardManifestonly ever covered the plain Transformer+Pooling pair,so a model that adds a Dense projection or Normalize module downloaded
"successfully" while still missing a file
transformer.LoadModelneeded— surfacing later as a confusing shim 404 ("model not downloaded" for a
model whose declared manifest was fully present).
expandManifestnowreads the model's own
modules.json— fetched ahead of the main loop inDownload, read from the resolved snapshot with no network access inResolvePin— and folds in whatever extra files it declares, using thesizes/hashes the repository info already reports. Registry models are
untouched; only a raw repository id, which has no hand-verified manifest
to begin with, gets this treatment.
infoshim.go(an unreachableErrServerClosedfilter and a conditional whose body was
_ = err) and an idiom fix inpin.go(errors.Newinstead offmt.Errorfwith no verbs)..gitignore: also ignorecmd/codamigo/codamigo, the binary left by abare
go build ./cmd/codamigo/run from inside that directory.Testing
make fmt,make vet,golangci-lint run ./...(0 issues),gosec,nilawayall clean (pre-existingnilawayfinding ininfoshim_test.gois unrelated and present on the base branch too).make testandmake test-racepass across all packages.make buildsucceeds.TestExpandManifest_*(unit),TestDownload_DiscoversExtraModuleFiles(download-time, fake HF server),
TestResolvePin_DiscoversDenseModuleFromDisk(load-time, offline).