[WIP] document how to add new sources

[deitch]

based on the extensive slack discussions with @sipsma and @tonistiigi

I have been trying to create an OCI layout source so it can be used as a build-context. Also potentially others. The maintainers have indicated they would accept it, but I struggled to understand the required behaviour and interface.

Once @sipsma is explaining it, let's capture it for future users.

When this actually captures everything in a useful way, we can remove the draft status and make it a real PR.

[deitch]

There is a lot I do not understand here, and so could not capture. I will comment a lot inline. But fundamentally, what is it that the solver really expects of the Source and SourceInstance? At the highest level, I sort of expected two things:

• CacheKey: solver: "here is a reference, give me a cache key"; source: "ok, here is the unique identifier for it"
• Snapshot: solver: "here is a reference, give me the contents"; source: "ok, here is a tar stream of the contents"

Clearly, it does not do that. It does more, but also differently. What, why, how? Who actually handles the caching? What are the responsibilities of the Source and SourceInstance?

[deitch]

Is this correct? Is it an idempotent "id->key" function?

[deitch]

Who creates these? Who consumes them?

[deitch]

This part really threw me for a loop. The solver doesn't know more than, "here is an ID, give me a cache key". How can it know of multiple index integers? When does it change them? What is its meaning?

[deitch]

What is this unique across? For example, for container image in a registry, is this just going to be the hash of the root manifest, e.g. right now, docker.io/library/nginx:1.21.6-alpine resolves to sha256:5a0df7fb7c8c03e4158ae9974bfbd6a15da2bdfdeded4fb694367ec812325d31, so would the cache key just be sha256:5a0df7fb7c8c03e4158ae9974bfbd6a15da2bdfdeded4fb694367ec812325d31?

[deitch]

Also, how does this tie into the discussion above about index and cacheDone?

[deitch]

Given that the previous is the key, what is this?

[deitch]

What are examples of some opts here? And who consumes them, given that the SourceInstance only returns them, so who consumes them and how?

[deitch]

What does this actually mean? What is an example?

[deitch]

I have no idea what local does, actually. I read the code, but I am quite unsure what it does.

And when docker-image "pulls" the image, does it store it in some local cache? Or is it just passing the data to the solver? How does it handle various layers? It looks (from the code) like it merges them together to create a single filesystem view? And how does that all get garbage-collected?

[deitch]

Same as this question

[deitch]

This isn't a io.Reader or TarReader, it is an ImmutableRef. What does that mean? How does it use it to get the actual content in a useful way?

[deitch]

OK, so it isn't the SourceInstance passing a stream of content that the solver will cache in its own cache, mount, etc.

Instead, it is that the SourceInstance is expected to cache it locally, returning instead a struct that implements ImmutableRef, i.e. something that the solver can use to mount the content in the filesystem.

So then what does the whole ImmutableRef interface do? The interface is:

type ImmutableRef interface {
	Ref
	Clone() ImmutableRef
	// Finalize commits the snapshot to the driver if it's not already.
	// This means the snapshot can no longer be mounted as mutable.
	Finalize(context.Context) error

	Extract(ctx context.Context, s session.Group) error // +progress
	GetRemotes(ctx context.Context, createIfNeeded bool, cfg config.RefConfig, all bool, s session.Group) ([]*solver.Remote, error)
	LayerChain() RefList
}

What is the SourceInstance expected to provide here? There's a lot more than "give me a mount point and I will mount it."

[AkihiroSuda]

I’m feeling this should be placed in a directory like /docs/internal/, as this isn’t expected to be read by end users

[deitch]

I'm fine with moving this there. Let's get all of the missing content in first.

[deitch]

The only place I found using cacheDone was docker-image source. But I could not figure out when and how it uses it. The entire cacheDone and index seem to boil down to these lines:

	cacheDone = index > 0
	if index == 0 || p.configKey == "" {
		return p.manifestKey, p.manifest.MainManifestDesc.Digest.String(), cacheOpts, cacheDone, nil
	}
	return p.configKey, p.manifest.MainManifestDesc.Digest.String(), cacheOpts, cacheDone, nil

Which appears to mean: "the first time I am called, index=0, so return cacheDone=false. That means I will get called again with index=1, at which time return cacheDone=true."

There doesn't appear to be any other logic that differs from call 0 and call 1, so what purpose does it serve?