Add IPFS as a remote cache #3510
Conversation
|
Can we just exec the Kubo binary to reduce the Go dependencies? |
Could we check what the difference in binary and buildkit image size would be for both cases. Also, maybe there is a smaller tool or a minimal build that we could include in the image. |
Not really for binary footprint, rather for avoiding the vendor hell. |
Yes, but if including kubo increases binary(image) size too much compared to vendoring then we might still prefer vendoring. We need to know the numbers(and if there are alternatives). |
Another alternative would be to reimplement the IPFS API client with the stdlib net/http |
|
|
||
| if !exists { | ||
| layerDone := progress.OneOff(ctx, fmt.Sprintf("writing layer %s", l.Blob)) | ||
| dt, err := content.ReadBlob(ctx, dgstPair.Provider, dgstPair.Descriptor) |
There was a problem hiding this comment.
Can we use io.Copy here instead of fully reading the layer to slice?
| go func() { | ||
| for { | ||
| j, more := <-out | ||
| if more { | ||
| logrus.Debugf("added item: %+v", j) | ||
| cid = j.Cid | ||
| } else { | ||
| logrus.Debugf("added all items") | ||
| done <- true | ||
| return | ||
| } | ||
| } | ||
| }() |
There was a problem hiding this comment.
This should be cancellable via ctx?
| } | ||
|
|
||
| logrus.Debugf("unpinning previous pin: %s\n", prevPinCID.Cid) | ||
| _, err = clusterClient.Unpin(ctx, *prevPinCID) |
There was a problem hiding this comment.
Is it enough to only unpinning 1 CID? Isn't it possible that previously multiple CIDs being associated to that pin name?
|
Find below the size comparison: TL;DR
See below the binaries used for the comparison: DetailsBinary size without the IPFS implementation.Branch: Original: make binaries
...
ls -lh ./bin/
...
-rwxr-xr-x@ 1 felipecruz staff 50M 18 Jan 16:14 buildkitdBinary size using IPFS Go libraries (vendoring)Branch: make binaries
...
ls -lh ./bin/
...
-rwxr-xr-x@ 1 felipecruz staff 66M 18 Jan 16:01 buildkitd |
Maybe these binaries should be only present in a separate image like That might be also helpful for some enterprise companies that have "no P2P" policies. |
If we do that then it probably makes sense to move the s3/azure backends also to that image (cc @bpaquet) . And include things like Nydus if they are ready that is somewhat supported today but not in release image(cc @hsiangkao ). |
Hi! Actually I'm not responsible for main Nydus implementation (One part of my main jobs is in-kernel EROFS), I'm Ccing proper Nydus people here. cc @imeoer @jiangliu @changweige |
|
@tonistiigi @AkihiroSuda If image size is a concern, nydus is also ready to build related binary into a separate image like |
Hi,
I'd like to open this PR to introduce IPFS as an additional remote cache for BuildKit. It contains a first implementation to export and import the build cache to IPFS and replicate it among all the peers.
Motivation
The main motivation is to explore new ways to achieve faster build times by leveraging the use of IPSF as a remote, distributed cache of image blobs among multiple peers of a cluster.
Use-case
In a company, software developers that work on the same project happen to build an application that is likely to have been already built by a teammate/peer. Exporting the BuildKit cache to a remote registry is a convenient solution, however, sometimes it can be time-consuming to download the cache, especially when building large images.
By distributing the cache using IPFS across many cluster peers, the blocks that compose the blobs can be downloaded in parallel from multiple peers instead of from one single place (a remote registry).
Try me
Set up an IPFS cluster if you don't have one. See this example.
Create a builder from
felipecruz/buildkit:ipfs-clusterdocker buildx build \ --cache-to type=ipfs,cluster_api=192.168.65.2:9094,daemon_api=192.168.65.2:5001,mode=max \ -t my-image .docker buildx build \ --cache-from type=ipfs,cluster_api=192.168.65.2:9094,daemon_api=192.168.65.2:5001,mode=max \ -t my-image ./cc @tonistiigi @crazy-max @AkihiroSuda