cached K8S CSI driver

[airhorns]

This makes cached able to operate in a new mode -- as a Kubernetes Container Storage Interface driver, where it can be the thing that powers K8S volumes directly! CSIs are usually used to implement strange and foreign storage interfaces, like GCP persistent disks or NFS or whatever weird mounts you can figure out in linux.

In order for the hardlinks to be on the same device and thus actually improve performance, the cache daemon has to have privileged access to the the volumes that pods mount. We can do that with ninja editing stuff in /var/lib/kubernetes in a privileged container, or we can do it the safer way that k8s endorses! It actually still ends up modifying stuff in /var/lib/kubernetes and has the same escalated privileges, but, a big win is that the CSI daemon needs no network interface, and only actually communicates with other trusted components (the kubelet). In our case, we don't need to implement very much of the CSI interface (and we report as such with the fancy capabilities RPCs), we just want an emptyDir on the same device as wherever the cache is stored. Turns out, this is pretty easy!

How this will work:

• we'll deploy cached as a daemonset throughout the cluster, running in CSI mode talking to the kubelet
• pods that want to use the DL cache will have volume declarations that look like this (instead of emptyDir):

      volumes:
       - name: gadget
         csi:
           driver: com.gadget.dateilager.cached
           volumeAttributes:
             placeCacheAtPath: "cache" # matches
           readOnly: false

• when a new one of these volumes shows up, the kubelet will make GRPC calls to the CSI socket, which this PR makes cached implement
• cached won't actually mount anything, but instead just drop the cache in an empty folder and set the permissions up right (the kubelet actually mounts the folder into the docker container as it always does) using the existing implementation from the previous PR
• when the pod terminates, cached gets another RPC, and removes the whole folder (but not the shared golden copy of the cache).

Woop woop!

[airhorns]

• Improve HardlinkDir performance #91
• cached K8S CSI driver #89 👈
• cached cache population daemon #85
• Small changes to prep for caching daemon  #90
• Build dateilager within the dockerfile using the nix environment #86
• Add a prerelease github workflow for building and publishing unfinished versions #87
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @airhorns and the rest of your teammates on Graphite

[angelini]

Why do these methods on Cached exist in another file?

[airhorns]

I figured it was nice to separate them since its the CSI stuff vs our stuff, but if that's against convention I will merge them?

[angelini]

I don't think this will cleanly stop on graceful shutdown. You likely want a cancellable context passed in here that we can abort on <-osSignals

[airhorns]

Would you mind terribly shooting me a diff? I feel like I could figure it out but it would take you approximately 10 seconds to do

[angelini]

If the CSI server caused this to fail, you need to also stop the GRPC server.

[angelini]

FYI, I'm going to push commits to this branch