Some tracker and activator related questions

[oOraph]

Hello :). I'm currently playing with this really cool project -> nice design !

Several (mostly bpf related) questions:

1. Why is the vmlinux.h.gz btf file provided to generate bpf programs ? This is less robust than generating and extracting it from the target kernel at (cross)compile time no ? Do you have any compatibility inventory between the types in this file and the kernel versions the bpf programs compiled from it will run onto ?

2. Can you confirm I'm understanding the following correctly ?

regarding the bpf programs there are two different programs:

• The tracker one: kretprobe from the pid tracker that will intercept tcp accept events and check whether some tcp activity datetime should be updated in the monitored pid map (to know when to scale down or not)
• The proxy/activator one: tc ingress/egress filters setup when the activator takes the relay for checkpointed/suspended pids to fwd requests toward the proxy/activator in charge of bufferizing requests while scaling back up and fwding to the backend once up -> why prefer tc filters over an xdp implem ?

5. Regarding this issue:
   use Pod IP to track connections instead of container PID #85
   when you say: "if the socket is created by a nested process of 2 layers deep it will not be tracked." -> do you mean because here you only look for a 'pid' either equal to the tgid or the ppid in the map ?
   

   zeropod/socket/kprobe.c

   Line 75 in e75ed9a

   // we use the tgid as our pid as it represents the pid from userspace

   
   tracking with the pod ip instead of the pid would have drawbacks as well no ? (like not tracking accepts called from within the container on the lo interface but I agree this is an edge case we may ignore)

6. Can you detail the current limitations regarding the maximum number of pods that can be tracked/checkpointed on a single node ?
   What would happen if there were more than this ?
   For example when I look here:
   

   zeropod/socket/kprobe.c

   Line 16 in e75ed9a

   __uint(max_entries, 1024); // should be enough pids?

   
   if I understand corrrectly we cannot track more than 1024 pids (eg 1024 init processes/containers/pods) on a given node right ?
   But the consequence of this is not that fatal right ? looking here:
   

   zeropod/socket/ebpf.go

   Line 181 in e75ed9a

   func (c *EBPFTracker) TrackPid(pid uint32) error {

   
   https://github.com/ctrox/zeropod/blob/main/shim/container.go#L403
   it just means the next pods/container init pids will never be a candidate for scale down right ?

7. There is sth that I'm not understanding with the noop (aka fallback) tracker:
   here:
   

   zeropod/socket/noop.go

   Line 28 in e75ed9a

   func (n NoopTracker) LastActivity(pid uint32) (time.Time, error) {

   
   Setting the last activity to the idle scale down duration will systematically trigger a scaledown after the annotated duration no ?
   don't we prefer never to scale down if the bpf tracker cannot be setup ?

thanks :)

[oOraph]

actually I found the answer to the first question in the makefile:

# to improve reproducibility of the bpf builds, we dump the vmlinux.h and
# store it compressed in git instead of dumping it during the build.
update-vmlinux:

[oOraph]

Another question:
why do you need to embed criu within the manager image here

zeropod/cmd/manager/Dockerfile

Line 21 in e75ed9a

COPY --from=criu /bin /bin

since you already install it on node with the installer ?
(here:

zeropod/cmd/installer/main.go

Line 142 in e75ed9a

if err := installCriu(ctx); err != nil {

from the patched criu docker image specified in cmdline)

[ctrox]

Yeah the vmlinux.h is from a x86 machine as of now but it would probably be wise to have one for each architecture, although it should work fine unless we start to make use of some enums that only exist for one specific architecture. The e2e tests cover both x86 and arm64 so it should show up if something does not work as expected.

why do you need to embed criu within the manager image here

That is required as the manger runs in a container and I did not want to add a hostmount to execute binaries, that seems like a bad idea. So it was simpler to just also add it to the manager image.

Can you confirm I'm understanding the following correctly ?
The tracker one: kretprobe from the pid tracker that will intercept tcp accept events and check whether some tcp activity datetime should be updated in the monitored pid map (to know when to scale down or not)

Correct.

The proxy/activator one: tc ingress/egress filters setup when the activator takes the relay for checkpointed/suspended pids to fwd requests toward the proxy/activator in charge of bufferizing requests while scaling back up and fwding to the backend once up -> why prefer tc filters over an xdp implem ?

Also correct, I used tc because xdp did not support egress at the time and I think that's still the case today.

Regarding this issue:
do you mean because here you only look for a 'pid' either equal to the tgid or the ppid in the map ?

Yes exactly.

tracking with the pod ip instead of the pid would have drawbacks as well no ? (like not tracking accepts called from within the container on the lo interface but I agree this is an edge case we may ignore)

Yep, that's also true but we could use a combination of pod IP and listening port to distinguish multiple containers within a pod. But yeah localhost might be an issue, for example for port-forward traffic. I have experimented a bit on this but have put it aside for the time being.

Can you detail the current limitations regarding the maximum number of pods that can be tracked/checkpointed on a single node ?
it just means the next pods/container init pids will never be a candidate for scale down right ?

Yeah, but the other way around. It will just scale down after the duration is up since it won't have tracked any connections to that pid. The map is BPF_MAP_TYPE_LRU_HASH so it will evict the least recently used entry. As it removes the pid on scale down, the node would need to be running >1024 pods at the same time so I think this is not that likely to be a problem.

For the rest of the BPF maps (for the activator) it's also unlikely to be an issue since there is one map per pod. So it can redirect to 128 different ports per pod, which should probably be enough for most use-cases 😄

Setting the last activity to the idle scale down duration will systematically trigger a scaledown after the annotated duration no ?
don't we prefer never to scale down if the bpf tracker cannot be setup ?

The noop tracker can't track connections so it will just scale down after that fixed duration. At least that was my thinking about this :)

If the activator also fails to be setup (which is very likely to be the case if the tracker setup fails due to missing/outdated eBPF support), it will disable scaledown completely since it can't restore at all without the activator.

[oOraph]

thanks for the details :) ! (closing discussion)