Add a GRPC HealthCheck endpoint

[jan-g]

This can be used by kubelet to monitor the pod for readiness, and as a backup by the operator to catch pipelines that've run to completion.

[jan-g]

Quick draft of the idea, going to plumb this together with an operator that can keep tabs on it.

The point here is that the readiness probe should really survive after the one-shot pipeline is complete, so that we get a positive signal about it.

[mihaitodor]

Hey @jan-g, thank you for raising this! Just a quick reaction for now:

The point here is that the readiness probe should really survive after the one-shot pipeline is complete, so that we get a positive signal about it.

Would you not be able to leverage shutdown_delay for this purpose?

[jan-g]

| Would you not be able to leverage shutdown_delay for this purpose?

I'm not 100% convinced that I can, at least not as reliably. If I've a bunch of pods running then I don't necessarily have an upper bound for how long it'll take to poll these. By wiring this into the grpc readiness signal I distribute my monitoring over a bunch of kubelets, and use the k8s cp as an event collector. I've marked the PR as a draft because I'd like to get all the moving parts lined up to see it working end-to-end.

[jan-g]

Right, I've run this up with the contianer configured with

readinessProbe:
  grpc:
    port: 2999

and I'm seeing completed pipelines move to unready (and remain there). I have a second PR knocking around to spot those and sweep them up, rather than having them restart continually.

[jan-g]

Question here is mostly styllistic: do we want to drive something like this (including port selection) via convi/env var or something?

Also, "can we achieve the same result via the existing machinery?" but it looks like that's quite geared around cleaning up within a fixed period of pipeline completion. LMKWYT.

[Jeffail]

Thanks @jan-g, LGTM overall, there's an error breaking tests:

cmd/redpanda-connect-cloud/main.go:27:2: "github.com/redpanda-data/benthos/v4/public/service" imported and not used

[Jeffail]

LGTM, thanks @jan-g! I made one small alteration so that this only applies when running the run command, that allows us to continue using create and list and so on without issues.