Worker pod liveness probe unstable behavior

[ilya-vasiuk]

Describe the bug
The liveness probe in the lago-events-worker container fails when using the command with direct command substitution (grep $(hostname)). The probe times out even though the process is running correctly. This is due to the way the shell handles the command substitution in the current configuration.

To Reproduce
Steps to reproduce the behavior:

• Deploy the Lago application in a Kubernetes cluster.
• Observe that the liveness probe fails with a timeout, causing the container to restart even though the process is healthy.
• Verify by manually executing the command via kubectl exec to see that the command fails to find the hostname when using direct substitution.

$ kubectl exec -it lago-events-worker-xxxxxxx-yyyyy -n lago -- /bin/bash -l -c "bundle exec sidekiqmon processes | grep $(hostname) || exit 1"
Defaulted container "lago-events-worker" out of: lago-events-worker, wait-for-redis (init), wait-for-migrations (init)
command terminated with exit code 1

Expected behavior
The liveness probe should correctly verify that the process corresponding to the current hostname is running. When executed properly, the probe should succeed without timing out.

Support
Version: 1.17.4

Additional context
The issue appears to be related to the way command substitution is handled when passed directly as part of the liveness probe command. A working fix is to wrap the command in single quotes and assign the output of hostname to a variable within the shell:

livenessProbe:
  exec:
    command:
      - /bin/bash
      - -l
      - -c
      - 'host=$(hostname); bundle exec sidekiqmon processes | grep -q "$host" || exit 1'

This change prevents the premature interpolation of $(hostname) and allows the command to execute correctly within the container.

[jdenquin]

👋 @ilya-vasiuk, we added an other healthcheck for workers, its not documented right now.
You can do an http healthcheck on http://localhost instead of the sidekiqmon method

[ilya-vasiuk]

Thanks, @jdenquin, I'll try this approach