Support configurable lease duration (e.g., via env var or flag)

[damdinsharav]

Currently, the lease duration used in lease.NewKubernetes is hardcoded through an empty lease.Options{} struct in pkg/export/setup/setup.go, which defaults to 15 seconds.

It would be helpful to allow configuring LeaseDuration via environment variable, CLI flag, or a field in HAOptions. This would make it easier to adapt the operator to different environments without needing to modify the code directly.

Thanks for the great work on this project!

[bernot-dev]

Can you share any more about how configuring the lease duration is useful to you?

[damdinsharav]

Thanks for the quick response!

In our setup, we aim to minimize downtime during disruptions, such as node failures or pod evictions. The default lease duration of 15 seconds introduces a delay in failover, during which no instance holds leadership. To improve responsiveness, we’d like to reduce the lease duration to around 3 seconds.

This would allow a standby instance to take over more quickly, ensuring more consistent availability in HA scenarios.

[bernot-dev]

I am happy to review a PR for this change. Note that in future versions, the export package will live in GoogleCloudPlatform/prometheus instead of prometheus-engine.

See: https://github.com/GoogleCloudPlatform/prometheus/blob/release-2.53.5-gmp/google/export/setup/setup.go#L236

[bwplotka]

👍🏽

Yup, the change should be against https://github.com/GoogleCloudPlatform/prometheus/blob/release-2.53.5-gmp/google/export/setup/setup.go#L236 from now on.

On separate note, just a heads up that we are slowly moving to the world where you will be able to use vanilla Prometheus OSS binary. In this case, Prometheus don't have HA write support. We can either help everyone by proposing/contributing HA write feature to OSS (strong justification will need to be needed, given another complexity), OR provide tutorials/config examples for using https://github.com/jlevesy/prometheus-elector (which has some adoption already prometheus/prometheus#11665 (comment)). It would be epic to work together on some solution that works for you!

[damdinsharav]

Thank you so much for the heads up. I have made a PR here.

[bwplotka]

Fixed on 2.53.5 fork - release in progress.

Thanks!