feat(logs): make Elasticsearch/OpenSearch log retention configurable

[LevSky22]

Intent to contribute

I would like to contribute a small self-hosting/logs improvement: make the Nango Elasticsearch/OpenSearch log retention period configurable through an environment variable, while preserving the current default behavior.

Problem

Self-hosted Nango currently hard-codes structured log retention to 15 days in the Elasticsearch/OpenSearch lifecycle policy setup.

This is reasonable as a default, but it is not always a good fit for smaller self-hosted deployments that use Nango Proxy heavily. For high-volume API traffic, for example frequent Microsoft Graph / Outlook polling through the proxy, 15 days of operation/message logs can create avoidable Elasticsearch/OpenSearch disk and memory pressure.

Operators can manually patch ILM/ISM policies after startup, but Nango can recreate/update its log policies on startup, so this becomes a local operational workaround rather than durable configuration.

Current behavior

• Elasticsearch log retention is hard-coded in packages/logs/lib/es/schema.ts:
  • operations policy: delete.min_age = '15d'
  • messages policy: delete.min_age = '15d'
• OpenSearch ISM retention is hard-coded in packages/logs/lib/opensearch/ismPolicies.ts:
  • description says delete after 15d
  • transition condition uses min_index_age = '15d'
• NANGO_LOGS_ES_WARM_MIN_AGE is configurable, but the delete/retention age is not.

Proposed solution

Add a retention env var with a default of 15d, for example:

NANGO_LOGS_ES_RETENTION_PERIOD=15d

Then self-hosters could set:

NANGO_LOGS_ES_RETENTION_PERIOD=3d

Expected behavior:

• Default remains 15d for backward compatibility.
• Elasticsearch operations and messages delete phases use the configured value.
• OpenSearch ISM policies use the configured value for min_index_age.
• Existing NANGO_LOGS_ES_WARM_MIN_AGE behavior remains unchanged.

Implementation shape:

• Add the env var in packages/utils/lib/environment/parse.ts.
• Use it in packages/logs/lib/es/schema.ts for both policyOperations and policyMessages.
• Use it in packages/logs/lib/opensearch/ismPolicies.ts for both the policy description and conditions.min_index_age.

Local validation

I validated the operational need and behavior on a self-hosted Nango deployment:

• Enabled Nango structured logs with Elasticsearch.
• Confirmed Nango creates daily operation/message indices and ILM policies.
• Changed the active ILM delete phase from 15d to 3d.
• Confirmed both operations and messages policies reported delete.min_age = 3d.
• Confirmed Nango continued writing log documents successfully after the retention change.
• Created a disposable Elasticsearch ILM policy/index with delete.min_age = 1m, temporarily lowered indices.lifecycle.poll_interval from the default 10m to 10s, and confirmed the test index moved into the delete phase and was removed. The poll interval was restored to default afterward, and the temporary policy/index were cleaned up.

This validation used a local image patch/workaround because there is currently no first-class env var for this setting.

I also tested a candidate source change locally:

• npx vitest run packages/utils/lib/environment/parse.unit.test.ts packages/logs/lib/es/schema.unit.test.ts packages/logs/lib/opensearch/ismPolicies.unit.test.ts passed.
• npm run test:unit -- --run --dir=packages/logs passed.
• npm run test:unit -- --run --dir=packages/utils/lib/environment passed.
• npm run build -w @nangohq/logs passed.

The full monorepo npm run ts-build -- --pretty false did not complete cleanly in my local checkout due to unrelated workspace/package-resolution and existing request schema type errors outside the retention change.

Why this helps

This keeps Nango's current default intact while giving self-hosted operators a simple way to right-size log retention for their traffic volume and infrastructure budget.

[linear-code]

NAN-6089