From a50371f9b273a001062ba32386e3ffaf68048efe Mon Sep 17 00:00:00 2001 From: Julien Pivotto <291750+roidelapluie@users.noreply.github.com> Date: Wed, 1 Jul 2026 16:20:35 +0200 Subject: [PATCH 1/5] Add query cost estimation and limits proposal Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com> --- proposals/00089-query-cost.md | 133 ++++++++++++++++++++++++++++++++++ 1 file changed, 133 insertions(+) create mode 100644 proposals/00089-query-cost.md diff --git a/proposals/00089-query-cost.md b/proposals/00089-query-cost.md new file mode 100644 index 0000000..d79b751 --- /dev/null +++ b/proposals/00089-query-cost.md @@ -0,0 +1,133 @@ +## Query cost estimation and limits + +* **Owners:** + * Julien Pivotto [@roidelapluie](https://github.com/roidelapluie) + +* **Implementation Status:** Not Implemented. + +* **Related Issues and PRs:** + * `` + +* **Other docs or links:** + +> TL;DR: A single expensive query can hurt a whole Prometheus. We have knobs to cap it (`--query.max-samples`, `--query.timeout`), but no way to tell a user *before* they run a query how expensive it is, and no per-query, reloadable ceilings. This proposal adds a cheap cost *estimate* (series touched, samples scanned) exposed through `/api/v1/query_cost`, reloadable cost *limits* enforced during execution, and an estimated-vs-actual `cost` object on the query response. All behind a `query-cost` feature flag. + +## Why + +Prometheus already protects itself from runaway queries, but the tools are blunt: + +* `--query.max-samples` caps peak samples in memory, not the total scanned. +* `--query.timeout` and `--query.max-concurrency` are process-wide flags, not reloadable and not per-query. +* Nothing tells a user, an autocomplete UI, or an alerting rule author how heavy a query is *before* it runs. + +Operators want ceilings they can tune without a restart. Users and tools (Grafana, dashboards, recording rules) want a cheap way to gauge cost up front so they can refuse or rewrite a query before it lands on the server. + +### Pitfalls of the current solution + +* The existing limits are set at startup. Changing them means a restart. +* They are global. A single tenant or dashboard cannot be given a tighter budget. +* There is no pre-execution estimate. The only way to learn a query's cost today is to run it, which is exactly what we want to avoid for the expensive ones. +* `--query.max-samples` measures peak in-memory samples, which does not map cleanly to "how much index and how many samples did this touch". + +## Goals + +* Give a cheap, index-based cost *estimate* (series touched, samples scanned) without executing the query. +* Expose the estimate through a new API so clients can gauge cost before running a query. +* Add reloadable cost limits (`query_max_series`, `query_max_samples_scanned`, `query_max_duration`) enforced during execution. +* Let a client *lower* those ceilings per query, never raise them. +* Surface estimated-vs-actual cost on the normal query response, so the estimate can be validated against reality. +* Keep it all opt-in behind a feature flag until the model is proven. + +### Audience + +Operators running shared Prometheus servers, and UI/tooling authors (Grafana, recording rules) that build queries on a user's behalf. + +## Non-Goals + +* Not replacing `--query.max-samples`, `--query.timeout`, or `--query.max-concurrency`. +* Not a billing or chargeback system. The numbers are upper bounds, not exact accounting. +* Not a slow-query log. +* Not per-tenant configuration, as Prometheus is not multi-tenant. Limits are global, with per-query lowering only. +* Not exact cost prediction. The estimate is intentionally cheap and approximate. + +## How + +Three pieces, all gated by `--enable-feature=query-cost`. + +**1. Estimation (`promql.EstimateCost`).** Parse the query, walk it for every vector and matrix selector, compute the effective time window each selector reads (mirroring the engine's `getTimeRangesForSelector`/`populateSeries`), and ask storage for the series count per selector via a single querier over the union window. `SeriesTouched` is the sum across selectors — an upper bound, because a series shared between selectors is counted once per selector. `SamplesScanned` models the engine's incremental per-step reads (full range window at step 0, then only the samples that advance past the previous cutoff), scaled by a measured average per-point cost so native-histogram points are sized by bucket rather than counted as one float unit. The estimate is index-only apart from decoding at most `histogramSampleLimit` (50) points per selector to size histograms. + +**2. API.** Two new endpoints estimate cost without executing: + +``` +GET|POST /api/v1/query_cost +GET|POST /api/v1/query_range_cost +``` + +They take the same parameters as `/api/v1/query` and `/api/v1/query_range` and return: + +```json +{ + "estimate": { + "seriesTouched": 42, + "samplesScanned": 5040 + } +} +``` + +The instant and range endpoints also gain a `cost` parameter. When set, the response `data` carries an estimated-vs-actual comparison: + +```json +"cost": { + "estimated": { "seriesTouched": 42, "samplesScanned": 5040 }, + "actual": { "seriesTouched": 40, "samplesScanned": 4980, "peakSamples": 320 } +} +``` + +Note: `cost=1` adds a second index lookup on top of executing the query. + +**3. Limits.** Three reloadable knobs under `global:`: + +```yaml +global: + query_max_series: 0 # 0 = no limit + query_max_samples_scanned: 0 + query_max_duration: 0s +``` + +These are enforced *during* execution against the query's actual running cost, not against the estimate: a query is rejected as soon as it loads too many series or scans too many samples, and `query_max_duration` surfaces as a query timeout. A client may lower any ceiling for a single request via `max_series`, `max_samples_scanned`, `max_query_duration`; these can only tighten, never loosen, the operator-set value. The estimate is never used to reject a query — enforcement is always on the real cost. + +### Testing and verification + +* Unit tests for limit enforcement (reject paths) in `promql`. +* Estimation-accuracy tests against known fixtures, plus the `cost` object which lets us compare estimated and actual on every executed query. +* API tests for the new endpoints and the `cost` parameter. +* OpenAPI golden files updated for the new paths and schemas. + +### Migration + +Purely additive and behind a feature flag. Default config (all limits `0`) changes no behaviour. Nothing to migrate. + +### Known unknowns + +* **Estimate accuracy.** `SeriesTouched` over-counts shared series and series with no in-window samples; `SamplesScanned` assumes samples land exactly at the scrape interval and that sampled series are representative. Is an upper bound the right contract, or do we want something tighter? +* **Scrape interval.** The estimator uses the global scrape interval; per-target intervals are not modelled. +* **Subqueries.** Only one level of nesting is modelled exactly. +* **Lookback delta.** The storage-only estimator uses the package default, not the engine's configured value. +* **Agent mode.** Estimation is unavailable (no queryable index). +* **Config surface.** Should limits live under `global:`, or a dedicated `query:` section? + +## Alternatives + +1. **Estimate from postings cardinality directly, bypassing `storage.Querier`.** Cheaper, but ties the estimator to the TSDB index and breaks for any other `storage.Queryable` (remote read, federation). Using the portable `Select` path keeps it storage-agnostic. +2. **Reject queries based on the estimate.** Rejected: the estimate is an upper bound and can be wrong in both directions. Rejecting on an estimate would refuse queries that would actually run fine. Enforcement is on real cost; the estimate is advisory only. +3. **Reuse `--query.max-samples` and friends.** They are start-time flags measuring peak in-memory samples, not reloadable and not per-query. Extending them to be reloadable and per-query would overload their meaning; new, clearly-scoped knobs are cleaner. +4. **Do nothing / client-side estimation.** Clients cannot cheaply see the server's index cardinality, so any client-side guess is worse than a server estimate. + +## Action Plan + +* [ ] `promql.EstimateCost` and the sample-unit cost model +* [ ] `/api/v1/query_cost` and `/api/v1/query_range_cost` endpoints +* [ ] `cost` parameter on instant/range queries (estimated vs actual) +* [ ] Reloadable `query_max_series` / `query_max_samples_scanned` / `query_max_duration` under `global:` +* [ ] Per-query lowering via `max_series` / `max_samples_scanned` / `max_query_duration` +* [ ] `query-cost` feature flag, docs, OpenAPI spec, UI surfacing From 6274dc8690da1a25a3515138a171f9fd22da8af2 Mon Sep 17 00:00:00 2001 From: Julien Pivotto <291750+roidelapluie@users.noreply.github.com> Date: Fri, 14 Aug 2026 12:14:33 +0200 Subject: [PATCH 2/5] Query Cost: Update based on further work Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com> --- proposals/00089-query-cost.md | 44 +++++++++++++++++++++-------------- 1 file changed, 26 insertions(+), 18 deletions(-) diff --git a/proposals/00089-query-cost.md b/proposals/00089-query-cost.md index d79b751..4f59193 100644 --- a/proposals/00089-query-cost.md +++ b/proposals/00089-query-cost.md @@ -25,7 +25,7 @@ Operators want ceilings they can tune without a restart. Users and tools (Grafan ### Pitfalls of the current solution * The existing limits are set at startup. Changing them means a restart. -* They are global. A single tenant or dashboard cannot be given a tighter budget. +* They are global. A single dashboard, query, cannot be given a tighter budget. * There is no pre-execution estimate. The only way to learn a query's cost today is to run it, which is exactly what we want to avoid for the expensive ones. * `--query.max-samples` measures peak in-memory samples, which does not map cleanly to "how much index and how many samples did this touch". @@ -40,7 +40,7 @@ Operators want ceilings they can tune without a restart. Users and tools (Grafan ### Audience -Operators running shared Prometheus servers, and UI/tooling authors (Grafana, recording rules) that build queries on a user's behalf. +Operators running shared Prometheus servers, and UI/tooling authors (Grafana) that build queries on a user's behalf. ## Non-Goals @@ -54,7 +54,16 @@ Operators running shared Prometheus servers, and UI/tooling authors (Grafana, re Three pieces, all gated by `--enable-feature=query-cost`. -**1. Estimation (`promql.EstimateCost`).** Parse the query, walk it for every vector and matrix selector, compute the effective time window each selector reads (mirroring the engine's `getTimeRangesForSelector`/`populateSeries`), and ask storage for the series count per selector via a single querier over the union window. `SeriesTouched` is the sum across selectors — an upper bound, because a series shared between selectors is counted once per selector. `SamplesScanned` models the engine's incremental per-step reads (full range window at step 0, then only the samples that advance past the previous cutoff), scaled by a measured average per-point cost so native-histogram points are sized by bucket rather than counted as one float unit. The estimate is index-only apart from decoding at most `histogramSampleLimit` (50) points per selector to size histograms. +**1. Estimation (`promql.EstimateCost`).** Parse the query, walk it for every vector and matrix selector, compute the effective time window each selector reads (mirroring the engine's `getTimeRangesForSelector`/`populateSeries`), and ask storage for the series count per selector via a single querier over the union window. `SeriesTouched` is the sum across selectors — an upper bound, because a series shared between selectors is counted once per selector. `SamplesScanned` models the engine's incremental per-step reads (full range window at step 0, then only the samples that advance past the previous cutoff), scaled by a measured average per-point cost so native-histogram points are sized by bucket rather than counted as one float unit. The estimate is index-only apart from decoding at most 50 (like `histogramSampleLimit`) points per selector. + +`SamplesScanned` is intended to approximate the *samples read* statistic introduced in [prometheus/prometheus#18081](https://github.com/prometheus/prometheus/pull/18081) — the total samples the engine reads from storage — not the older *total samples* (peak in-memory) statistic. Series with no in-window samples still count, so the figure stays an upper bound. + +The per-point density can be derived two ways: + +* **Scrape-interval (fallback, index-only).** Assume samples land at the global scrape interval and compute window ÷ interval. Cheapest, but wrong for series scraped at a different interval and for remote-written series, which have no scrape interval at all. Used only when nothing can be sampled (see below). +* **Chunk sampling (always-on, not opt-in).** The estimator samples automatically, with no user-facing knob, whenever the storage exposes `storage.ChunkQueryable`: it reads up to a fixed `chunkSampleLimit` (50) chunks' `NumSamples` header to measure the selector's real sample interval, and decodes the first point of up to a fixed `histogramSampleLimit` (50) series to size native-histogram points by bucket count. Reading a chunk header is far cheaper than decoding its samples, so this stays much cheaper than executing the query. + +Sampling prefers the *real* query window over a nearby proxy window whenever that real window is cheap enough: if a selector's actual chunk count (for density) or series count (for point cost) already fits within the 50-item budget, the estimator samples directly from `[sel.mint, sel.maxt]` and gets an exact rather than extrapolated measurement. Only when the real window has more chunks/series than the budget affords does it fall back to sampling a bounded, narrow window near the query's end and extrapolating. **2. API.** Two new endpoints estimate cost without executing: @@ -74,7 +83,7 @@ They take the same parameters as `/api/v1/query` and `/api/v1/query_range` and r } ``` -The instant and range endpoints also gain a `cost` parameter. When set, the response `data` carries an estimated-vs-actual comparison: +The instant and range endpoints also gain a `cost=true` boolean parameter. When set, the response `data` carries an estimated-vs-actual comparison: ```json "cost": { @@ -83,7 +92,7 @@ The instant and range endpoints also gain a `cost` parameter. When set, the resp } ``` -Note: `cost=1` adds a second index lookup on top of executing the query. +`cost` is a plain on/off switch, not a set of levels: `cost=true` (any non-bool value errors) enables the comparison. There is no separate opt-in for chunk sampling, because chunk-metadata sampling always runs automatically wherever the storage supports it (see How, section 1). **3. Limits.** Three reloadable knobs under `global:`: @@ -94,7 +103,9 @@ global: query_max_duration: 0s ``` -These are enforced *during* execution against the query's actual running cost, not against the estimate: a query is rejected as soon as it loads too many series or scans too many samples, and `query_max_duration` surfaces as a query timeout. A client may lower any ceiling for a single request via `max_series`, `max_samples_scanned`, `max_query_duration`; these can only tighten, never loosen, the operator-set value. The estimate is never used to reject a query — enforcement is always on the real cost. +These are enforced *during* execution against the query's actual running cost, not against the estimate: a query is rejected as soon as it loads too many series or scans too many samples, and `query_max_duration` surfaces as a query timeout. A client may lower any ceiling for a single request via `max_series`, `max_samples_scanned`, `max_query_duration`. These can only tighten, never loosen, the operator-set value: a request that asks for a value above the server ceiling is silently clamped down to that ceiling, with no error, rather than rejected. The estimate is never used to reject a query — enforcement is always on the real cost. + +`query_max_duration` overlaps with the existing `-query.timeout` flag and `timeout` URL parameter, and is the reloadable, config-file equivalent of the former. To avoid two ways of doing the same thing, once `query_max_duration` proves out we propose to deprecate the `-query.timeout` *flag* in its favour. The per-query `timeout` URL parameter is retained and behaves like the other per-query overrides: it can only lower the effective ceiling, not raise it above `query_max_duration`. ### Testing and verification @@ -105,29 +116,26 @@ These are enforced *during* execution against the query's actual running cost, n ### Migration -Purely additive and behind a feature flag. Default config (all limits `0`) changes no behaviour. Nothing to migrate. +Purely additive and behind a feature flag. Default config (all limits `0`) changes no behaviour. ### Known unknowns -* **Estimate accuracy.** `SeriesTouched` over-counts shared series and series with no in-window samples; `SamplesScanned` assumes samples land exactly at the scrape interval and that sampled series are representative. Is an upper bound the right contract, or do we want something tighter? -* **Scrape interval.** The estimator uses the global scrape interval; per-target intervals are not modelled. +* **Estimate accuracy.** `SeriesTouched` still over-counts shared series and series with no in-window samples; `SamplesScanned` assumes samples land exactly at the measured or scrape interval and that sampled series are representative. Partially resolved when a selector's real window fits within the 50-chunk/50-series sample budget, the measurement is now taken from that real window instead of extrapolated from a nearby proxy window, so small selectors get an exact rather than approximate density (see How, section 1). Larger selectors still extrapolate from a bounded sample. Is an upper bound the right contract for those, or do we want something tighter? +* **Scrape interval.** Mostly resolved for TSDB-backed storage: the estimator measures the real density from chunk metadata automatically whenever it's available, with no configuration needed. The caller-supplied scrape interval remains a fallback only for a plain `storage.Queryable` with no chunk metadata (e.g. some remote-read backends), or when a selector's window has nothing to sample. +* **Sampling representativeness.** The sample budget is a fixed internal constant (50 chunks / 50 series), not a configurable knob. Which chunks/series to sample when a selector's real window doesn't fit the budget (the first ones returned by `Select` over a narrow window near the query's end) is an open question for large selectors — a poorly chosen sample could skew the extrapolation. * **Subqueries.** Only one level of nesting is modelled exactly. -* **Lookback delta.** The storage-only estimator uses the package default, not the engine's configured value. -* **Agent mode.** Estimation is unavailable (no queryable index). * **Config surface.** Should limits live under `global:`, or a dedicated `query:` section? +* **Units** Is it enough to return number of samples/series or do we want to return bytes? ## Alternatives 1. **Estimate from postings cardinality directly, bypassing `storage.Querier`.** Cheaper, but ties the estimator to the TSDB index and breaks for any other `storage.Queryable` (remote read, federation). Using the portable `Select` path keeps it storage-agnostic. -2. **Reject queries based on the estimate.** Rejected: the estimate is an upper bound and can be wrong in both directions. Rejecting on an estimate would refuse queries that would actually run fine. Enforcement is on real cost; the estimate is advisory only. +2. **Reject queries based on the estimate.** Rejected as the default: the estimate is an upper bound and can be wrong in both directions, so rejecting on it would refuse queries that would actually run fine. Enforcement is on real cost; the estimate is advisory only. There is a fair argument that letting a query that will almost certainly be limited run and fetch data anyway is wasteful. If the estimate proves accurate enough in practice (validated via the `cost` object's estimated-vs-actual comparison), an *opt-in* upfront rejection — reject before execution when the estimate clearly exceeds a ceiling — could be added later as a follow-up without changing the real-cost enforcement that remains the backstop. 3. **Reuse `--query.max-samples` and friends.** They are start-time flags measuring peak in-memory samples, not reloadable and not per-query. Extending them to be reloadable and per-query would overload their meaning; new, clearly-scoped knobs are cleaner. 4. **Do nothing / client-side estimation.** Clients cannot cheaply see the server's index cardinality, so any client-side guess is worse than a server estimate. ## Action Plan -* [ ] `promql.EstimateCost` and the sample-unit cost model -* [ ] `/api/v1/query_cost` and `/api/v1/query_range_cost` endpoints -* [ ] `cost` parameter on instant/range queries (estimated vs actual) -* [ ] Reloadable `query_max_series` / `query_max_samples_scanned` / `query_max_duration` under `global:` -* [ ] Per-query lowering via `max_series` / `max_samples_scanned` / `max_query_duration` -* [ ] `query-cost` feature flag, docs, OpenAPI spec, UI surfacing +- Implementation of the API endpoints +- Take feedback from the endpoint +- Work on enforcement / accuracy From b57e20adb3c83afe87ce7908b18cc27415409b55 Mon Sep 17 00:00:00 2001 From: Julien Pivotto <291750+roidelapluie@users.noreply.github.com> Date: Fri, 14 Aug 2026 12:36:32 +0200 Subject: [PATCH 3/5] Update and rename query cost proposal to 0089 Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com> --- proposals/{00089-query-cost.md => 0089-query-cost.md} | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) rename proposals/{00089-query-cost.md => 0089-query-cost.md} (92%) diff --git a/proposals/00089-query-cost.md b/proposals/0089-query-cost.md similarity index 92% rename from proposals/00089-query-cost.md rename to proposals/0089-query-cost.md index 4f59193..1e06339 100644 --- a/proposals/00089-query-cost.md +++ b/proposals/0089-query-cost.md @@ -31,7 +31,7 @@ Operators want ceilings they can tune without a restart. Users and tools (Grafan ## Goals -* Give a cheap, index-based cost *estimate* (series touched, samples scanned) without executing the query. +* Give a cheap cost *estimate* (series touched, samples scanned) without executing the query fully. * Expose the estimate through a new API so clients can gauge cost before running a query. * Add reloadable cost limits (`query_max_series`, `query_max_samples_scanned`, `query_max_duration`) enforced during execution. * Let a client *lower* those ceilings per query, never raise them. @@ -103,9 +103,9 @@ global: query_max_duration: 0s ``` -These are enforced *during* execution against the query's actual running cost, not against the estimate: a query is rejected as soon as it loads too many series or scans too many samples, and `query_max_duration` surfaces as a query timeout. A client may lower any ceiling for a single request via `max_series`, `max_samples_scanned`, `max_query_duration`. These can only tighten, never loosen, the operator-set value: a request that asks for a value above the server ceiling is silently clamped down to that ceiling, with no error, rather than rejected. The estimate is never used to reject a query — enforcement is always on the real cost. +These are enforced *during* execution against the query's actual running cost, not against the estimate: a query is rejected as soon as it loads too many series or scans too many samples, and `query_max_duration` surfaces as a query timeout. A client may lower any ceiling for a single request via `max_series`, `max_samples_scanned`, `max_query_duration`. These can only tighten, never loosen, the operator-set value: a request that asks for a value above the server ceiling is rejected, making it clear to the caller that the requested limit was not applied, rather than being silently clamped down. The estimate is never used to reject a query — enforcement is always on the real cost. -`query_max_duration` overlaps with the existing `-query.timeout` flag and `timeout` URL parameter, and is the reloadable, config-file equivalent of the former. To avoid two ways of doing the same thing, once `query_max_duration` proves out we propose to deprecate the `-query.timeout` *flag* in its favour. The per-query `timeout` URL parameter is retained and behaves like the other per-query overrides: it can only lower the effective ceiling, not raise it above `query_max_duration`. +`query_max_duration` is a normalization of the existing `-query.timeout` flag and `timeout` URL parameter, not a new concept: same semantics, but reloadable and config-file based. To avoid two ways of doing the same thing, the `-query.timeout` *flag* will be deprecated in favour of `query_max_duration`. The per-query `timeout` URL parameter is retained and behaves like the other per-query overrides: it can only lower the effective ceiling, not raise it above `query_max_duration`. ### Testing and verification From ea68cd1f8ead8927673a6437f84140803827956f Mon Sep 17 00:00:00 2001 From: Julien Pivotto <291750+roidelapluie@users.noreply.github.com> Date: Tue, 18 Aug 2026 10:25:57 +0200 Subject: [PATCH 4/5] Fix markdown formatting in query cost proposal Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com> --- proposals/0089-query-cost.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0089-query-cost.md b/proposals/0089-query-cost.md index 1e06339..5a2f438 100644 --- a/proposals/0089-query-cost.md +++ b/proposals/0089-query-cost.md @@ -25,7 +25,7 @@ Operators want ceilings they can tune without a restart. Users and tools (Grafan ### Pitfalls of the current solution * The existing limits are set at startup. Changing them means a restart. -* They are global. A single dashboard, query, cannot be given a tighter budget. +* They are global. A single dashboard or query cannot be given a tighter budget. * There is no pre-execution estimate. The only way to learn a query's cost today is to run it, which is exactly what we want to avoid for the expensive ones. * `--query.max-samples` measures peak in-memory samples, which does not map cleanly to "how much index and how many samples did this touch". From 4f0041849effef497220f7bbcd8234bb679e1036 Mon Sep 17 00:00:00 2001 From: Julien Pivotto <291750+roidelapluie@users.noreply.github.com> Date: Fri, 28 Aug 2026 12:50:44 +0200 Subject: [PATCH 5/5] Query Cost: clarify fallback sampling window and cost param semantics Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com> --- proposals/0089-query-cost.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/proposals/0089-query-cost.md b/proposals/0089-query-cost.md index 5a2f438..3c143c2 100644 --- a/proposals/0089-query-cost.md +++ b/proposals/0089-query-cost.md @@ -54,7 +54,7 @@ Operators running shared Prometheus servers, and UI/tooling authors (Grafana) th Three pieces, all gated by `--enable-feature=query-cost`. -**1. Estimation (`promql.EstimateCost`).** Parse the query, walk it for every vector and matrix selector, compute the effective time window each selector reads (mirroring the engine's `getTimeRangesForSelector`/`populateSeries`), and ask storage for the series count per selector via a single querier over the union window. `SeriesTouched` is the sum across selectors — an upper bound, because a series shared between selectors is counted once per selector. `SamplesScanned` models the engine's incremental per-step reads (full range window at step 0, then only the samples that advance past the previous cutoff), scaled by a measured average per-point cost so native-histogram points are sized by bucket rather than counted as one float unit. The estimate is index-only apart from decoding at most 50 (like `histogramSampleLimit`) points per selector. +**1. Estimation (`promql.EstimateCost`).** Parse the query, walk it for every vector and matrix selector, compute the effective time window each selector reads (mirroring the engine's `getTimeRangesForSelector`/`populateSeries`), and ask storage for the series count per selector via a single querier over the union window. `SeriesTouched` is the sum across selectors — an upper bound, because a series shared between selectors is counted once per selector. `SamplesScanned` models the engine's incremental per-step reads (full range window at step 0, then only the samples that advance past the previous cutoff), scaled by a measured average per-point cost so native-histogram points are sized by bucket rather than counted as one float unit. The estimate is index-only apart from decoding at most 50 (like `histogramSampleLimit`) points per selector. Every sampling budget below is *per selector*, not per series: whatever the selector's cardinality, the estimator faults in at most 50 chunks and decodes at most 50 first points for it in total. `SamplesScanned` is intended to approximate the *samples read* statistic introduced in [prometheus/prometheus#18081](https://github.com/prometheus/prometheus/pull/18081) — the total samples the engine reads from storage — not the older *total samples* (peak in-memory) statistic. Series with no in-window samples still count, so the figure stays an upper bound. @@ -63,7 +63,9 @@ The per-point density can be derived two ways: * **Scrape-interval (fallback, index-only).** Assume samples land at the global scrape interval and compute window ÷ interval. Cheapest, but wrong for series scraped at a different interval and for remote-written series, which have no scrape interval at all. Used only when nothing can be sampled (see below). * **Chunk sampling (always-on, not opt-in).** The estimator samples automatically, with no user-facing knob, whenever the storage exposes `storage.ChunkQueryable`: it reads up to a fixed `chunkSampleLimit` (50) chunks' `NumSamples` header to measure the selector's real sample interval, and decodes the first point of up to a fixed `histogramSampleLimit` (50) series to size native-histogram points by bucket count. Reading a chunk header is far cheaper than decoding its samples, so this stays much cheaper than executing the query. -Sampling prefers the *real* query window over a nearby proxy window whenever that real window is cheap enough: if a selector's actual chunk count (for density) or series count (for point cost) already fits within the 50-item budget, the estimator samples directly from `[sel.mint, sel.maxt]` and gets an exact rather than extrapolated measurement. Only when the real window has more chunks/series than the budget affords does it fall back to sampling a bounded, narrow window near the query's end and extrapolating. +Sampling prefers the *real* query window whenever that window is cheap enough: if a selector's actual chunk count (for density) or series count (for point cost) already fits within the 50-item budget, the estimator samples directly from `[sel.mint, sel.maxt]` and gets an exact rather than extrapolated measurement. Only when the real window has more chunks/series than the budget affords does it fall back to a *fallback sampling window* and extrapolate. + +The **fallback sampling window** is a short window ending at the selector's own `maxt` — `[sel.maxt - w, sel.maxt]` with `w = max(5m, 8 × scrape interval)` capped at 30m. It must follow the selector, not the query: a selector carrying an `offset` or `@` modifier reads a shifted window, and the density of today's data says nothing about the density of data a month ago — the series may have been scraped at a different interval, or may not exist today at all, in which case sampling near the query's end measures nothing and degrades to the global scrape interval for a selector whose real window is full of measurable chunks. Sampling at the selector's end costs no more: the budget caps the work at 50 chunk headers wherever the window sits. It remains a stand-in for the selector's real window: density and per-point cost measured there are assumed to hold over the whole window. **2. API.** Two new endpoints estimate cost without executing: @@ -92,7 +94,7 @@ The instant and range endpoints also gain a `cost=true` boolean parameter. When } ``` -`cost` is a plain on/off switch, not a set of levels: `cost=true` (any non-bool value errors) enables the comparison. There is no separate opt-in for chunk sampling, because chunk-metadata sampling always runs automatically wherever the storage supports it (see How, section 1). +`cost` is a plain on/off switch, not a set of levels: `cost=true` (any non-bool value errors) enables the comparison. It always runs a fresh estimation for the executed query, so it is not free — that is the point of the parameter: it exists to validate the estimator against reality, not to report cost cheaply. An actual-cost-only mode is deliberately absent because the actual figures are already available through `stats` (`samplesRead`, `totalSeries`, `peakSamples`); `cost` adds only the estimated side and the pairing. There is no separate opt-in for chunk sampling, because chunk-metadata sampling always runs automatically wherever the storage supports it (see How, section 1). **3. Limits.** Three reloadable knobs under `global:`: @@ -120,9 +122,9 @@ Purely additive and behind a feature flag. Default config (all limits `0`) chang ### Known unknowns -* **Estimate accuracy.** `SeriesTouched` still over-counts shared series and series with no in-window samples; `SamplesScanned` assumes samples land exactly at the measured or scrape interval and that sampled series are representative. Partially resolved when a selector's real window fits within the 50-chunk/50-series sample budget, the measurement is now taken from that real window instead of extrapolated from a nearby proxy window, so small selectors get an exact rather than approximate density (see How, section 1). Larger selectors still extrapolate from a bounded sample. Is an upper bound the right contract for those, or do we want something tighter? -* **Scrape interval.** Mostly resolved for TSDB-backed storage: the estimator measures the real density from chunk metadata automatically whenever it's available, with no configuration needed. The caller-supplied scrape interval remains a fallback only for a plain `storage.Queryable` with no chunk metadata (e.g. some remote-read backends), or when a selector's window has nothing to sample. -* **Sampling representativeness.** The sample budget is a fixed internal constant (50 chunks / 50 series), not a configurable knob. Which chunks/series to sample when a selector's real window doesn't fit the budget (the first ones returned by `Select` over a narrow window near the query's end) is an open question for large selectors — a poorly chosen sample could skew the extrapolation. +* **Estimate accuracy.** `SeriesTouched` still over-counts shared series and series with no in-window samples; `SamplesScanned` assumes samples land exactly at the measured or scrape interval and that sampled series are representative. Partially resolved when a selector's real window fits within the 50-chunk/50-series sample budget, the measurement is now taken from that real window instead of extrapolated from the fallback sampling window, so small selectors get an exact rather than approximate density (see How, section 1). Larger selectors still extrapolate from a bounded sample. Is an upper bound the right contract for those, or do we want something tighter? +* **Scrape interval.** Mostly resolved for TSDB-backed storage: the estimator measures the real density from chunk metadata automatically whenever it's available, with no configuration needed. The scrape interval remains a fallback only for a plain `storage.Queryable` with no chunk metadata (e.g. some remote-read backends), or when a selector's window has nothing to sample. It is not an API parameter: it is `global.scrape_interval` from the config file, passed to `promql.EstimateCost` by the API layer. Note that if per-series metadata carried the real scrape interval and we propagated it, the fallback could be exact per series instead of a global guess. +* **Sampling representativeness.** The sample budget is a fixed internal constant (50 chunks / 50 series), not a configurable knob. Which chunks/series to sample when a selector's real window doesn't fit the budget (the first ones returned by `Select` over the fallback sampling window) is an open question for large selectors — a poorly chosen sample could skew the extrapolation. Sampling at the selector's `maxt` can also fault in a cold block for a far-offset selector, which the `query_cost` endpoint would otherwise not touch; bounded at 50 chunk headers, this looks acceptable, but it is worth measuring. * **Subqueries.** Only one level of nesting is modelled exactly. * **Config surface.** Should limits live under `global:`, or a dedicated `query:` section? * **Units** Is it enough to return number of samples/series or do we want to return bytes?