Fixes flaky serverless reporting tests #244993
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ymao1
changed the title
Contributor
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#9963[❌] x-pack/platform/test/serverless/api_integration/configs/observability/config.group1.ts: 173/200 tests passed. |
ymao1
added
release_note:skip
Contributor
|
Pinging @elastic/response-ops (Team:ResponseOps) |
dmlemeshko
approved these changes
Dec 4, 2025
Contributor
Author
|
@elasticmachine merge upstream |
Contributor
💚 Build Succeeded
Metrics [docs]
History
cc @ymao1 |
Contributor
|
Starting backport for target branches: 9.2 |
This was referenced Dec 4, 2025
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Dec 4, 2025
Fixes elastic#237285 ## Summary The serverless version of these tests were failing with `Response aborted while reading the body` errors, which means the report is short-circuiting before completing. I noticed a previous change elastic#185878 to increase the time the test waits for a report to `10m` however, the queue timeout for reporting was still set to the default `4m` so this increases the queue timeout for reporting to `10m` in these tests to give it more time to complete. This should also address the following tests (all failed due to `Response aborted while reading the body`): - elastic#244269 - elastic#238317 - elastic#238182 - elastic#236220 Flaky test runner runs: There were some failures but seem unrelated... ``` 2025-12-02 22:03:51 UTC | ERROR UNHANDLED ERROR -- | -- 2025-12-02 22:03:51 UTC | ERROR ResponseError: index_not_found_exception 2025-12-02 22:03:51 UTC | Root causes: 2025-12-02 22:03:51 UTC | index_not_found_exception: no such index [.kibana-event-log-ds] 2025-12-02 22:03:51 UTC | at SniffingTransport._request (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1764712453512576396/elastic/kibana-flaky-test-suite-runner/kibana/node_modules/@elastic/elasticsearch/node_modules/@elastic/transport/src/Transport.ts:591:17) 2025-12-02 22:03:51 UTC | at processTicksAndRejections (node:internal/process/task_queues:105:5) 2025-12-02 22:03:51 UTC | at /opt/buildkite-agent/builds/bk-agent-prod-gcp-1764712453512576396/elastic/kibana-flaky-test-suite-runner/kibana/node_modules/@elastic/elasticsearch/node_modules/@elastic/transport/src/Transport.ts:697:22 2025-12-02 22:03:51 UTC | at SniffingTransport.request (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1764712453512576396/elastic/kibana-flaky-test-suite-runner/kibana/node_modules/@elastic/elasticsearch/node_modules/@elastic/transport/src/Transport.ts:694:14) 2025-12-02 22:03:51 UTC | at Indices.putSettings (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1764712453512576396/elastic/kibana-flaky-test-suite-runner/kibana/node_modules/@elastic/src/api/api/indices.ts:3529:12) 2025-12-02 22:03:51 UTC | at putFastRefreshSettings (es.ts:54:5) 2025-12-02 22:03:51 UTC | at es.ts:142:5 2025-12-02 22:03:51 UTC | at lifecycle_phase.ts:77:11 2025-12-02 22:03:51 UTC | at async Promise.all (index 2) 2025-12-02 22:03:51 UTC | at LifecyclePhase.trigger (lifecycle_phase.ts:74:5) 2025-12-02 22:03:51 UTC | at functional_test_runner.ts:115:9 2025-12-02 22:03:51 UTC | at FunctionalTestRunner.runHarness (functional_test_runner.ts:267:14) 2025-12-02 22:03:51 UTC | at FunctionalTestRunner.run (functional_test_runner.ts:49:12) 2025-12-02 22:03:51 UTC | at runFtr (run_ftr.ts:24:24) 2025-12-02 22:03:51 UTC | at run_tests.ts:180:11 2025-12-02 22:03:51 UTC | at withProcRunner (with_proc_runner.ts:30:5) 2025-12-02 22:03:51 UTC | at run_tests.ts:105:7 2025-12-02 22:03:51 UTC | at tooling_log.ts:95:18 2025-12-02 22:03:51 UTC | at runTests (run_tests.ts:67:5) 2025-12-02 22:03:51 UTC | at description (cli.ts:25:7) 2025-12-02 22:03:51 UTC | at run.ts:69:10 2025-12-02 22:03:51 UTC | at withProcRunner (with_proc_runner.ts:30:5) 2025-12-02 22:03:51 UTC | at run (run.ts:67:5) ``` - https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9963 Co-authored-by: Dzmitry Lemechko <[email protected]> Co-authored-by: Elastic Machine <[email protected]> (cherry picked from commit 58886b6)
Contributor
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Dec 4, 2025
# Backport This will backport the following commits from `main` to `9.2`: - [Fixes flaky serverless reporting tests (#244993)](#244993) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Ying Mao","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-12-04T15:15:00Z","message":"Fixes flaky serverless reporting tests (#244993)\n\nFixes https://github.com/elastic/kibana/issues/237285\n\n## Summary\n\nThe serverless version of these tests were failing with `Response\naborted while reading the body` errors, which means the report is\nshort-circuiting before completing. I noticed a previous change\nhttps://github.com//pull/185878 to increase the time the\ntest waits for a report to `10m` however, the queue timeout for\nreporting was still set to the default `4m` so this increases the queue\ntimeout for reporting to `10m` in these tests to give it more time to\ncomplete.\n\nThis should also address the following tests (all failed due to\n`Response aborted while reading the body`):\n- https://github.com/elastic/kibana/issues/244269\n- https://github.com/elastic/kibana/issues/238317\n- https://github.com/elastic/kibana/issues/238182\n- https://github.com/elastic/kibana/issues/236220\n\nFlaky test runner runs:\nThere were some failures but seem unrelated...\n\n```\n\n2025-12-02 22:03:51 UTC | ERROR UNHANDLED ERROR\n-- | --\n2025-12-02 22:03:51 UTC | ERROR ResponseError: index_not_found_exception\n2025-12-02 22:03:51 UTC | Root causes:\n2025-12-02 22:03:51 UTC | index_not_found_exception: no such index [.kibana-event-log-ds]\n2025-12-02 22:03:51 UTC | at SniffingTransport._request (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1764712453512576396/elastic/kibana-flaky-test-suite-runner/kibana/node_modules/@elastic/elasticsearch/node_modules/@elastic/transport/src/Transport.ts:591:17)\n2025-12-02 22:03:51 UTC | at processTicksAndRejections (node:internal/process/task_queues:105:5)\n2025-12-02 22:03:51 UTC | at /opt/buildkite-agent/builds/bk-agent-prod-gcp-1764712453512576396/elastic/kibana-flaky-test-suite-runner/kibana/node_modules/@elastic/elasticsearch/node_modules/@elastic/transport/src/Transport.ts:697:22\n2025-12-02 22:03:51 UTC | at SniffingTransport.request (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1764712453512576396/elastic/kibana-flaky-test-suite-runner/kibana/node_modules/@elastic/elasticsearch/node_modules/@elastic/transport/src/Transport.ts:694:14)\n2025-12-02 22:03:51 UTC | at Indices.putSettings (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1764712453512576396/elastic/kibana-flaky-test-suite-runner/kibana/node_modules/@elastic/src/api/api/indices.ts:3529:12)\n2025-12-02 22:03:51 UTC | at putFastRefreshSettings (es.ts:54:5)\n2025-12-02 22:03:51 UTC | at es.ts:142:5\n2025-12-02 22:03:51 UTC | at lifecycle_phase.ts:77:11\n2025-12-02 22:03:51 UTC | at async Promise.all (index 2)\n2025-12-02 22:03:51 UTC | at LifecyclePhase.trigger (lifecycle_phase.ts:74:5)\n2025-12-02 22:03:51 UTC | at functional_test_runner.ts:115:9\n2025-12-02 22:03:51 UTC | at FunctionalTestRunner.runHarness (functional_test_runner.ts:267:14)\n2025-12-02 22:03:51 UTC | at FunctionalTestRunner.run (functional_test_runner.ts:49:12)\n2025-12-02 22:03:51 UTC | at runFtr (run_ftr.ts:24:24)\n2025-12-02 22:03:51 UTC | at run_tests.ts:180:11\n2025-12-02 22:03:51 UTC | at withProcRunner (with_proc_runner.ts:30:5)\n2025-12-02 22:03:51 UTC | at run_tests.ts:105:7\n2025-12-02 22:03:51 UTC | at tooling_log.ts:95:18\n2025-12-02 22:03:51 UTC | at runTests (run_tests.ts:67:5)\n2025-12-02 22:03:51 UTC | at description (cli.ts:25:7)\n2025-12-02 22:03:51 UTC | at run.ts:69:10\n2025-12-02 22:03:51 UTC | at withProcRunner (with_proc_runner.ts:30:5)\n2025-12-02 22:03:51 UTC | at run (run.ts:67:5)\n```\n\n-\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9963\n\nCo-authored-by: Dzmitry Lemechko <[email protected]>\nCo-authored-by: Elastic Machine <[email protected]>","sha":"58886b6dea380724036828036fab169b062413c6","branchLabelMapping":{"^v9.3.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:ResponseOps","Feature:Reporting:Framework","backport:version","v9.3.0","v9.2.3"],"title":"Fixes flaky serverless reporting tests","number":244993,"url":"https://github.com/elastic/kibana/pull/244993","mergeCommit":{"message":"Fixes flaky serverless reporting tests (#244993)\n\nFixes https://github.com/elastic/kibana/issues/237285\n\n## Summary\n\nThe serverless version of these tests were failing with `Response\naborted while reading the body` errors, which means the report is\nshort-circuiting before completing. I noticed a previous change\nhttps://github.com//pull/185878 to increase the time the\ntest waits for a report to `10m` however, the queue timeout for\nreporting was still set to the default `4m` so this increases the queue\ntimeout for reporting to `10m` in these tests to give it more time to\ncomplete.\n\nThis should also address the following tests (all failed due to\n`Response aborted while reading the body`):\n- https://github.com/elastic/kibana/issues/244269\n- https://github.com/elastic/kibana/issues/238317\n- https://github.com/elastic/kibana/issues/238182\n- https://github.com/elastic/kibana/issues/236220\n\nFlaky test runner runs:\nThere were some failures but seem unrelated...\n\n```\n\n2025-12-02 22:03:51 UTC | ERROR UNHANDLED ERROR\n-- | --\n2025-12-02 22:03:51 UTC | ERROR ResponseError: index_not_found_exception\n2025-12-02 22:03:51 UTC | Root causes:\n2025-12-02 22:03:51 UTC | index_not_found_exception: no such index [.kibana-event-log-ds]\n2025-12-02 22:03:51 UTC | at SniffingTransport._request (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1764712453512576396/elastic/kibana-flaky-test-suite-runner/kibana/node_modules/@elastic/elasticsearch/node_modules/@elastic/transport/src/Transport.ts:591:17)\n2025-12-02 22:03:51 UTC | at processTicksAndRejections (node:internal/process/task_queues:105:5)\n2025-12-02 22:03:51 UTC | at /opt/buildkite-agent/builds/bk-agent-prod-gcp-1764712453512576396/elastic/kibana-flaky-test-suite-runner/kibana/node_modules/@elastic/elasticsearch/node_modules/@elastic/transport/src/Transport.ts:697:22\n2025-12-02 22:03:51 UTC | at SniffingTransport.request (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1764712453512576396/elastic/kibana-flaky-test-suite-runner/kibana/node_modules/@elastic/elasticsearch/node_modules/@elastic/transport/src/Transport.ts:694:14)\n2025-12-02 22:03:51 UTC | at Indices.putSettings (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1764712453512576396/elastic/kibana-flaky-test-suite-runner/kibana/node_modules/@elastic/src/api/api/indices.ts:3529:12)\n2025-12-02 22:03:51 UTC | at putFastRefreshSettings (es.ts:54:5)\n2025-12-02 22:03:51 UTC | at es.ts:142:5\n2025-12-02 22:03:51 UTC | at lifecycle_phase.ts:77:11\n2025-12-02 22:03:51 UTC | at async Promise.all (index 2)\n2025-12-02 22:03:51 UTC | at LifecyclePhase.trigger (lifecycle_phase.ts:74:5)\n2025-12-02 22:03:51 UTC | at functional_test_runner.ts:115:9\n2025-12-02 22:03:51 UTC | at FunctionalTestRunner.runHarness (functional_test_runner.ts:267:14)\n2025-12-02 22:03:51 UTC | at FunctionalTestRunner.run (functional_test_runner.ts:49:12)\n2025-12-02 22:03:51 UTC | at runFtr (run_ftr.ts:24:24)\n2025-12-02 22:03:51 UTC | at run_tests.ts:180:11\n2025-12-02 22:03:51 UTC | at withProcRunner (with_proc_runner.ts:30:5)\n2025-12-02 22:03:51 UTC | at run_tests.ts:105:7\n2025-12-02 22:03:51 UTC | at tooling_log.ts:95:18\n2025-12-02 22:03:51 UTC | at runTests (run_tests.ts:67:5)\n2025-12-02 22:03:51 UTC | at description (cli.ts:25:7)\n2025-12-02 22:03:51 UTC | at run.ts:69:10\n2025-12-02 22:03:51 UTC | at withProcRunner (with_proc_runner.ts:30:5)\n2025-12-02 22:03:51 UTC | at run (run.ts:67:5)\n```\n\n-\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9963\n\nCo-authored-by: Dzmitry Lemechko <[email protected]>\nCo-authored-by: Elastic Machine <[email protected]>","sha":"58886b6dea380724036828036fab169b062413c6"}},"sourceBranch":"main","suggestedTargetBranches":["9.2"],"targetPullRequestStates":[{"branch":"main","label":"v9.3.0","branchLabelMappingKey":"^v9.3.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/244993","number":244993,"mergeCommit":{"message":"Fixes flaky serverless reporting tests (#244993)\n\nFixes https://github.com/elastic/kibana/issues/237285\n\n## Summary\n\nThe serverless version of these tests were failing with `Response\naborted while reading the body` errors, which means the report is\nshort-circuiting before completing. I noticed a previous change\nhttps://github.com//pull/185878 to increase the time the\ntest waits for a report to `10m` however, the queue timeout for\nreporting was still set to the default `4m` so this increases the queue\ntimeout for reporting to `10m` in these tests to give it more time to\ncomplete.\n\nThis should also address the following tests (all failed due to\n`Response aborted while reading the body`):\n- https://github.com/elastic/kibana/issues/244269\n- https://github.com/elastic/kibana/issues/238317\n- https://github.com/elastic/kibana/issues/238182\n- https://github.com/elastic/kibana/issues/236220\n\nFlaky test runner runs:\nThere were some failures but seem unrelated...\n\n```\n\n2025-12-02 22:03:51 UTC | ERROR UNHANDLED ERROR\n-- | --\n2025-12-02 22:03:51 UTC | ERROR ResponseError: index_not_found_exception\n2025-12-02 22:03:51 UTC | Root causes:\n2025-12-02 22:03:51 UTC | index_not_found_exception: no such index [.kibana-event-log-ds]\n2025-12-02 22:03:51 UTC | at SniffingTransport._request (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1764712453512576396/elastic/kibana-flaky-test-suite-runner/kibana/node_modules/@elastic/elasticsearch/node_modules/@elastic/transport/src/Transport.ts:591:17)\n2025-12-02 22:03:51 UTC | at processTicksAndRejections (node:internal/process/task_queues:105:5)\n2025-12-02 22:03:51 UTC | at /opt/buildkite-agent/builds/bk-agent-prod-gcp-1764712453512576396/elastic/kibana-flaky-test-suite-runner/kibana/node_modules/@elastic/elasticsearch/node_modules/@elastic/transport/src/Transport.ts:697:22\n2025-12-02 22:03:51 UTC | at SniffingTransport.request (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1764712453512576396/elastic/kibana-flaky-test-suite-runner/kibana/node_modules/@elastic/elasticsearch/node_modules/@elastic/transport/src/Transport.ts:694:14)\n2025-12-02 22:03:51 UTC | at Indices.putSettings (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1764712453512576396/elastic/kibana-flaky-test-suite-runner/kibana/node_modules/@elastic/src/api/api/indices.ts:3529:12)\n2025-12-02 22:03:51 UTC | at putFastRefreshSettings (es.ts:54:5)\n2025-12-02 22:03:51 UTC | at es.ts:142:5\n2025-12-02 22:03:51 UTC | at lifecycle_phase.ts:77:11\n2025-12-02 22:03:51 UTC | at async Promise.all (index 2)\n2025-12-02 22:03:51 UTC | at LifecyclePhase.trigger (lifecycle_phase.ts:74:5)\n2025-12-02 22:03:51 UTC | at functional_test_runner.ts:115:9\n2025-12-02 22:03:51 UTC | at FunctionalTestRunner.runHarness (functional_test_runner.ts:267:14)\n2025-12-02 22:03:51 UTC | at FunctionalTestRunner.run (functional_test_runner.ts:49:12)\n2025-12-02 22:03:51 UTC | at runFtr (run_ftr.ts:24:24)\n2025-12-02 22:03:51 UTC | at run_tests.ts:180:11\n2025-12-02 22:03:51 UTC | at withProcRunner (with_proc_runner.ts:30:5)\n2025-12-02 22:03:51 UTC | at run_tests.ts:105:7\n2025-12-02 22:03:51 UTC | at tooling_log.ts:95:18\n2025-12-02 22:03:51 UTC | at runTests (run_tests.ts:67:5)\n2025-12-02 22:03:51 UTC | at description (cli.ts:25:7)\n2025-12-02 22:03:51 UTC | at run.ts:69:10\n2025-12-02 22:03:51 UTC | at withProcRunner (with_proc_runner.ts:30:5)\n2025-12-02 22:03:51 UTC | at run (run.ts:67:5)\n```\n\n-\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9963\n\nCo-authored-by: Dzmitry Lemechko <[email protected]>\nCo-authored-by: Elastic Machine <[email protected]>","sha":"58886b6dea380724036828036fab169b062413c6"}},{"branch":"9.2","label":"v9.2.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Ying Mao <[email protected]> Co-authored-by: Dzmitry Lemechko <[email protected]> Co-authored-by: Elastic Machine <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #237285
Summary
The serverless version of these tests were failing with
Response aborted while reading the bodyerrors, which means the report is short-circuiting before completing. I noticed a previous change #185878 to increase the time the test waits for a report to10mhowever, the queue timeout for reporting was still set to the default4mso this increases the queue timeout for reporting to10min these tests to give it more time to complete.This should also address the following tests (all failed due to
Response aborted while reading the body):Flaky test runner runs:
There were some failures but seem unrelated...