[Security Solution] Tests for prebuilt rule upgrade workflow #202078
Open
15 tasks done
Assignees
Labels
9.1 candidate
Feature:Prebuilt Detection Rules
Security Solution Prebuilt Detection Rules area
Team:Detection Rule Management
Security Detection Rule Management Team
Team:Detections and Resp
Security Detection Response Team
Team: SecuritySolution
Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.
test
test-coverage
issues & PRs for improving code test coverage
test-plan
v8.18.0
v8.19.0
v9.0.0
v9.1.0
Comments
banderror
added
8.18 candidate
Feature:Prebuilt Detection Rules
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
|
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
|
Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management) |
89 tasks
banderror
changed the title
|
@jpdjere @maximpn @nikitaindik I'd suggest us to split the work on the upgrade test coverage between @maximpn and @jpdjere. Considering the fact that @maximpn and @nikitaindik are busy with finalizing the ThreeWayDiff UI and fixing bugs in it, maybe @jpdjere could address most of this ticket, especially because he is very familiar with the requirements. Let's sync up on it when @jpdjere is back next week. |
14 tasks
|
cc @pborgonovi |
This was referenced Dec 10, 2024
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Dec 19, 2024
Partially resolves: elastic#202078 ## Summary This PR refactors the test plans involving the Installation and Upgrade Workflows: - breaks down the gigantic `installation_and_upgrade.md` file into three files: - `installation.md` covering exclusively prebuilt rule install workflows - `upgrade.md` covering exclusively prebuilt rule upgrade workflows - `package_installation_and_upgrade.md` which covers scenarios for the installation and upgrades of the Prebuilt Rules package itself. The *Upgrade of Prebuilt Rules* (`upgrade.md`) was modified as follows: - expanded to cover scenarios detailed in the linked ticket (`Upgrading prebuilt rules one-by-one without preview` and `Upgrading prebuilt rules in bulk` in the Rule Updates table, including cases with and without conflicts, as well as a mixture of both) - internally refactored so that sections that will become legacy with the release of Milestone 3 were pushed to the end and marked as pending deletion once the feature is live. - from point above, the test plan now is in a better position to add the scenarios for the Three-Way-Diff upgrade workflow. cc @maximpn @nikitaindik ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: Maxim Palenov <maxim.palenov@elastic.co> (cherry picked from commit cd74ecf)
|
The ticket was closed automatically after merging #203331. Reopening it since all scope is not covered yet. |
JoseLuisGJ
pushed a commit
to JoseLuisGJ/kibana
that referenced
this issue
Dec 19, 2024
Partially resolves: elastic#202078 ## Summary This PR refactors the test plans involving the Installation and Upgrade Workflows: - breaks down the gigantic `installation_and_upgrade.md` file into three files: - `installation.md` covering exclusively prebuilt rule install workflows - `upgrade.md` covering exclusively prebuilt rule upgrade workflows - `package_installation_and_upgrade.md` which covers scenarios for the installation and upgrades of the Prebuilt Rules package itself. The *Upgrade of Prebuilt Rules* (`upgrade.md`) was modified as follows: - expanded to cover scenarios detailed in the linked ticket (`Upgrading prebuilt rules one-by-one without preview` and `Upgrading prebuilt rules in bulk` in the Rule Updates table, including cases with and without conflicts, as well as a mixture of both) - internally refactored so that sections that will become legacy with the release of Milestone 3 were pushed to the end and marked as pending deletion once the feature is live. - from point above, the test plan now is in a better position to add the scenarios for the Three-Way-Diff upgrade workflow. cc @maximpn @nikitaindik ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: Maxim Palenov <maxim.palenov@elastic.co>
2 tasks
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Feb 26, 2025
…part 3 (elastic#212420) **Epic:** elastic#174168 **Partially addresses:** elastic#202068, elastic#202078, elastic#202079 **Follow-up to:** elastic#211472 ## Summary We're cleaning up and refactoring our existing test plans for prebuilt rule customization, upgrade, and export/import workflows. Specifically, this PR: - Updates the common structure of test plans: - Adds a tip to each test plan's header to help with navigating all our test plans for prebuilt rules. - Introduces a new `Requirements` section to each one and consolidates assumptions, technical requirements and product requirements under it. - Each `Product requirements` section now contains relevant user stories and other product details. In the document flow this is kept close to the `Scenarios` section. - Updates the test plan template according to the updated structure. - Moves the documentation about customizable and non-customizable rule fields to the common info document. - Restores the lost changes from commit elastic@bde4612. No "functional" changes have been made to any test plans, such as adding, removing, or updating any scenarios. The only functional-like changes were made to the user stories when creating/updating the product requirements sections. This PR wraps up the refactoring efforts aimed to prepare the test plans for being "functionally" changed and improved in follow-up PRs. In the following PRs, we're going to cover the logic described in elastic#210358, address any gaps in the coverage, and improve the scenarios structure and wording. ## Review tip It might be easier to review this PR commit-by-commit as each of them contains logically cohesive changes. (cherry picked from commit c961659)
kibanamachine
added a commit
that referenced
this issue
Feb 26, 2025
…lans, part 3 (#212420) (#212481) # Backport This will backport the following commits from `main` to `9.0`: - [[Security Solution] Refactor prebuilt rule customization test plans, part 3 (#212420)](#212420) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Georgii Gorbachev","email":"georgii.gorbachev@elastic.co"},"sourceCommit":{"committedDate":"2025-02-26T08:39:32Z","message":"[Security Solution] Refactor prebuilt rule customization test plans, part 3 (#212420)\n\n**Epic:** https://github.com/elastic/kibana/issues/174168\n**Partially addresses:**\nhttps://github.com//issues/202068,\nhttps://github.com//issues/202078,\nhttps://github.com//issues/202079\n**Follow-up to:** https://github.com/elastic/kibana/pull/211472\n\n## Summary\n\nWe're cleaning up and refactoring our existing test plans for prebuilt\nrule customization, upgrade, and export/import workflows.\n\nSpecifically, this PR:\n\n- Updates the common structure of test plans:\n- Adds a tip to each test plan's header to help with navigating all our\ntest plans for prebuilt rules.\n- Introduces a new `Requirements` section to each one and consolidates\nassumptions, technical requirements and product requirements under it.\n- Each `Product requirements` section now contains relevant user stories\nand other product details. In the document flow this is kept close to\nthe `Scenarios` section.\n - Updates the test plan template according to the updated structure.\n- Moves the documentation about customizable and non-customizable rule\nfields to the common info document.\n- Restores the lost changes from commit\nhttps://github.com/elastic/kibana/commit/bde4612e5256b3b779961736291ccae2e328c310.\n\nNo \"functional\" changes have been made to any test plans, such as\nadding, removing, or updating any scenarios. The only functional-like\nchanges were made to the user stories when creating/updating the product\nrequirements sections.\n\nThis PR wraps up the refactoring efforts aimed to prepare the test plans\nfor being \"functionally\" changed and improved in follow-up PRs. In the\nfollowing PRs, we're going to cover the logic described in\nhttps://github.com//issues/210358, address any gaps in the\ncoverage, and improve the scenarios structure and wording.\n\n## Review tip\n\nIt might be easier to review this PR commit-by-commit as each of them\ncontains logically cohesive changes.","sha":"c9616593c1deeb52f73fa5720f16288073943f3e","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","test-plan","v9.0.0","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Prebuilt Detection Rules","backport:version","v8.18.0","v9.1.0","v8.19.0"],"title":"[Security Solution] Refactor prebuilt rule customization test plans, part 3","number":212420,"url":"https://github.com/elastic/kibana/pull/212420","mergeCommit":{"message":"[Security Solution] Refactor prebuilt rule customization test plans, part 3 (#212420)\n\n**Epic:** https://github.com/elastic/kibana/issues/174168\n**Partially addresses:**\nhttps://github.com//issues/202068,\nhttps://github.com//issues/202078,\nhttps://github.com//issues/202079\n**Follow-up to:** https://github.com/elastic/kibana/pull/211472\n\n## Summary\n\nWe're cleaning up and refactoring our existing test plans for prebuilt\nrule customization, upgrade, and export/import workflows.\n\nSpecifically, this PR:\n\n- Updates the common structure of test plans:\n- Adds a tip to each test plan's header to help with navigating all our\ntest plans for prebuilt rules.\n- Introduces a new `Requirements` section to each one and consolidates\nassumptions, technical requirements and product requirements under it.\n- Each `Product requirements` section now contains relevant user stories\nand other product details. In the document flow this is kept close to\nthe `Scenarios` section.\n - Updates the test plan template according to the updated structure.\n- Moves the documentation about customizable and non-customizable rule\nfields to the common info document.\n- Restores the lost changes from commit\nhttps://github.com/elastic/kibana/commit/bde4612e5256b3b779961736291ccae2e328c310.\n\nNo \"functional\" changes have been made to any test plans, such as\nadding, removing, or updating any scenarios. The only functional-like\nchanges were made to the user stories when creating/updating the product\nrequirements sections.\n\nThis PR wraps up the refactoring efforts aimed to prepare the test plans\nfor being \"functionally\" changed and improved in follow-up PRs. In the\nfollowing PRs, we're going to cover the logic described in\nhttps://github.com//issues/210358, address any gaps in the\ncoverage, and improve the scenarios structure and wording.\n\n## Review tip\n\nIt might be easier to review this PR commit-by-commit as each of them\ncontains logically cohesive changes.","sha":"c9616593c1deeb52f73fa5720f16288073943f3e"}},"sourceBranch":"main","suggestedTargetBranches":["9.0","8.18","8.x"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/212420","number":212420,"mergeCommit":{"message":"[Security Solution] Refactor prebuilt rule customization test plans, part 3 (#212420)\n\n**Epic:** https://github.com/elastic/kibana/issues/174168\n**Partially addresses:**\nhttps://github.com//issues/202068,\nhttps://github.com//issues/202078,\nhttps://github.com//issues/202079\n**Follow-up to:** https://github.com/elastic/kibana/pull/211472\n\n## Summary\n\nWe're cleaning up and refactoring our existing test plans for prebuilt\nrule customization, upgrade, and export/import workflows.\n\nSpecifically, this PR:\n\n- Updates the common structure of test plans:\n- Adds a tip to each test plan's header to help with navigating all our\ntest plans for prebuilt rules.\n- Introduces a new `Requirements` section to each one and consolidates\nassumptions, technical requirements and product requirements under it.\n- Each `Product requirements` section now contains relevant user stories\nand other product details. In the document flow this is kept close to\nthe `Scenarios` section.\n - Updates the test plan template according to the updated structure.\n- Moves the documentation about customizable and non-customizable rule\nfields to the common info document.\n- Restores the lost changes from commit\nhttps://github.com/elastic/kibana/commit/bde4612e5256b3b779961736291ccae2e328c310.\n\nNo \"functional\" changes have been made to any test plans, such as\nadding, removing, or updating any scenarios. The only functional-like\nchanges were made to the user stories when creating/updating the product\nrequirements sections.\n\nThis PR wraps up the refactoring efforts aimed to prepare the test plans\nfor being \"functionally\" changed and improved in follow-up PRs. In the\nfollowing PRs, we're going to cover the logic described in\nhttps://github.com//issues/210358, address any gaps in the\ncoverage, and improve the scenarios structure and wording.\n\n## Review tip\n\nIt might be easier to review this PR commit-by-commit as each of them\ncontains logically cohesive changes.","sha":"c9616593c1deeb52f73fa5720f16288073943f3e"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>
kibanamachine
added a commit
that referenced
this issue
Feb 26, 2025
…plans, part 3 (#212420) (#212479) # Backport This will backport the following commits from `main` to `8.18`: - [[Security Solution] Refactor prebuilt rule customization test plans, part 3 (#212420)](#212420) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Georgii Gorbachev","email":"georgii.gorbachev@elastic.co"},"sourceCommit":{"committedDate":"2025-02-26T08:39:32Z","message":"[Security Solution] Refactor prebuilt rule customization test plans, part 3 (#212420)\n\n**Epic:** https://github.com/elastic/kibana/issues/174168\n**Partially addresses:**\nhttps://github.com//issues/202068,\nhttps://github.com//issues/202078,\nhttps://github.com//issues/202079\n**Follow-up to:** https://github.com/elastic/kibana/pull/211472\n\n## Summary\n\nWe're cleaning up and refactoring our existing test plans for prebuilt\nrule customization, upgrade, and export/import workflows.\n\nSpecifically, this PR:\n\n- Updates the common structure of test plans:\n- Adds a tip to each test plan's header to help with navigating all our\ntest plans for prebuilt rules.\n- Introduces a new `Requirements` section to each one and consolidates\nassumptions, technical requirements and product requirements under it.\n- Each `Product requirements` section now contains relevant user stories\nand other product details. In the document flow this is kept close to\nthe `Scenarios` section.\n - Updates the test plan template according to the updated structure.\n- Moves the documentation about customizable and non-customizable rule\nfields to the common info document.\n- Restores the lost changes from commit\nhttps://github.com/elastic/kibana/commit/bde4612e5256b3b779961736291ccae2e328c310.\n\nNo \"functional\" changes have been made to any test plans, such as\nadding, removing, or updating any scenarios. The only functional-like\nchanges were made to the user stories when creating/updating the product\nrequirements sections.\n\nThis PR wraps up the refactoring efforts aimed to prepare the test plans\nfor being \"functionally\" changed and improved in follow-up PRs. In the\nfollowing PRs, we're going to cover the logic described in\nhttps://github.com//issues/210358, address any gaps in the\ncoverage, and improve the scenarios structure and wording.\n\n## Review tip\n\nIt might be easier to review this PR commit-by-commit as each of them\ncontains logically cohesive changes.","sha":"c9616593c1deeb52f73fa5720f16288073943f3e","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","test-plan","v9.0.0","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Prebuilt Detection Rules","backport:version","v8.18.0","v9.1.0","v8.19.0"],"title":"[Security Solution] Refactor prebuilt rule customization test plans, part 3","number":212420,"url":"https://github.com/elastic/kibana/pull/212420","mergeCommit":{"message":"[Security Solution] Refactor prebuilt rule customization test plans, part 3 (#212420)\n\n**Epic:** https://github.com/elastic/kibana/issues/174168\n**Partially addresses:**\nhttps://github.com//issues/202068,\nhttps://github.com//issues/202078,\nhttps://github.com//issues/202079\n**Follow-up to:** https://github.com/elastic/kibana/pull/211472\n\n## Summary\n\nWe're cleaning up and refactoring our existing test plans for prebuilt\nrule customization, upgrade, and export/import workflows.\n\nSpecifically, this PR:\n\n- Updates the common structure of test plans:\n- Adds a tip to each test plan's header to help with navigating all our\ntest plans for prebuilt rules.\n- Introduces a new `Requirements` section to each one and consolidates\nassumptions, technical requirements and product requirements under it.\n- Each `Product requirements` section now contains relevant user stories\nand other product details. In the document flow this is kept close to\nthe `Scenarios` section.\n - Updates the test plan template according to the updated structure.\n- Moves the documentation about customizable and non-customizable rule\nfields to the common info document.\n- Restores the lost changes from commit\nhttps://github.com/elastic/kibana/commit/bde4612e5256b3b779961736291ccae2e328c310.\n\nNo \"functional\" changes have been made to any test plans, such as\nadding, removing, or updating any scenarios. The only functional-like\nchanges were made to the user stories when creating/updating the product\nrequirements sections.\n\nThis PR wraps up the refactoring efforts aimed to prepare the test plans\nfor being \"functionally\" changed and improved in follow-up PRs. In the\nfollowing PRs, we're going to cover the logic described in\nhttps://github.com//issues/210358, address any gaps in the\ncoverage, and improve the scenarios structure and wording.\n\n## Review tip\n\nIt might be easier to review this PR commit-by-commit as each of them\ncontains logically cohesive changes.","sha":"c9616593c1deeb52f73fa5720f16288073943f3e"}},"sourceBranch":"main","suggestedTargetBranches":["9.0","8.18","8.x"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/212420","number":212420,"mergeCommit":{"message":"[Security Solution] Refactor prebuilt rule customization test plans, part 3 (#212420)\n\n**Epic:** https://github.com/elastic/kibana/issues/174168\n**Partially addresses:**\nhttps://github.com//issues/202068,\nhttps://github.com//issues/202078,\nhttps://github.com//issues/202079\n**Follow-up to:** https://github.com/elastic/kibana/pull/211472\n\n## Summary\n\nWe're cleaning up and refactoring our existing test plans for prebuilt\nrule customization, upgrade, and export/import workflows.\n\nSpecifically, this PR:\n\n- Updates the common structure of test plans:\n- Adds a tip to each test plan's header to help with navigating all our\ntest plans for prebuilt rules.\n- Introduces a new `Requirements` section to each one and consolidates\nassumptions, technical requirements and product requirements under it.\n- Each `Product requirements` section now contains relevant user stories\nand other product details. In the document flow this is kept close to\nthe `Scenarios` section.\n - Updates the test plan template according to the updated structure.\n- Moves the documentation about customizable and non-customizable rule\nfields to the common info document.\n- Restores the lost changes from commit\nhttps://github.com/elastic/kibana/commit/bde4612e5256b3b779961736291ccae2e328c310.\n\nNo \"functional\" changes have been made to any test plans, such as\nadding, removing, or updating any scenarios. The only functional-like\nchanges were made to the user stories when creating/updating the product\nrequirements sections.\n\nThis PR wraps up the refactoring efforts aimed to prepare the test plans\nfor being \"functionally\" changed and improved in follow-up PRs. In the\nfollowing PRs, we're going to cover the logic described in\nhttps://github.com//issues/210358, address any gaps in the\ncoverage, and improve the scenarios structure and wording.\n\n## Review tip\n\nIt might be easier to review this PR commit-by-commit as each of them\ncontains logically cohesive changes.","sha":"c9616593c1deeb52f73fa5720f16288073943f3e"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>
kibanamachine
added a commit
that referenced
this issue
Feb 26, 2025
…lans, part 3 (#212420) (#212480) # Backport This will backport the following commits from `main` to `8.x`: - [[Security Solution] Refactor prebuilt rule customization test plans, part 3 (#212420)](#212420) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Georgii Gorbachev","email":"georgii.gorbachev@elastic.co"},"sourceCommit":{"committedDate":"2025-02-26T08:39:32Z","message":"[Security Solution] Refactor prebuilt rule customization test plans, part 3 (#212420)\n\n**Epic:** https://github.com/elastic/kibana/issues/174168\n**Partially addresses:**\nhttps://github.com//issues/202068,\nhttps://github.com//issues/202078,\nhttps://github.com//issues/202079\n**Follow-up to:** https://github.com/elastic/kibana/pull/211472\n\n## Summary\n\nWe're cleaning up and refactoring our existing test plans for prebuilt\nrule customization, upgrade, and export/import workflows.\n\nSpecifically, this PR:\n\n- Updates the common structure of test plans:\n- Adds a tip to each test plan's header to help with navigating all our\ntest plans for prebuilt rules.\n- Introduces a new `Requirements` section to each one and consolidates\nassumptions, technical requirements and product requirements under it.\n- Each `Product requirements` section now contains relevant user stories\nand other product details. In the document flow this is kept close to\nthe `Scenarios` section.\n - Updates the test plan template according to the updated structure.\n- Moves the documentation about customizable and non-customizable rule\nfields to the common info document.\n- Restores the lost changes from commit\nhttps://github.com/elastic/kibana/commit/bde4612e5256b3b779961736291ccae2e328c310.\n\nNo \"functional\" changes have been made to any test plans, such as\nadding, removing, or updating any scenarios. The only functional-like\nchanges were made to the user stories when creating/updating the product\nrequirements sections.\n\nThis PR wraps up the refactoring efforts aimed to prepare the test plans\nfor being \"functionally\" changed and improved in follow-up PRs. In the\nfollowing PRs, we're going to cover the logic described in\nhttps://github.com//issues/210358, address any gaps in the\ncoverage, and improve the scenarios structure and wording.\n\n## Review tip\n\nIt might be easier to review this PR commit-by-commit as each of them\ncontains logically cohesive changes.","sha":"c9616593c1deeb52f73fa5720f16288073943f3e","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","test-plan","v9.0.0","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Prebuilt Detection Rules","backport:version","v8.18.0","v9.1.0","v8.19.0"],"title":"[Security Solution] Refactor prebuilt rule customization test plans, part 3","number":212420,"url":"https://github.com/elastic/kibana/pull/212420","mergeCommit":{"message":"[Security Solution] Refactor prebuilt rule customization test plans, part 3 (#212420)\n\n**Epic:** https://github.com/elastic/kibana/issues/174168\n**Partially addresses:**\nhttps://github.com//issues/202068,\nhttps://github.com//issues/202078,\nhttps://github.com//issues/202079\n**Follow-up to:** https://github.com/elastic/kibana/pull/211472\n\n## Summary\n\nWe're cleaning up and refactoring our existing test plans for prebuilt\nrule customization, upgrade, and export/import workflows.\n\nSpecifically, this PR:\n\n- Updates the common structure of test plans:\n- Adds a tip to each test plan's header to help with navigating all our\ntest plans for prebuilt rules.\n- Introduces a new `Requirements` section to each one and consolidates\nassumptions, technical requirements and product requirements under it.\n- Each `Product requirements` section now contains relevant user stories\nand other product details. In the document flow this is kept close to\nthe `Scenarios` section.\n - Updates the test plan template according to the updated structure.\n- Moves the documentation about customizable and non-customizable rule\nfields to the common info document.\n- Restores the lost changes from commit\nhttps://github.com/elastic/kibana/commit/bde4612e5256b3b779961736291ccae2e328c310.\n\nNo \"functional\" changes have been made to any test plans, such as\nadding, removing, or updating any scenarios. The only functional-like\nchanges were made to the user stories when creating/updating the product\nrequirements sections.\n\nThis PR wraps up the refactoring efforts aimed to prepare the test plans\nfor being \"functionally\" changed and improved in follow-up PRs. In the\nfollowing PRs, we're going to cover the logic described in\nhttps://github.com//issues/210358, address any gaps in the\ncoverage, and improve the scenarios structure and wording.\n\n## Review tip\n\nIt might be easier to review this PR commit-by-commit as each of them\ncontains logically cohesive changes.","sha":"c9616593c1deeb52f73fa5720f16288073943f3e"}},"sourceBranch":"main","suggestedTargetBranches":["9.0","8.18","8.x"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/212420","number":212420,"mergeCommit":{"message":"[Security Solution] Refactor prebuilt rule customization test plans, part 3 (#212420)\n\n**Epic:** https://github.com/elastic/kibana/issues/174168\n**Partially addresses:**\nhttps://github.com//issues/202068,\nhttps://github.com//issues/202078,\nhttps://github.com//issues/202079\n**Follow-up to:** https://github.com/elastic/kibana/pull/211472\n\n## Summary\n\nWe're cleaning up and refactoring our existing test plans for prebuilt\nrule customization, upgrade, and export/import workflows.\n\nSpecifically, this PR:\n\n- Updates the common structure of test plans:\n- Adds a tip to each test plan's header to help with navigating all our\ntest plans for prebuilt rules.\n- Introduces a new `Requirements` section to each one and consolidates\nassumptions, technical requirements and product requirements under it.\n- Each `Product requirements` section now contains relevant user stories\nand other product details. In the document flow this is kept close to\nthe `Scenarios` section.\n - Updates the test plan template according to the updated structure.\n- Moves the documentation about customizable and non-customizable rule\nfields to the common info document.\n- Restores the lost changes from commit\nhttps://github.com/elastic/kibana/commit/bde4612e5256b3b779961736291ccae2e328c310.\n\nNo \"functional\" changes have been made to any test plans, such as\nadding, removing, or updating any scenarios. The only functional-like\nchanges were made to the user stories when creating/updating the product\nrequirements sections.\n\nThis PR wraps up the refactoring efforts aimed to prepare the test plans\nfor being \"functionally\" changed and improved in follow-up PRs. In the\nfollowing PRs, we're going to cover the logic described in\nhttps://github.com//issues/210358, address any gaps in the\ncoverage, and improve the scenarios structure and wording.\n\n## Review tip\n\nIt might be easier to review this PR commit-by-commit as each of them\ncontains logically cohesive changes.","sha":"c9616593c1deeb52f73fa5720f16288073943f3e"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>
JoseLuisGJ
pushed a commit
to JoseLuisGJ/kibana
that referenced
this issue
Feb 27, 2025
…t rule customization feature (elastic#209260) **Addresses:** elastic#202078 **Partially implements test plan:** elastic#205922 ## Summary This PR implements the Rule Upgrade test plan added in elastic#205922 and elastic#203331. ## Details Tests cover only enabled customization and organized in a following way - `diffable_rule_fields` folder contains per field tests. It focuses on `DiffableRule`* fields. Each field has rule preview and upgrade scenarios throughly covered. Assertion on stats are included in rule preview tests to make sure the maximum coverage with reasonable execution time. **Diffable rule fields` tests were moved to a separate execution group to avoid exceeding execution limits.** - `preview_prebuilt_rules_upgrade.ts` integration tests on `/internal/prebuilt_rules/upgrade/_review` endpoint. It uses only `name` and `tags` fields to minimize tests complexity. Basically it should be considered as smoke tests. - `upgrade_prebuilt_rules` integrations test on `/internal/prebuilt_rules/upgrade/_perform` endpoint. It uses only `name` and `tags` fields to minimize tests complexity. Basically it should be considered as smoke tests. This PR supersedes elastic#205217 and elastic#205339. * `DiffableRule` was added in the scope of prebuilt rules customization epic to serve rule upgrades preview and performing upgrades. It represents slightly reorganized rule fields to simplify prebuilt rule upgrade workflow handling. There are utility functions transforming between `RuleResponse` and `DiffableRule`. ## Flaky test runner Flaky test runs for 100 iterations were successful (elastic#209260 (comment) and elastic#209260 (comment)).
JoseLuisGJ
pushed a commit
to JoseLuisGJ/kibana
that referenced
this issue
Feb 27, 2025
…part 3 (elastic#212420) **Epic:** elastic#174168 **Partially addresses:** elastic#202068, elastic#202078, elastic#202079 **Follow-up to:** elastic#211472 ## Summary We're cleaning up and refactoring our existing test plans for prebuilt rule customization, upgrade, and export/import workflows. Specifically, this PR: - Updates the common structure of test plans: - Adds a tip to each test plan's header to help with navigating all our test plans for prebuilt rules. - Introduces a new `Requirements` section to each one and consolidates assumptions, technical requirements and product requirements under it. - Each `Product requirements` section now contains relevant user stories and other product details. In the document flow this is kept close to the `Scenarios` section. - Updates the test plan template according to the updated structure. - Moves the documentation about customizable and non-customizable rule fields to the common info document. - Restores the lost changes from commit elastic@bde4612. No "functional" changes have been made to any test plans, such as adding, removing, or updating any scenarios. The only functional-like changes were made to the user stories when creating/updating the product requirements sections. This PR wraps up the refactoring efforts aimed to prepare the test plans for being "functionally" changed and improved in follow-up PRs. In the following PRs, we're going to cover the logic described in elastic#210358, address any gaps in the coverage, and improve the scenarios structure and wording. ## Review tip It might be easier to review this PR commit-by-commit as each of them contains logically cohesive changes.
SoniaSanzV
pushed a commit
to SoniaSanzV/kibana
that referenced
this issue
Mar 4, 2025
…lans, part 3 (elastic#212420) (elastic#212480) # Backport This will backport the following commits from `main` to `8.x`: - [[Security Solution] Refactor prebuilt rule customization test plans, part 3 (elastic#212420)](elastic#212420) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Georgii Gorbachev","email":"georgii.gorbachev@elastic.co"},"sourceCommit":{"committedDate":"2025-02-26T08:39:32Z","message":"[Security Solution] Refactor prebuilt rule customization test plans, part 3 (elastic#212420)\n\n**Epic:** https://github.com/elastic/kibana/issues/174168\n**Partially addresses:**\nhttps://github.com/elastic/issues/202068,\nhttps://github.com/elastic/issues/202078,\nhttps://github.com/elastic/issues/202079\n**Follow-up to:** https://github.com/elastic/kibana/pull/211472\n\n## Summary\n\nWe're cleaning up and refactoring our existing test plans for prebuilt\nrule customization, upgrade, and export/import workflows.\n\nSpecifically, this PR:\n\n- Updates the common structure of test plans:\n- Adds a tip to each test plan's header to help with navigating all our\ntest plans for prebuilt rules.\n- Introduces a new `Requirements` section to each one and consolidates\nassumptions, technical requirements and product requirements under it.\n- Each `Product requirements` section now contains relevant user stories\nand other product details. In the document flow this is kept close to\nthe `Scenarios` section.\n - Updates the test plan template according to the updated structure.\n- Moves the documentation about customizable and non-customizable rule\nfields to the common info document.\n- Restores the lost changes from commit\nhttps://github.com/elastic/kibana/commit/bde4612e5256b3b779961736291ccae2e328c310.\n\nNo \"functional\" changes have been made to any test plans, such as\nadding, removing, or updating any scenarios. The only functional-like\nchanges were made to the user stories when creating/updating the product\nrequirements sections.\n\nThis PR wraps up the refactoring efforts aimed to prepare the test plans\nfor being \"functionally\" changed and improved in follow-up PRs. In the\nfollowing PRs, we're going to cover the logic described in\nhttps://github.com/elastic/issues/210358, address any gaps in the\ncoverage, and improve the scenarios structure and wording.\n\n## Review tip\n\nIt might be easier to review this PR commit-by-commit as each of them\ncontains logically cohesive changes.","sha":"c9616593c1deeb52f73fa5720f16288073943f3e","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","test-plan","v9.0.0","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Prebuilt Detection Rules","backport:version","v8.18.0","v9.1.0","v8.19.0"],"title":"[Security Solution] Refactor prebuilt rule customization test plans, part 3","number":212420,"url":"https://github.com/elastic/kibana/pull/212420","mergeCommit":{"message":"[Security Solution] Refactor prebuilt rule customization test plans, part 3 (elastic#212420)\n\n**Epic:** https://github.com/elastic/kibana/issues/174168\n**Partially addresses:**\nhttps://github.com/elastic/issues/202068,\nhttps://github.com/elastic/issues/202078,\nhttps://github.com/elastic/issues/202079\n**Follow-up to:** https://github.com/elastic/kibana/pull/211472\n\n## Summary\n\nWe're cleaning up and refactoring our existing test plans for prebuilt\nrule customization, upgrade, and export/import workflows.\n\nSpecifically, this PR:\n\n- Updates the common structure of test plans:\n- Adds a tip to each test plan's header to help with navigating all our\ntest plans for prebuilt rules.\n- Introduces a new `Requirements` section to each one and consolidates\nassumptions, technical requirements and product requirements under it.\n- Each `Product requirements` section now contains relevant user stories\nand other product details. In the document flow this is kept close to\nthe `Scenarios` section.\n - Updates the test plan template according to the updated structure.\n- Moves the documentation about customizable and non-customizable rule\nfields to the common info document.\n- Restores the lost changes from commit\nhttps://github.com/elastic/kibana/commit/bde4612e5256b3b779961736291ccae2e328c310.\n\nNo \"functional\" changes have been made to any test plans, such as\nadding, removing, or updating any scenarios. The only functional-like\nchanges were made to the user stories when creating/updating the product\nrequirements sections.\n\nThis PR wraps up the refactoring efforts aimed to prepare the test plans\nfor being \"functionally\" changed and improved in follow-up PRs. In the\nfollowing PRs, we're going to cover the logic described in\nhttps://github.com/elastic/issues/210358, address any gaps in the\ncoverage, and improve the scenarios structure and wording.\n\n## Review tip\n\nIt might be easier to review this PR commit-by-commit as each of them\ncontains logically cohesive changes.","sha":"c9616593c1deeb52f73fa5720f16288073943f3e"}},"sourceBranch":"main","suggestedTargetBranches":["9.0","8.18","8.x"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/212420","number":212420,"mergeCommit":{"message":"[Security Solution] Refactor prebuilt rule customization test plans, part 3 (elastic#212420)\n\n**Epic:** https://github.com/elastic/kibana/issues/174168\n**Partially addresses:**\nhttps://github.com/elastic/issues/202068,\nhttps://github.com/elastic/issues/202078,\nhttps://github.com/elastic/issues/202079\n**Follow-up to:** https://github.com/elastic/kibana/pull/211472\n\n## Summary\n\nWe're cleaning up and refactoring our existing test plans for prebuilt\nrule customization, upgrade, and export/import workflows.\n\nSpecifically, this PR:\n\n- Updates the common structure of test plans:\n- Adds a tip to each test plan's header to help with navigating all our\ntest plans for prebuilt rules.\n- Introduces a new `Requirements` section to each one and consolidates\nassumptions, technical requirements and product requirements under it.\n- Each `Product requirements` section now contains relevant user stories\nand other product details. In the document flow this is kept close to\nthe `Scenarios` section.\n - Updates the test plan template according to the updated structure.\n- Moves the documentation about customizable and non-customizable rule\nfields to the common info document.\n- Restores the lost changes from commit\nhttps://github.com/elastic/kibana/commit/bde4612e5256b3b779961736291ccae2e328c310.\n\nNo \"functional\" changes have been made to any test plans, such as\nadding, removing, or updating any scenarios. The only functional-like\nchanges were made to the user stories when creating/updating the product\nrequirements sections.\n\nThis PR wraps up the refactoring efforts aimed to prepare the test plans\nfor being \"functionally\" changed and improved in follow-up PRs. In the\nfollowing PRs, we're going to cover the logic described in\nhttps://github.com/elastic/issues/210358, address any gaps in the\ncoverage, and improve the scenarios structure and wording.\n\n## Review tip\n\nIt might be easier to review this PR commit-by-commit as each of them\ncontains logically cohesive changes.","sha":"c9616593c1deeb52f73fa5720f16288073943f3e"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>
banderror
added a commit
that referenced
this issue
Mar 10, 2025
…#212817) **Epic:** #174168 **Partially addresses:** #202078, #210358 ## Summary We started to rework and introduce functional changes to our existing test plans for prebuilt rule customization, upgrade, and export/import workflows. Specifically, this PR: - Creates a new test plan for prebuilt rule upgrade notifications on the Rule Management, Rule Details, and Rule Editing pages. The filename is `prebuilt_rule_upgrade_notifications.md`. - Extracts the existing scenarios for upgrade notifications on the Rule Management page from `prebuilt_rule_upgrade_without_preview.md` to `prebuilt_rule_upgrade_notifications.md`. Also, updates them according to the most recent UI behavior. - Adds new scenarios for upgrade notifications on the Rule Details page to `prebuilt_rule_upgrade_notifications.md`. - Adds new scenarios for upgrade notifications on the Rule Editing page to `prebuilt_rule_upgrade_notifications.md`. The new test plan should be in line with the changes discussed in #210358.
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Mar 10, 2025
…elastic#212817) **Epic:** elastic#174168 **Partially addresses:** elastic#202078, elastic#210358 ## Summary We started to rework and introduce functional changes to our existing test plans for prebuilt rule customization, upgrade, and export/import workflows. Specifically, this PR: - Creates a new test plan for prebuilt rule upgrade notifications on the Rule Management, Rule Details, and Rule Editing pages. The filename is `prebuilt_rule_upgrade_notifications.md`. - Extracts the existing scenarios for upgrade notifications on the Rule Management page from `prebuilt_rule_upgrade_without_preview.md` to `prebuilt_rule_upgrade_notifications.md`. Also, updates them according to the most recent UI behavior. - Adds new scenarios for upgrade notifications on the Rule Details page to `prebuilt_rule_upgrade_notifications.md`. - Adds new scenarios for upgrade notifications on the Rule Editing page to `prebuilt_rule_upgrade_notifications.md`. The new test plan should be in line with the changes discussed in elastic#210358. (cherry picked from commit da0480b)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Mar 10, 2025
…elastic#212817) **Epic:** elastic#174168 **Partially addresses:** elastic#202078, elastic#210358 ## Summary We started to rework and introduce functional changes to our existing test plans for prebuilt rule customization, upgrade, and export/import workflows. Specifically, this PR: - Creates a new test plan for prebuilt rule upgrade notifications on the Rule Management, Rule Details, and Rule Editing pages. The filename is `prebuilt_rule_upgrade_notifications.md`. - Extracts the existing scenarios for upgrade notifications on the Rule Management page from `prebuilt_rule_upgrade_without_preview.md` to `prebuilt_rule_upgrade_notifications.md`. Also, updates them according to the most recent UI behavior. - Adds new scenarios for upgrade notifications on the Rule Details page to `prebuilt_rule_upgrade_notifications.md`. - Adds new scenarios for upgrade notifications on the Rule Editing page to `prebuilt_rule_upgrade_notifications.md`. The new test plan should be in line with the changes discussed in elastic#210358. (cherry picked from commit da0480b)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Mar 10, 2025
…elastic#212817) **Epic:** elastic#174168 **Partially addresses:** elastic#202078, elastic#210358 ## Summary We started to rework and introduce functional changes to our existing test plans for prebuilt rule customization, upgrade, and export/import workflows. Specifically, this PR: - Creates a new test plan for prebuilt rule upgrade notifications on the Rule Management, Rule Details, and Rule Editing pages. The filename is `prebuilt_rule_upgrade_notifications.md`. - Extracts the existing scenarios for upgrade notifications on the Rule Management page from `prebuilt_rule_upgrade_without_preview.md` to `prebuilt_rule_upgrade_notifications.md`. Also, updates them according to the most recent UI behavior. - Adds new scenarios for upgrade notifications on the Rule Details page to `prebuilt_rule_upgrade_notifications.md`. - Adds new scenarios for upgrade notifications on the Rule Editing page to `prebuilt_rule_upgrade_notifications.md`. The new test plan should be in line with the changes discussed in elastic#210358. (cherry picked from commit da0480b)
kibanamachine
added a commit
that referenced
this issue
Mar 10, 2025
…ations (#212817) (#213716) # Backport This will backport the following commits from `main` to `8.x`: - [[Security Solution] Test plan for prebuilt rule upgrade notifications (#212817)](#212817) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Georgii Gorbachev","email":"georgii.gorbachev@elastic.co"},"sourceCommit":{"committedDate":"2025-03-10T11:50:12Z","message":"[Security Solution] Test plan for prebuilt rule upgrade notifications (#212817)\n\n**Epic:** https://github.com/elastic/kibana/issues/174168\n**Partially addresses:**\nhttps://github.com//issues/202078,\nhttps://github.com//issues/210358\n\n## Summary\n\nWe started to rework and introduce functional changes to our existing\ntest plans for prebuilt rule customization, upgrade, and export/import\nworkflows.\n\nSpecifically, this PR:\n\n- Creates a new test plan for prebuilt rule upgrade notifications on the\nRule Management, Rule Details, and Rule Editing pages. The filename is\n`prebuilt_rule_upgrade_notifications.md`.\n- Extracts the existing scenarios for upgrade notifications on the Rule\nManagement page from `prebuilt_rule_upgrade_without_preview.md` to\n`prebuilt_rule_upgrade_notifications.md`. Also, updates them according\nto the most recent UI behavior.\n- Adds new scenarios for upgrade notifications on the Rule Details page\nto `prebuilt_rule_upgrade_notifications.md`.\n- Adds new scenarios for upgrade notifications on the Rule Editing page\nto `prebuilt_rule_upgrade_notifications.md`.\n\nThe new test plan should be in line with the changes discussed in\nhttps://github.com//issues/210358.","sha":"da0480bde922b82741240a2aeeb1c33bd2497c66","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","test-plan","v9.0.0","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Prebuilt Detection Rules","backport:version","v8.18.0","v9.1.0","v8.19.0"],"title":"[Security Solution] Test plan for prebuilt rule upgrade notifications","number":212817,"url":"https://github.com/elastic/kibana/pull/212817","mergeCommit":{"message":"[Security Solution] Test plan for prebuilt rule upgrade notifications (#212817)\n\n**Epic:** https://github.com/elastic/kibana/issues/174168\n**Partially addresses:**\nhttps://github.com//issues/202078,\nhttps://github.com//issues/210358\n\n## Summary\n\nWe started to rework and introduce functional changes to our existing\ntest plans for prebuilt rule customization, upgrade, and export/import\nworkflows.\n\nSpecifically, this PR:\n\n- Creates a new test plan for prebuilt rule upgrade notifications on the\nRule Management, Rule Details, and Rule Editing pages. The filename is\n`prebuilt_rule_upgrade_notifications.md`.\n- Extracts the existing scenarios for upgrade notifications on the Rule\nManagement page from `prebuilt_rule_upgrade_without_preview.md` to\n`prebuilt_rule_upgrade_notifications.md`. Also, updates them according\nto the most recent UI behavior.\n- Adds new scenarios for upgrade notifications on the Rule Details page\nto `prebuilt_rule_upgrade_notifications.md`.\n- Adds new scenarios for upgrade notifications on the Rule Editing page\nto `prebuilt_rule_upgrade_notifications.md`.\n\nThe new test plan should be in line with the changes discussed in\nhttps://github.com//issues/210358.","sha":"da0480bde922b82741240a2aeeb1c33bd2497c66"}},"sourceBranch":"main","suggestedTargetBranches":["9.0","8.18","8.x"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/212817","number":212817,"mergeCommit":{"message":"[Security Solution] Test plan for prebuilt rule upgrade notifications (#212817)\n\n**Epic:** https://github.com/elastic/kibana/issues/174168\n**Partially addresses:**\nhttps://github.com//issues/202078,\nhttps://github.com//issues/210358\n\n## Summary\n\nWe started to rework and introduce functional changes to our existing\ntest plans for prebuilt rule customization, upgrade, and export/import\nworkflows.\n\nSpecifically, this PR:\n\n- Creates a new test plan for prebuilt rule upgrade notifications on the\nRule Management, Rule Details, and Rule Editing pages. The filename is\n`prebuilt_rule_upgrade_notifications.md`.\n- Extracts the existing scenarios for upgrade notifications on the Rule\nManagement page from `prebuilt_rule_upgrade_without_preview.md` to\n`prebuilt_rule_upgrade_notifications.md`. Also, updates them according\nto the most recent UI behavior.\n- Adds new scenarios for upgrade notifications on the Rule Details page\nto `prebuilt_rule_upgrade_notifications.md`.\n- Adds new scenarios for upgrade notifications on the Rule Editing page\nto `prebuilt_rule_upgrade_notifications.md`.\n\nThe new test plan should be in line with the changes discussed in\nhttps://github.com//issues/210358.","sha":"da0480bde922b82741240a2aeeb1c33bd2497c66"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>
kibanamachine
added a commit
that referenced
this issue
Mar 10, 2025
…ations (#212817) (#213717) # Backport This will backport the following commits from `main` to `9.0`: - [[Security Solution] Test plan for prebuilt rule upgrade notifications (#212817)](#212817) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Georgii Gorbachev","email":"georgii.gorbachev@elastic.co"},"sourceCommit":{"committedDate":"2025-03-10T11:50:12Z","message":"[Security Solution] Test plan for prebuilt rule upgrade notifications (#212817)\n\n**Epic:** https://github.com/elastic/kibana/issues/174168\n**Partially addresses:**\nhttps://github.com//issues/202078,\nhttps://github.com//issues/210358\n\n## Summary\n\nWe started to rework and introduce functional changes to our existing\ntest plans for prebuilt rule customization, upgrade, and export/import\nworkflows.\n\nSpecifically, this PR:\n\n- Creates a new test plan for prebuilt rule upgrade notifications on the\nRule Management, Rule Details, and Rule Editing pages. The filename is\n`prebuilt_rule_upgrade_notifications.md`.\n- Extracts the existing scenarios for upgrade notifications on the Rule\nManagement page from `prebuilt_rule_upgrade_without_preview.md` to\n`prebuilt_rule_upgrade_notifications.md`. Also, updates them according\nto the most recent UI behavior.\n- Adds new scenarios for upgrade notifications on the Rule Details page\nto `prebuilt_rule_upgrade_notifications.md`.\n- Adds new scenarios for upgrade notifications on the Rule Editing page\nto `prebuilt_rule_upgrade_notifications.md`.\n\nThe new test plan should be in line with the changes discussed in\nhttps://github.com//issues/210358.","sha":"da0480bde922b82741240a2aeeb1c33bd2497c66","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","test-plan","v9.0.0","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Prebuilt Detection Rules","backport:version","v8.18.0","v9.1.0","v8.19.0"],"title":"[Security Solution] Test plan for prebuilt rule upgrade notifications","number":212817,"url":"https://github.com/elastic/kibana/pull/212817","mergeCommit":{"message":"[Security Solution] Test plan for prebuilt rule upgrade notifications (#212817)\n\n**Epic:** https://github.com/elastic/kibana/issues/174168\n**Partially addresses:**\nhttps://github.com//issues/202078,\nhttps://github.com//issues/210358\n\n## Summary\n\nWe started to rework and introduce functional changes to our existing\ntest plans for prebuilt rule customization, upgrade, and export/import\nworkflows.\n\nSpecifically, this PR:\n\n- Creates a new test plan for prebuilt rule upgrade notifications on the\nRule Management, Rule Details, and Rule Editing pages. The filename is\n`prebuilt_rule_upgrade_notifications.md`.\n- Extracts the existing scenarios for upgrade notifications on the Rule\nManagement page from `prebuilt_rule_upgrade_without_preview.md` to\n`prebuilt_rule_upgrade_notifications.md`. Also, updates them according\nto the most recent UI behavior.\n- Adds new scenarios for upgrade notifications on the Rule Details page\nto `prebuilt_rule_upgrade_notifications.md`.\n- Adds new scenarios for upgrade notifications on the Rule Editing page\nto `prebuilt_rule_upgrade_notifications.md`.\n\nThe new test plan should be in line with the changes discussed in\nhttps://github.com//issues/210358.","sha":"da0480bde922b82741240a2aeeb1c33bd2497c66"}},"sourceBranch":"main","suggestedTargetBranches":["9.0","8.18","8.x"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/212817","number":212817,"mergeCommit":{"message":"[Security Solution] Test plan for prebuilt rule upgrade notifications (#212817)\n\n**Epic:** https://github.com/elastic/kibana/issues/174168\n**Partially addresses:**\nhttps://github.com//issues/202078,\nhttps://github.com//issues/210358\n\n## Summary\n\nWe started to rework and introduce functional changes to our existing\ntest plans for prebuilt rule customization, upgrade, and export/import\nworkflows.\n\nSpecifically, this PR:\n\n- Creates a new test plan for prebuilt rule upgrade notifications on the\nRule Management, Rule Details, and Rule Editing pages. The filename is\n`prebuilt_rule_upgrade_notifications.md`.\n- Extracts the existing scenarios for upgrade notifications on the Rule\nManagement page from `prebuilt_rule_upgrade_without_preview.md` to\n`prebuilt_rule_upgrade_notifications.md`. Also, updates them according\nto the most recent UI behavior.\n- Adds new scenarios for upgrade notifications on the Rule Details page\nto `prebuilt_rule_upgrade_notifications.md`.\n- Adds new scenarios for upgrade notifications on the Rule Editing page\nto `prebuilt_rule_upgrade_notifications.md`.\n\nThe new test plan should be in line with the changes discussed in\nhttps://github.com//issues/210358.","sha":"da0480bde922b82741240a2aeeb1c33bd2497c66"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>
kibanamachine
added a commit
that referenced
this issue
Mar 10, 2025
…cations (#212817) (#213715) # Backport This will backport the following commits from `main` to `8.18`: - [[Security Solution] Test plan for prebuilt rule upgrade notifications (#212817)](#212817) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Georgii Gorbachev","email":"georgii.gorbachev@elastic.co"},"sourceCommit":{"committedDate":"2025-03-10T11:50:12Z","message":"[Security Solution] Test plan for prebuilt rule upgrade notifications (#212817)\n\n**Epic:** https://github.com/elastic/kibana/issues/174168\n**Partially addresses:**\nhttps://github.com//issues/202078,\nhttps://github.com//issues/210358\n\n## Summary\n\nWe started to rework and introduce functional changes to our existing\ntest plans for prebuilt rule customization, upgrade, and export/import\nworkflows.\n\nSpecifically, this PR:\n\n- Creates a new test plan for prebuilt rule upgrade notifications on the\nRule Management, Rule Details, and Rule Editing pages. The filename is\n`prebuilt_rule_upgrade_notifications.md`.\n- Extracts the existing scenarios for upgrade notifications on the Rule\nManagement page from `prebuilt_rule_upgrade_without_preview.md` to\n`prebuilt_rule_upgrade_notifications.md`. Also, updates them according\nto the most recent UI behavior.\n- Adds new scenarios for upgrade notifications on the Rule Details page\nto `prebuilt_rule_upgrade_notifications.md`.\n- Adds new scenarios for upgrade notifications on the Rule Editing page\nto `prebuilt_rule_upgrade_notifications.md`.\n\nThe new test plan should be in line with the changes discussed in\nhttps://github.com//issues/210358.","sha":"da0480bde922b82741240a2aeeb1c33bd2497c66","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","test-plan","v9.0.0","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Prebuilt Detection Rules","backport:version","v8.18.0","v9.1.0","v8.19.0"],"title":"[Security Solution] Test plan for prebuilt rule upgrade notifications","number":212817,"url":"https://github.com/elastic/kibana/pull/212817","mergeCommit":{"message":"[Security Solution] Test plan for prebuilt rule upgrade notifications (#212817)\n\n**Epic:** https://github.com/elastic/kibana/issues/174168\n**Partially addresses:**\nhttps://github.com//issues/202078,\nhttps://github.com//issues/210358\n\n## Summary\n\nWe started to rework and introduce functional changes to our existing\ntest plans for prebuilt rule customization, upgrade, and export/import\nworkflows.\n\nSpecifically, this PR:\n\n- Creates a new test plan for prebuilt rule upgrade notifications on the\nRule Management, Rule Details, and Rule Editing pages. The filename is\n`prebuilt_rule_upgrade_notifications.md`.\n- Extracts the existing scenarios for upgrade notifications on the Rule\nManagement page from `prebuilt_rule_upgrade_without_preview.md` to\n`prebuilt_rule_upgrade_notifications.md`. Also, updates them according\nto the most recent UI behavior.\n- Adds new scenarios for upgrade notifications on the Rule Details page\nto `prebuilt_rule_upgrade_notifications.md`.\n- Adds new scenarios for upgrade notifications on the Rule Editing page\nto `prebuilt_rule_upgrade_notifications.md`.\n\nThe new test plan should be in line with the changes discussed in\nhttps://github.com//issues/210358.","sha":"da0480bde922b82741240a2aeeb1c33bd2497c66"}},"sourceBranch":"main","suggestedTargetBranches":["9.0","8.18","8.x"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/212817","number":212817,"mergeCommit":{"message":"[Security Solution] Test plan for prebuilt rule upgrade notifications (#212817)\n\n**Epic:** https://github.com/elastic/kibana/issues/174168\n**Partially addresses:**\nhttps://github.com//issues/202078,\nhttps://github.com//issues/210358\n\n## Summary\n\nWe started to rework and introduce functional changes to our existing\ntest plans for prebuilt rule customization, upgrade, and export/import\nworkflows.\n\nSpecifically, this PR:\n\n- Creates a new test plan for prebuilt rule upgrade notifications on the\nRule Management, Rule Details, and Rule Editing pages. The filename is\n`prebuilt_rule_upgrade_notifications.md`.\n- Extracts the existing scenarios for upgrade notifications on the Rule\nManagement page from `prebuilt_rule_upgrade_without_preview.md` to\n`prebuilt_rule_upgrade_notifications.md`. Also, updates them according\nto the most recent UI behavior.\n- Adds new scenarios for upgrade notifications on the Rule Details page\nto `prebuilt_rule_upgrade_notifications.md`.\n- Adds new scenarios for upgrade notifications on the Rule Editing page\nto `prebuilt_rule_upgrade_notifications.md`.\n\nThe new test plan should be in line with the changes discussed in\nhttps://github.com//issues/210358.","sha":"da0480bde922b82741240a2aeeb1c33bd2497c66"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>
27 tasks
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this issue
Mar 22, 2025
…eview (Rule Updates table) v2 (elastic#205922) **Addresses:** elastic#202078 ## Summary This PR extends rule upgrade test plan with customizable and non-customizable field examples. Rule upgrade workflow test plan (excluding Rule Upgrade flyout) was initially extended in elastic#203331. elastic#203331 adds the following rule upgrade workflow scenarios - Scenario: User can upgrade conflict-free prebuilt rules one by one - Scenario: User cannot upgrade prebuilt rules one by one from Rules Update table if they have conflicts - Scenario: User can upgrade multiple conflict-free prebuilt rules selected on the page - Scenario: User cannot upgrade selected prebuilt rules with conflicts - Scenario: User can upgrade all available conflict-free prebuilt rules at once* - Scenario: User cannot upgrade all prebuilt rules at once if they have upgrade conflicts - Scenario: User can upgrade only conflict-free rules when a mix of rules with and without conflicts are selected for upgrade - Scenario: User can upgrade only conflict-free rules when attempting to upgrade all rules - Scenario: User can upgrade rule with rule type change individually - Scenario: User can not bulk upgrade selected rules with rule type changes - Scenario: User can not bulk upgrade all rules with rule type changes - Scenario: API does not upgrade prebuilt rules if they are up to date --------- Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this issue
Mar 22, 2025
…ew (Rule Upgrade flyout) v1 (elastic#205645) **Epics:** elastic/security-team#1974 (internal), elastic#174168 **Addresses:** elastic#202078 ## Summary This PR adds a test plan for rule upgrade flyout. --------- Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this issue
Mar 22, 2025
…part 1 (elastic#211300) **Epic:** elastic#174168 **Partially addresses:** elastic#202068, elastic#202078, elastic#202079 ## Summary We're cleaning up and refactoring our existing test plans for prebuilt rule customization, upgrade, and export/import workflows. Specifically, in this PR we're consistently: - Changing the file names and test plan titles. - Adding or expanding test plan summaries. - Regenerating tables of contents using the [Markdown All in One](https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one) VS Code extension. No "functional" changes have been made to any test plans, such as adding, removing, or updating any scenarios. This refactoring prepares the test plans for being "functionally" changed and improved in follow-up PRs. For example, we're going to cover the logic described in elastic#210358 and address any other gaps in coverage.
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this issue
Mar 22, 2025
…part 2 (elastic#211472) **Epic:** elastic#174168 **Partially addresses:** elastic#202068, elastic#202078, elastic#202079 **Follow-up to:** elastic#211300 ## Summary We're cleaning up and refactoring our existing test plans for prebuilt rule customization, upgrade, and export/import workflows. Specifically, this PR: - Creates an "entrypoint" file that should help navigate all the test plans for prebuilt rules. - Creates a file for keeping common information about prebuilt rules that can be shared between the test plans. - Extracts duplicated terminology to the file with common information. - Extracts duplicated assumptions to the file with common information. - Extracts duplicated non-functional requirements to the file with common information. - Adds user stories to each test plan. - Updates links to tickets in every test plan. No "functional" changes have been made to any test plans, such as adding, removing, or updating any scenarios. This refactoring prepares the test plans for being "functionally" changed and improved in follow-up PRs. For example, we're going to cover the logic described in elastic#210358 and address any other gaps in coverage. ## Review tip It might be easier to review this PR commit-by-commit as each of them contains logically cohesive changes.
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this issue
Mar 22, 2025
…t rule customization feature (elastic#209260) **Addresses:** elastic#202078 **Partially implements test plan:** elastic#205922 ## Summary This PR implements the Rule Upgrade test plan added in elastic#205922 and elastic#203331. ## Details Tests cover only enabled customization and organized in a following way - `diffable_rule_fields` folder contains per field tests. It focuses on `DiffableRule`* fields. Each field has rule preview and upgrade scenarios throughly covered. Assertion on stats are included in rule preview tests to make sure the maximum coverage with reasonable execution time. **Diffable rule fields` tests were moved to a separate execution group to avoid exceeding execution limits.** - `preview_prebuilt_rules_upgrade.ts` integration tests on `/internal/prebuilt_rules/upgrade/_review` endpoint. It uses only `name` and `tags` fields to minimize tests complexity. Basically it should be considered as smoke tests. - `upgrade_prebuilt_rules` integrations test on `/internal/prebuilt_rules/upgrade/_perform` endpoint. It uses only `name` and `tags` fields to minimize tests complexity. Basically it should be considered as smoke tests. This PR supersedes elastic#205217 and elastic#205339. * `DiffableRule` was added in the scope of prebuilt rules customization epic to serve rule upgrades preview and performing upgrades. It represents slightly reorganized rule fields to simplify prebuilt rule upgrade workflow handling. There are utility functions transforming between `RuleResponse` and `DiffableRule`. ## Flaky test runner Flaky test runs for 100 iterations were successful (elastic#209260 (comment) and elastic#209260 (comment)).
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this issue
Mar 22, 2025
…part 3 (elastic#212420) **Epic:** elastic#174168 **Partially addresses:** elastic#202068, elastic#202078, elastic#202079 **Follow-up to:** elastic#211472 ## Summary We're cleaning up and refactoring our existing test plans for prebuilt rule customization, upgrade, and export/import workflows. Specifically, this PR: - Updates the common structure of test plans: - Adds a tip to each test plan's header to help with navigating all our test plans for prebuilt rules. - Introduces a new `Requirements` section to each one and consolidates assumptions, technical requirements and product requirements under it. - Each `Product requirements` section now contains relevant user stories and other product details. In the document flow this is kept close to the `Scenarios` section. - Updates the test plan template according to the updated structure. - Moves the documentation about customizable and non-customizable rule fields to the common info document. - Restores the lost changes from commit elastic@bde4612. No "functional" changes have been made to any test plans, such as adding, removing, or updating any scenarios. The only functional-like changes were made to the user stories when creating/updating the product requirements sections. This PR wraps up the refactoring efforts aimed to prepare the test plans for being "functionally" changed and improved in follow-up PRs. In the following PRs, we're going to cover the logic described in elastic#210358, address any gaps in the coverage, and improve the scenarios structure and wording. ## Review tip It might be easier to review this PR commit-by-commit as each of them contains logically cohesive changes.
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this issue
Mar 22, 2025
…elastic#212817) **Epic:** elastic#174168 **Partially addresses:** elastic#202078, elastic#210358 ## Summary We started to rework and introduce functional changes to our existing test plans for prebuilt rule customization, upgrade, and export/import workflows. Specifically, this PR: - Creates a new test plan for prebuilt rule upgrade notifications on the Rule Management, Rule Details, and Rule Editing pages. The filename is `prebuilt_rule_upgrade_notifications.md`. - Extracts the existing scenarios for upgrade notifications on the Rule Management page from `prebuilt_rule_upgrade_without_preview.md` to `prebuilt_rule_upgrade_notifications.md`. Also, updates them according to the most recent UI behavior. - Adds new scenarios for upgrade notifications on the Rule Details page to `prebuilt_rule_upgrade_notifications.md`. - Adds new scenarios for upgrade notifications on the Rule Editing page to `prebuilt_rule_upgrade_notifications.md`. The new test plan should be in line with the changes discussed in elastic#210358.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Epics: https://github.com/elastic/security-team/issues/1974 (internal), #174168
Summary
Write test plans (one or a few) and create a comprehensive suite of automated tests for the workflow of upgrading prebuilt rules.
Features to cover:
Please cover both the features under the feature flag turned ON and OFF.
Related tickets
Related functional tickets to cover with tests:
POST /upgrade/_reviewAPI endpoint's contract and functionality #180153/upgrade/_reviewupgrade workflow #180393POST /upgrade/_performAPI endpoint's contract and functionality #166376/upgrade/_performendpoint upgrade workflow #186544exceptions_list,authorandlicensefrom Diffable Rule #196213Related bugs to cover with tests:
baseversion #200904Related PRs
Test plans for diff algorithms
data_sourcefield diff algorithm test plan #189669queryfields diff algorithm #192529typefield diff algorithm #193372Test plans for prebuilt rule upgrade workflow
Test coverage for prebuilt rule upgrade workflow
The text was updated successfully, but these errors were encountered: