-
Notifications
You must be signed in to change notification settings - Fork 22
Search tests #721
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Search tests #721
Changes from all commits
658fb4d
153ff07
e981db1
86f60d0
1143e4a
5baa9f8
90ca7e8
ebefa2c
c3b58a6
29b0c74
41a6ead
115fbe1
d506a7b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| Feature: Search | ||
| As a Devsecops Engineer | ||
| I want to perform searching across vulnerabilities, SBOMs and packages, specific searches for CVE IDs, SBOM titles, package names and show results that are easy to navigate to the specific item of interest. | ||
|
|
||
| Background: | ||
| Given User is authenticated | ||
| And User is on the Search page | ||
|
|
||
| Scenario: User visits search page without filling anything | ||
| Then a total number of 17 "SBOMs" should be visible in the tab | ||
| And a total number of 5537 "Packages" should be visible in the tab | ||
| And a total number of 29 "Vulnerabilities" should be visible in the tab | ||
| And a total number of 57 "Advisories" should be visible in the tab | ||
|
|
||
| Scenario Outline: User toggles the "<types>" list and manipulates the list | ||
| When User selects the Tab "<types>" | ||
| Then the "<types>" list should have specific filter set | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. In my opinion, the wording of this step is not clear. We could make this step more useful (and reusable), if the step was called something like |
||
| And the "<types>" list should be sortable | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We should probably make sure each column is sortable. BUT I think that this may be redundant, as this is already being tested here.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Though on second thought it may be a good idea to still test this, as BDD scenarios are likely to run separately from the tests written by Carlos, but the test step could re-use the methods defined in his page objects.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. To be fair, I am kind of inclined to rewriting this, so the test just checks relevant I am currently not sure, whether the sorting is performed on frontend or backend, burt I feel like it should be tested either with some sort of React component test, or an API test. I tried to test it using an E2E test, but I have not been able to make it not flake in the Github CI.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The methods |
||
| And the "<types>" list should be limited to 10 items | ||
| And the user should be able to switch to next "<types>" items | ||
| And the user should be able to increase pagination for the "<types>" | ||
|
Comment on lines
+19
to
+21
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. All these steps seems to Pagination related which can be verified with a single step definition using |
||
| And First column on the search results should have the link to "<types>" explorer pages | ||
|
|
||
| Examples: | ||
| |types| | ||
| |SBOMs| | ||
| # |Packages| | ||
| |Vulnerabilities| | ||
| |Advisories| | ||
|
|
||
| Scenario Outline: Download Links on the "<types>" Search Result list | ||
| When User selects the Tab "<types>" | ||
| Then Tab "<types>" is visible | ||
| And Download link should be available for the "<types>" list | ||
|
|
||
| Examples: | ||
| |types| | ||
| |SBOMs| | ||
| |Advisories| | ||
|
|
||
| Scenario Outline: Autofill shows results matched on <input> | ||
| When user starts typing a "<input>" in the search bar | ||
| Then the autofill dropdown should display items matching the "<input>" | ||
| And the results should be limited to 5 suggestions | ||
|
|
||
| Examples: | ||
| |input| | ||
| |quarkus| | ||
| |CVE-2022| | ||
| |policies| | ||
|
|
||
| Scenario: Search bar should not preview anything when no matches are found | ||
| And user starts typing a "non-existent name" in the search bar | ||
| Then The autofill drop down should not show any values | ||
|
|
||
| Scenario Outline: User searches for a specific "<type>" | ||
| When user types a "<type-instance>" in the search bar | ||
| And user presses Enter | ||
| And User selects the Tab "<types>" | ||
| Then the "<types>" list should display the specific "<type-instance>" | ||
| And the list should be limited to 10 items or less | ||
| And the user should be able to filter "<types>" | ||
| And user clicks on the "<type-instance>" "<type>" link | ||
| And the user should be navigated to the specific "<type-instance>" page | ||
|
|
||
| Examples: | ||
| |type|types|type-instance| | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can we have just type or types? |
||
| |SBOM|SBOMs|quarkus-bom| | ||
| |CVE|Vulnerabilities|CVE-2022-45787| | ||
| |Package|Packages|quarkus| | ||
| |Advisory|Advisories|CVE-2022-45787| | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,275 @@ | ||
| import { SearchPage } from "../../helpers/SearchPage"; | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. With Carlos' page object PR merged, we should probably consider these classes outdated and use the ones from that merge request if possible to avoid redundancy. |
||
| import { ToolbarTable } from "../../helpers/ToolbarTable"; | ||
| import { Tabs } from "../../helpers/Tabs"; | ||
| import { DetailsPage } from "../../helpers/DetailsPage"; | ||
| import { createBdd } from "playwright-bdd"; | ||
| import { expect } from "@playwright/test"; | ||
|
|
||
| export const { Given, When, Then } = createBdd(); | ||
|
|
||
| /** | ||
| * This function returns table identifier and column, which contains link to the details page | ||
| * @param type Catogory of the data to get the table identifier and column | ||
| */ | ||
| function getTableInfo(type: string): [string, string] { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Helper functions should not be defined in the step definition file. Please use the methods defined for the page objects as mentioned above, if applicable, and if you have to extend them, please do it there.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agree |
||
| switch (type) { | ||
| case "SBOMs": | ||
| case "SBOM": | ||
| return ["sbom-table", "Name"]; | ||
| case "Advisories": | ||
| case "Advisory": | ||
| return ["advisory-table", "ID"]; | ||
| case "Vulnerabilities": | ||
| case "CVE": | ||
| return ["Vulnerability table", "ID"]; | ||
| case "Packages": | ||
| case "Package": | ||
| return ["Package table", "Name"]; | ||
| default: | ||
| throw new Error(`Unknown type: ${type}`); | ||
| } | ||
| } | ||
|
|
||
| function getPaginationId(type: string): string { | ||
| switch (type) { | ||
| case "Vulnerabilities": | ||
| return "vulnerability-table-pagination-top"; | ||
| case "Advisories": | ||
| return "advisory-table-pagination-top"; | ||
| case "Packages": | ||
| return "package-table-pagination-top"; | ||
| case "SBOMs": | ||
| return "sbom-table-pagination-top"; | ||
| default: | ||
| throw new Error(`Unknown type: ${type}`); | ||
| } | ||
| } | ||
|
|
||
| function getColumns(type: string): string[] { | ||
| switch (type) { | ||
| case "Vulnerabilities": | ||
| return ["ID", "CVSS", "Date published"]; | ||
| case "Advisories": | ||
| return ["ID", "Revision"]; | ||
| case "Packages": | ||
| return ["Name", "Namespace", "Version"]; | ||
| case "SBOMs": | ||
| return ["Name", "Created on"]; | ||
| default: | ||
| throw new Error(`Unknown type: ${type}`); | ||
| } | ||
| } | ||
|
|
||
| Given("User is on the Search page", async ({ page }) => { | ||
| const searchPage = new SearchPage(page); | ||
| await searchPage.open(); | ||
| }); | ||
|
|
||
| Then( | ||
| "Download link should be available for the {string} list", | ||
| async ({ page }, type: string) => { | ||
| const table = new ToolbarTable(page, getTableInfo(type)[0]); | ||
| await table.verifyDownloadLink(type); | ||
| }, | ||
| ); | ||
|
|
||
| When( | ||
| "user starts typing a {string} in the search bar", | ||
| async ({ page }, searchText: string) => { | ||
| const searchPage = new SearchPage(page); | ||
| await searchPage.typeInSearchBox(searchText); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm pretty sure methods for something like this already exist in the repo. Again, please, let's avoid redundant code and use what can be reused. |
||
| }, | ||
| ); | ||
|
|
||
| Then("The autofill drop down should not show any values", async ({ page }) => { | ||
| const searchPage = new SearchPage(page); | ||
| await searchPage.autoFillIsNotVisible(); | ||
| }); | ||
|
|
||
| When( | ||
| "user types a {string} in the search bar", | ||
| async ({ page }, searchText: string) => { | ||
| const searchPage = new SearchPage(page); | ||
| await searchPage.typeInSearchBox(searchText); | ||
| }, | ||
| ); | ||
|
|
||
| When("user presses Enter", async ({ page }) => { | ||
| await page.keyboard.press("Enter"); | ||
| }); | ||
|
|
||
| Then( | ||
| "the {string} list should display the specific {string}", | ||
| async ({ page }, type: string, name: string) => { | ||
| const tabs = new Tabs(page); | ||
| await tabs.verifyTabIsSelected(type); | ||
| const info = getTableInfo(type); | ||
| const table = new ToolbarTable(page, info[0]); | ||
| await table.verifyColumnContainsText(info[1], name); | ||
| }, | ||
| ); | ||
|
|
||
| Then( | ||
| "the list should be limited to {int} items or less", | ||
| async ({ page }, count: number) => { | ||
| const table = new ToolbarTable(page, "sbom-table"); | ||
| await table.verifyTableHasUpToRows(count); | ||
| }, | ||
| ); | ||
|
|
||
| Then( | ||
| "user clicks on the {string} {string} link", | ||
| async ({ page }, arg: string, type: string) => { | ||
| const info = getTableInfo(type); | ||
| const table = new ToolbarTable(page, info[0]); | ||
| await table.openDetailsPage(arg, info[1]); | ||
| }, | ||
| ); | ||
|
|
||
| Then( | ||
| "the user should be navigated to the specific {string} page", | ||
| async ({ page }, arg: string) => { | ||
| const detailsPage = new DetailsPage(page); | ||
| await detailsPage.verifyPageHeader(arg); | ||
| }, | ||
| ); | ||
|
|
||
| Then( | ||
| "the user should be able to filter {string}", | ||
| async ({ page }, arg: string) => { | ||
| const table = new ToolbarTable(page, getTableInfo(arg)[0]); | ||
| if (arg === "SBOMs") { | ||
| await table.filterByDate("12/22/2025", "12/22/2025"); | ||
| await table.verifyColumnDoesNotContainText("Name", "quarkus-bom"); | ||
| await table.clearFilter(); | ||
| await table.verifyColumnContainsText("Name", "quarkus-bom"); | ||
| } else if (arg === "Vulnerabilities") { | ||
| await page.getByLabel("Critical").click(); | ||
| await table.verifyColumnDoesNotContainText("ID", "CVE-2022-45787"); | ||
| await table.clearFilter(); | ||
| await table.verifyColumnContainsText("ID", "CVE-2022-45787"); | ||
| } else if (arg === "Packages") { | ||
| await page.getByLabel("OCI").click(); | ||
| await table.verifyColumnDoesNotContainText("Name", "quarkus"); | ||
| await table.clearFilter(); | ||
| await table.verifyColumnContainsText("Name", "quarkus"); | ||
| } else if (arg === "Advisories") { | ||
| await table.filterByDate("12/22/2025", "12/22/2025"); | ||
| await table.verifyColumnDoesNotContainText("ID", "CVE-2022-45787"); | ||
| await table.clearFilter(); | ||
| await table.verifyColumnContainsText("ID", "CVE-2022-45787"); | ||
|
Comment on lines
+142
to
+160
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. All these values should be from Examples section of the feature file - Never hard code data into the low level or mid level keywords. |
||
| } | ||
| }, | ||
| ); | ||
|
|
||
| Then( | ||
| "the {string} list should have specific filter set", | ||
| async ({ page }, arg: string) => { | ||
| if (arg === "Vulnerabilities") { | ||
| await expect(page.locator("h4").getByText("CVSS")).toBeVisible(); | ||
| await expect(page.locator("h4").getByText("Created on")).toBeVisible(); | ||
| await expect( | ||
| page.locator('input[aria-label="Interval start"]'), | ||
| ).toBeVisible(); | ||
| await expect( | ||
| page.locator('input[aria-label="Interval end"]'), | ||
| ).toBeVisible(); | ||
| } else if (arg === "Advisories") { | ||
| await expect(page.locator("h4").getByText("Revision")).toBeVisible(); | ||
| await expect( | ||
| page.locator('input[aria-label="Interval start"]'), | ||
| ).toBeVisible(); | ||
| await expect( | ||
| page.locator('input[aria-label="Interval end"]'), | ||
| ).toBeVisible(); | ||
| } else if (arg === "Packages") { | ||
| await expect(page.getByRole("heading", { name: "Type" })).toBeVisible(); | ||
| await expect( | ||
| page.getByRole("heading", { name: "Architecture" }), | ||
| ).toBeVisible(); | ||
| } else if (arg === "SBOMs") { | ||
| await expect(page.getByText("Created onFrom To")).toBeVisible(); | ||
| } | ||
| }, | ||
| ); | ||
|
|
||
| Then("the {string} list should be sortable", async ({ page }, arg: string) => { | ||
| var columns: string[] = getColumns(arg); | ||
| var id: string = getPaginationId(arg); | ||
|
|
||
| const table = new ToolbarTable(page, getTableInfo(arg)[0]); | ||
| await table.verifySorting(`xpath=//div[@id="${id}"]`, columns); | ||
| }); | ||
|
|
||
| Then( | ||
| "the {string} list should be limited to {int} items", | ||
| async ({ page }, type: string, count: number) => { | ||
| const info = getTableInfo(type); | ||
| const table = new ToolbarTable(page, info[0]); | ||
| const tableTopPagination = `xpath=//div[@id="${getPaginationId(type)}"]`; | ||
| await table.selectPerPage(tableTopPagination, "10 per page"); | ||
| await table.verifyTableHasUpToRows(count); | ||
| }, | ||
| ); | ||
|
|
||
| Then( | ||
| "the user should be able to switch to next {string} items", | ||
| async ({ page }, arg: string) => { | ||
| var id: string = getPaginationId(arg); | ||
| const info = getTableInfo(arg); | ||
| const table = new ToolbarTable(page, info[0]); | ||
| await table.verifyPagination(`xpath=//div[@id="${id}"]`); | ||
| }, | ||
| ); | ||
|
|
||
| Then( | ||
| "the user should be able to increase pagination for the {string}", | ||
| async ({ page }, arg: string) => { | ||
| const info = getTableInfo(arg); | ||
| const table = new ToolbarTable(page, info[0]); | ||
| var id: string = getPaginationId(arg); | ||
| const tableTopPagination = `xpath=//div[@id="${id}"]`; | ||
| await table.verifyPagination(`xpath=//div[@id="${id}"]`); | ||
| await table.goToFirstPage(tableTopPagination); | ||
| await table.selectPerPage(tableTopPagination, "20 per page"); | ||
| await table.goToFirstPage(tableTopPagination); | ||
| await table.verifyTableHasUpToRows(20); | ||
|
Comment on lines
+226
to
+236
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This step seems to be redundant as "the user should be able to switch to next {string} items" has table.verifyPagination() method to verify the same. |
||
| }, | ||
| ); | ||
|
|
||
| Then( | ||
| "First column on the search results should have the link to {string} explorer pages", | ||
| async ({ page }, arg: string) => { | ||
| const info = getTableInfo(arg); | ||
| const table = new ToolbarTable(page, info[0]); | ||
| await table.verifyColumnContainsLink(info[1], arg); | ||
| }, | ||
| ); | ||
|
|
||
| Then( | ||
| "a total number of {int} {string} should be visible in the tab", | ||
| async ({ page }, count: number, arg: string) => { | ||
| await page.waitForLoadState("networkidle"); | ||
| const tabs = new Tabs(page); | ||
| await tabs.verifyTabHasAtLeastResults(arg, count); | ||
| }, | ||
| ); | ||
|
|
||
| Then( | ||
| "the autofill dropdown should display items matching the {string}", | ||
| async ({ page }, arg: string) => { | ||
| const searchPage = new SearchPage(page); | ||
| await searchPage.autoFillHasRelevantResults(arg); | ||
| }, | ||
| ); | ||
|
|
||
| Then( | ||
| "the results should be limited to {int} suggestions", | ||
| async ({ page }, arg: number) => { | ||
| const searchPage = new SearchPage(page); | ||
| expect(await searchPage.totalAutoFillResults()).toBeLessThanOrEqual( | ||
| arg * 4, | ||
| ); | ||
| await searchPage.expectCategoriesWithinLimitByHref(arg); | ||
| }, | ||
| ); | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We definitely shouldn't hardcode numbers of SBOMs, packages etc. This would make the test suite annoying to maintain, as we add more SBOMs with new tests. I'm not sure if this scenario should be used at all, to be honest. The only way I can imagine this could be useful is if we did an API check for number of SBOMs, packages etc. and compared it with the numbers shown on the search page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree. I asked about this before during our daily sync and I was told to hard-code it for now and then later re-write it using the API. The comparison is also not strict, meaning that there should be at least a given number of SBOMs, packages, etc, but a bigger number is also tolerated.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@matejnesuta Can you please parameterize these values with
Examplessection? something like this,