pfSense & OPNsense: test skipping with ENV ID #1048
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
philipanda
reviewed
Aug 29, 2025
macpijan
reviewed
Aug 29, 2025
philipanda
reviewed
Aug 29, 2025
philipanda
approved these changes
Aug 29, 2025
filipleple
approved these changes
Sep 2, 2025
filipleple
force-pushed
the
os_bsd_flags0000
branch
from
e8f0ad5 to
db6d7d1
Compare
dasharo-compatibility/os-opnsense.robot: dasharo-compatibility/os-pfsense.robot: Test suites are skipping in Suite Setup if BSD ENV IDs are not present in TESTED_LINUX_DISTROS variable. Signed-off-by: Mateusz Maciejewski <[email protected]>
New "tested distros" variable introduced to skip BSD tests (pfSense & OPN sense). Empty list defined in default.robot. Signed-off-by: Mateusz Maciejewski <[email protected]>
Tag semiauto added to installer loading test cases *001.* and to pfSense preseeding test case (PFS006.502). Signed-off-by: Mateusz Maciejewski <[email protected]>
Signed-off-by: Mateusz Maciejewski <[email protected]>
and TESTED_BSD_DISTROS populated for Protectli platforms. Signed-off-by: Mateusz Maciejewski <[email protected]>
matmacieje
force-pushed
the
os_bsd_flags0000
branch
from
db6d7d1 to
3caac45
Compare
macpijan
approved these changes
Sep 19, 2025
philipanda
requested changes
Sep 19, 2025
| "_id": "OPN001.001", | ||
| "name": "OPNsense stable (serial output) installation on Hard Disk", | ||
| "module": "Dasharo Compatibility", | ||
| "module": "Dasharo OSes", |
There was a problem hiding this comment.
Introducing new test modules will break a couple things related to, but not only, osfv-dashboard.
- the test ID validity checks in CI at
scripts/ci/test_ids/test_ids_legal.py - pre-commit matches every directory that starts with "dasharo*", so its fine: https://github.com/Dasharo/open-source-firmware-validation/blob/develop/.pre-commit-config.yaml#L121
- README will break too: https://github.com/Dasharo/open-source-firmware-validation/blob/develop/README.md?plain=1#L69
- (...more?)
Please search for any instances of the test module names and fix them if this change is to stay
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
dasharo-compatibility/os-opnsense.robot:
dasharo-compatibility/os-pfsense.robot:
Test suites are skipping in Suite Setup if BSD ENV IDs are not present in TESTED_LINUX_DISTROS variable.
BSD_os-opnsense_2025_08_29_10_43_00_PASS.zip
BSD_os-opnsense_2025_08_29_10_52_59_SKIP.zip
BSD_os-pfsense_2025_08_29_10_40_30_PASS.zip
BSD_os-pfsense_2025_08_29_10_52_48_SKIP.zip