[PM-14425] [BEEEP] Add test plans #1106

fedemkr · 2024-11-01T22:13:04Z

🎟️ Tracking

PM-14425

📔 Objective

Add test plans so we can separate Unit vs Snapshot vs View inspector tests vs All tests.

✂️ Tests split

This PR splits tests files (mostly *ViewTests) into three categories:

Unit: Same files and tests we were using but without including snapshot and view inspector tests. Filename ending with *Tests.
- One particular note is that a few of these tests may use InlineSnapshotTesting so these are the exceptions why the InlineSnapshotTesting package product is still needed in such targets.
Snapshot: Filename ending with *SnapshotTests and that only includes snapshot tests.
ViewInspector: Filename ending with *ViewInspectorTests and that only includes view inspector tests.

📁 Folder structure

There are new test plans per scheme (Password Manager, Authenticator and Kit) following the same folder structure as in other places:

TestPlans
TestPlans-bwa
TestPlans-bwk

🧪 New test plans

And each folder have the same structure of test plans for each scheme following the patterns described above:

TestPlan-Default.xctestplan: Default test plan including all tests for the scheme.
TestPlan-Snapshot.xctestplan: Test plan including only snapshot tests for the scheme.
TestPlan-Unit.xctestplan: Test plan including only unit (not snapshots nor view inspector) tests for the scheme.
TestPlan-ViewInspector.xctestplan: Test plan including only view inspector tests for the scheme.

Tip

An environment variable SKIP_SIMULATOR_CHECK_FOR_TESTS has been added to be checked in the base test case so we can avoid checking the simulator name and version when we want. This is particular useful on Unit test plans where it's configured.

⏰ Reminders before review

Contributor guidelines followed
All formatters and local linters executed and passed
Written new unit and / or integration tests where applicable
Protected functional changes with optionality (feature flags)
Used internationalization (i18n) for all UI strings
CI builds passed
Communicated to DevOps any deployment requirements
Updated any necessary documentation (Confluence, contributing docs) or informed the documentation team

🦮 Reviewer guidelines

👍 (:+1:) or similar for great changes
📝 (:memo:) or ℹ️ (:information_source:) for notes or general info
❓ (:question:) for questions
🤔 (:thinking:) or 💭 (:thought_balloon:) for more open inquiry that's not quite a confirmed issue and could potentially benefit from discussion
🎨 (:art:) for suggestions / improvements
❌ (:x:) or ⚠️ (:warning:) for more significant problems or concerns needing attention
🌱 (:seedling:) or ♻️ (:recycle:) for future improvements or indications of technical debt
⛏ (:pick:) for minor or nitpick changes

…s separately.

…ck in BitwardenTestCase for skip simulator checking.

… unit test plans so they are empty on remote and only generated locally each time if needed.

github-actions · 2024-11-15T17:43:24Z

Checkmarx One – Scan Summary & Details – 6677197c-db35-430d-a68d-49709e4bbbaf

Great job! No new security vulnerabilities introduced in this pull request

GlobalTestHelpers/Support/BitwardenTestCase.swift

Co-authored-by: Katherine Bertelsen <[email protected]>

fedemkr · 2024-11-15T19:57:18Z

Scripts/generate_test_plans.sh

+TEST_PLAN_SNAPSHOT_PATH="TestPlans/TestPlan-Snapshot.xctestplan"
+SNAPSHOT_TESTS_TARGET="BitwardenSharedTests"
+
+mint install atakankarsli/[email protected]


I've decided not to include the installation of xctestplanner in the Mintfile so it's only installed when actually being used. For now this script won't be called in CI/CD so there's no need to waste time installing xctestplanner .

matt-livefront

This is cool!

A few questions from me:

I'm seeing a few date/time failures in the tests. It looks like we previously set TZ=UTC as an environment variable. Could you add that to the test plans?
We used to have a splash screen that showed while testing. That relies on a -testing argument passed on launch, could you add that?
Should your comment for git update-index --skip-worktree TestPlans/TestPlan-Snapshot.xctestplan TestPlans/TestPlan-Unit.xctestplan be documented in the README?

Previous test settings:

It looks like the test plans don't include the Networking target, could you include that? I think that's why the test counts are different.

# Conflicts: # GlobalTestHelpers/Support/BitwardenTestCase.swift # project-pm.yml

codecov · 2025-10-07T12:54:15Z

Codecov Report

❌ Patch coverage is 28.13121% with 2892 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.26%. Comparing base (4376077) to head (55cd8ab).

Files with missing lines	Patch %	Lines
...em/AddEditItem/AddEditItemView+SnapshotTests.swift	0.00%	407 Missing ⚠️
...aultItem/ViewItem/ViewItemView+SnapshotTests.swift	0.00%	326 Missing ⚠️
.../Vault/VaultList/VaultListView+SnapshotTests.swift	0.00%	159 Missing ⚠️
...itSendItem/AddEditSendItemView+SnapshotTests.swift	0.00%	132 Missing ⚠️
...leSwitcher/ProfileSwitcherView+SnapshotTests.swift	0.00%	130 Missing ⚠️
...ntSecurity/AccountSecurityView+SnapshotTests.swift	0.00%	129 Missing ⚠️
...eSwitcher/ProfileSwitcherSheet+SnapshotTests.swift	0.00%	122 Missing ⚠️
...erator/Generator/GeneratorView+SnapshotTests.swift	0.00%	109 Missing ⚠️
...end/Send/SendList/SendListView+SnapshotTests.swift	0.00%	83 Missing ⚠️
...derLogs/FlightRecorderLogsView+SnapshotTests.swift	0.00%	70 Missing ⚠️
... and 47 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1106      +/-   ##
==========================================
+ Coverage   81.35%   85.26%   +3.90%     
==========================================
  Files         831     1693     +862     
  Lines       52483   143883   +91400     
==========================================
+ Hits        42697   122677   +79980     
- Misses       9786    21206   +11420

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

… and view inspector tests. Added Testplan-ViewInspector test plan and configure it on PM.

…(files ending with SnapshotTests) and view inspector tests (files ending with ViewInspectorTests). Moved methods and clear files in the meantime. Snapshots are still disabled.

… new test file structure and suffixes: Unit (files ending with Tests), Snapshots (files ending with SnapshotTest) and View inspector (files ending with ViewInspectorTests).

…can configure that as an environment variable from the Test plans as needed.

vvolkgang · 2025-10-09T21:15:57Z

TestPlans-bwk/TestPlan-ViewInspector.xctestplan

@@ -0,0 +1,37 @@
+{


💭 Thoughts on having all of these files under TestPlans folder and just naming them accordingly?

E.g. TestPlans/TestPlan-ViewInspector-bwk.xctestplan

That was my initial thought as well but then I saw other folders structure we are using and followed the same pattern, like GlobalTestHelpers, Configs, Scripts.
Nonetheless, I agree that it'd be better having the scheme specified per file instead of per folder in this case, but we would deviate from our current structure.
What do you think?

fedemkr added 4 commits November 1, 2024 13:45

Added Test plans into the project so we can run Unit vs Snapshot test…

68aa059

…s separately.

PM-14425 Added test plans, also generator script and env variable che…

1dcbdcf

…ck in BitwardenTestCase for skip simulator checking.

Merge branch 'main' into PM-14425/test-plans

134440b

PM-14425 Improved generate_test_plans script and cleaned snapshot and…

c299141

… unit test plans so they are empty on remote and only generated locally each time if needed.

KatherineInCode reviewed Nov 15, 2024

View reviewed changes

GlobalTestHelpers/Support/BitwardenTestCase.swift Outdated Show resolved Hide resolved

Update GlobalTestHelpers/Support/BitwardenTestCase.swift

18a10db

Co-authored-by: Katherine Bertelsen <[email protected]>

fedemkr requested a review from KatherineInCode November 15, 2024 19:52

fedemkr commented Nov 15, 2024

View reviewed changes

fedemkr marked this pull request as ready for review November 15, 2024 19:58

fedemkr requested review from matt-livefront, phil-livefront, brant-livefront and a team as code owners November 15, 2024 19:58

matt-livefront reviewed Nov 20, 2024

View reviewed changes

fedemkr added 3 commits September 29, 2025 13:52

Merge branch 'main' into PM-14425/test-plans

ffda64e

# Conflicts: # GlobalTestHelpers/Support/BitwardenTestCase.swift # project-pm.yml

Merge branch 'main' into PM-14425/test-plans

fd6cf8b

Merge branch 'main' into PM-14425/test-plans

5027b3b

fedemkr added 7 commits October 7, 2025 11:42

Merge branch 'main' into PM-14425/test-plans

cbd35a8

PM-14425 Updated test plans generation to take into account snapshots…

d5d6729

… and view inspector tests. Added Testplan-ViewInspector test plan and configure it on PM.

Merge branch 'main' into PM-14425/test-plans

74933bc

PM-14425 Split tests into regular tests (unit tests), snapshot tests …

d12847d

…(files ending with SnapshotTests) and view inspector tests (files ending with ViewInspectorTests). Moved methods and clear files in the meantime. Snapshots are still disabled.

PM-14425 Configured Test plans for BWPM, BWA and BWK depending on the…

b052601

… new test file structure and suffixes: Unit (files ending with Tests), Snapshots (files ending with SnapshotTest) and View inspector (files ending with ViewInspectorTests).

PM-14425 Added step to skip simulator check on base test case; so we …

76a2444

…can configure that as an environment variable from the Test plans as needed.

PM-14425 Remove generate test plans script.

55cd8ab

fedemkr requested a review from matt-livefront October 9, 2025 20:31

vvolkgang reviewed Oct 9, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[PM-14425] [BEEEP] Add test plans #1106

[PM-14425] [BEEEP] Add test plans #1106

Uh oh!

fedemkr commented Nov 1, 2024 •

edited

Loading

Uh oh!

github-actions bot commented Nov 15, 2024 •

edited

Loading

Uh oh!

Uh oh!

fedemkr Nov 15, 2024

Uh oh!

matt-livefront left a comment

Uh oh!

codecov bot commented Oct 7, 2025 •

edited

Loading

Uh oh!

vvolkgang Oct 9, 2025 •

edited

Loading

Uh oh!

fedemkr Oct 9, 2025 •

edited

Loading

Uh oh!

Uh oh!

[PM-14425] [BEEEP] Add test plans #1106

Are you sure you want to change the base?

[PM-14425] [BEEEP] Add test plans #1106

Uh oh!

Conversation

fedemkr commented Nov 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🎟️ Tracking

📔 Objective

⏰ Reminders before review

🦮 Reviewer guidelines

Uh oh!

github-actions bot commented Nov 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Great job! No new security vulnerabilities introduced in this pull request

Uh oh!

Uh oh!

fedemkr Nov 15, 2024

Choose a reason for hiding this comment

Uh oh!

matt-livefront left a comment

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

vvolkgang Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fedemkr Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

fedemkr commented Nov 1, 2024 •

edited

Loading

github-actions bot commented Nov 15, 2024 •

edited

Loading

codecov bot commented Oct 7, 2025 •

edited

Loading

vvolkgang Oct 9, 2025 •

edited

Loading

fedemkr Oct 9, 2025 •

edited

Loading