Adding remaining workflows and platforms.

[DaniilKl]

At the time of writing this issue DTS E2E tests miss some platforms (at least MSI Z790) as well as workflows (e.g. backup workflow, some workflows and test cases for different subscription and platforms). This should be analyzed and needed tests should be added.

[DaniilKl]

This list is in progress. Important note: the profiles should be generated for success paths only.

E2E

HCL Report

• Run on Qemu without mocking -> should pass even on errors

Notes:

• HCL report should only dump information about HW/FW. If some tool fails, this should be considered as not as error in HCL report but as an unavailability of needed information and reported by HCL. This will ensure we could get HCL report from both supported and unsupported platforms (Make functions in HCL Report not to exit dts-scripts#64).

Dasharo deploy

• Installation
  • Community (coreboot + UEFI) installation
    • MSI PRO Z690-A DDR4
    • MSI PRO Z690-A
    • Novacustom NV4x 11th Gen Tiger Lake
    • Novacustom NS5x/7x 11th Gen Tiger Lake
    • Novacustom NS5x/7x 12th Gen Alder Lake
    • Novacustom NV4x 12th Gen Alder Lake
    • Novacustom V540TU/V560TU 14th Gen Meteor Lake
    • Novacustom V540TNx/V560TNx 14th Gen Meteor Lake
  • DPP (coreboot + UEFI) installation with/without credentials -> should pass/should not pass
    • MSI PRO Z690-A DDR4
    • MSI PRO Z690-A
    • MSI PRO Z790-P DDR4
    • MSI PRO Z790-P
    • Dell Optiplex 7010
    • Dell OptiPlex 9010
    • PC Engines APU2
    • PC Engines APU3
    • PC Engines APU4
    • PC Engines APU6
    • ODROID H4
  • DPP (coreboot + SeaBIOS) Installation with/without credentials -> should pass/should not pass
    • PC Engines APU2
    • PC Engines APU3
    • PC Engines APU4
    • PC Engines APU6
  • DPP (Slim Bootloader+UEFI) installation with credentials -> should pass
    • ODROID H4
• Update
  • Generic update
    • Community (coreboot + UEFI) -> Community (coreboot + UEFI) -> Update should pass
      • MSI PRO Z690-A DDR4
      • MSI PRO Z690-A
      • Novacustom NV4x 11th Gen Tiger Lake
      • Novacustom NS5x/7x 11th Gen Tiger Lake
      • Novacustom NS5x/7x 12th Gen Alder Lake
      • Novacustom NV4x 12th Gen Alder Lake
      • Novacustom V540TU/V560TU 14th Gen Meteor Lake
      • Novacustom V540TNx/V560TNx 14th Gen Meteor Lake
    • DPP (coreboot + UEFI) -> DPP (coreboot + UEFI) with/without credentials -> update should/should not pass
      • MSI PRO Z690-A DDR4
      • MSI PRO Z690-A
      • MSI PRO Z790-P DDR4
      • MSI PRO Z790-P
      • Dell Optiplex 7010
      • Dell OptiPlex 9010
      • PC Engines APU2
      • PC Engines APU3
      • PC Engines APU4
      • PC Engines APU6
      • ODROID H4
    • DPP (coreboot + SeaBIOS) -> DPP (coreboot + SeaBIOS) with/without credentials -> update should/should not pass
      • PC Engines APU2
      • PC Engines APU3
      • PC Engines APU4
      • PC Engines APU6
    • DPP (coreboot + Heads) -> DPP (coreboot + Heads) with/without credentials -> update should/should not pass
      • Novacustom NV4x 12th Gen Alder Lake
      • Novacustom V540TU/V560TU 14th Gen Meteor Lake
      • MSI PRO Z790-P DDR4
      • MSI PRO Z790-P
      • MSI PRO Z690-A DDR4
      • MSI PRO Z690-A
  • UEFI Capsule Update
    • DPP (coreboot + UEFI) -> DPP (coreboot + UEFI) with/without credentials -> Update should/should not pass
      • MSI PRO Z790-P DDR4
      • MSI PRO Z790-P
      • MSI PRO Z690-A DDR4
      • MSI PRO Z690-A
      • Novacustom NV4x 11th Gen Tiger Lake
      • Novacustom NS5x/7x 11th Gen Tiger Lake
      • Novacustom NS5x/7x 12th Gen Alder Lake
      • Novacustom NV4x 12th Gen Alder Lake
      • Novacustom V540TU/V560TU 14th Gen Meteor Lake
      • Novacustom V540TNx/V560TNx 14th Gen Meteor Lake
• Transition to Heads
  • Community (coreboot + UEFI) -> DPP (coreboot + Heads) with/without credentials -> update should/should not pass
    • Novacustom NV4x 12th Gen Alder Lake
    • Novacustom V540TU/V560TU 14th Gen Meteor Lake
    • MSI PRO Z690-A DDR4
    • MSI PRO Z690-A
  • DPP (coreboot + UEFI) -> DPP (coreboot + Heads) with/without credentials -> update should/should not pass
    • MSI PRO Z790-P DDR4
    • MSI PRO Z790-P
    • MSI PRO Z690-A DDR4
    • MSI PRO Z690-A
• Transition to UEFI
  • DPP (coreboot+SeaBIOS) -> DPP (coreboot+UEFI) with/without credentials -> update should/should not pass
    • PC Engines APU2
    • PC Engines APU3
    • PC Engines APU4
    • PC Engines APU6
• Transition to Dasharo (Slim Bootloader+UEFI)
  • DPP (coreboot+UEFI) -> DPP (Slim Bootloader+UEFI) with credentials -> update should pass
    • ODROID H4

Notes:

• Currently there is only one version of firmware that supports UEFI Capsule Update, therefore the start state of DTS is supposed to contain only DPP (coreboot + UEFI) version of firmware, because it is supposed that only DPP -> DPP UEFI Capsule Update will be available in DTS in near future.

Restore

• Restoring from cloud
• Restoring from local storage

Notes:

• I do not remember if this feature was ever used by our team/clients. Is it working? Is it useful?

DPP Credentials test cases (platform independent):

• Provide DPP Credentials with Dasharo firmware access and without DTS Extensions access -> DTS should have access to firmware only
• Provide DPP Credentials without Dasharo firmware access and with DTS Extensions access -> DTS should have access to extensions only
• Provide DPP Credentials with Dasharo firmware access and with DTS Extensions access -> DTS should have access to both products
• Provide DPP Credentials without Dasharo firmware access and without DTS Extensions access -> DTS should not have access to any of the products
• Replace already existing inside DTS credentials with new credentials -> the old credentials should be replaced without warnings/errors

Note: it is known that currently DPP credentials cannot be replaced without issues due to problems with mc (aka MinIO CLI) integration. This is to be fixed.

DTS Extensions test cases (platform independent):

• Install (DTS extensions are being installed during providing of DPP credentials) test DTS Extension
• Try to use the installed test DTS Extension

Note: this is the only functionalities for DTS Extensions right now.

Footer options tests

TBD

[pietrushnic]

s/Coreboot/coreboot/
s/Seabios/SeaBIOS/

Not sure what the conclusion is about h/Heads, but for now, I would follow: https://docs.dasharo.com/dasharo-naming-convention/#synopsis

[DaniilKl]

An additional test cases proposed by @m-iwanicki:

Test the verify_artifacts function to check whether it bloks deploying firmware with wrong signature and allows with correct signature.

[m-iwanicki]

@DaniilKl Do we need to test DPP -> Heads, Community -> Heads separately? Is there even any difference? There are only couple of cases where we check fw version and do something different based on it e.g. EC transition.

There is also Update UEFI Capsule DPP (coreboot + UEFI) -> DPP (coreboot + UEFI) case. I think we should check if DTS doesn't allow us to choose capsule update if our fw version is too low (doesn't support capsule). What if someone bought DPP subscription and wants to update and chooses capsule?

[DaniilKl]

I have added transition from coreboot+SeaBIOS to coreboot+UEFI for PC Engines.

[DaniilKl]

It is worth mentioning, that the UEFI Capsule Update, though supported by DTS is not used by any platforms currently. For the Heads update check this issue: Dasharo/dasharo-issues#1380.

[DaniilKl]

I have updated my comment to add Dasharo (Sliim Bootloader+UEFI) workflows.

[DaniilKl]

We have now the following list of tests:

The list

λ robot -L TRACE -v dts_config_ref:refs/heads/main -t "E2EH003.001*" dts/dts-e2e-helper.robot
==============================================================================
Dts-E2E-Helper                                                                
==============================================================================
E2EH003.001 Print names of test cases to be generated :: Print out... 
..msi-pro-z690-a-ddr5 Initial Deployment - DCR
msi-pro-z690-a-ddr5 Initial Deployment - DPP
msi-pro-z690-a-ddr5 UEFI Update - DCR
msi-pro-z690-a-ddr5 UEFI Update - DPP
msi-pro-z690-a-ddr5 UEFI->Heads Transition - DPP
msi-pro-z690-a-wifi-ddr4 Initial Deployment - DCR
msi-pro-z690-a-wifi-ddr4 Initial Deployment - DPP
msi-pro-z690-a-wifi-ddr4 UEFI Update - DCR
msi-pro-z690-a-wifi-ddr4 UEFI Update - DPP
msi-pro-z690-a-wifi-ddr4 UEFI->Heads Transition - DPP
msi-pro-z790-p-ddr5 Initial Deployment - DPP
msi-pro-z790-p-ddr5 UEFI Update - DPP
msi-pro-z790-p-ddr5 UEFI->Heads Transition - DPP
novacustom-ns50pu Initial Deployment - DCR
novacustom-ns50pu UEFI Update - DCR
novacustom-ns70mu Initial Deployment - DCR
novacustom-ns70mu UEFI Update - DCR
novacustom-ns70pu Initial Deployment - DCR
novacustom-ns70pu UEFI Update - DCR
novacustom-nv41mz Initial Deployment - DCR
novacustom-nv41mz UEFI Update - DCR
optiplex-7010 Initial Deployment - DPP
optiplex-7010 UEFI Update - DPP
optiplex-9010 Initial Deployment - DPP
optiplex-9010 UEFI Update - DPP
pcengines-apu2 UEFI Update - DPP
pcengines-apu2 SeaBIOS Update - DPP
pcengines-apu2 SeaBIOS->UEFI Transition - DPP
pcengines-apu3 UEFI Update - DPP
pcengines-apu3 SeaBIOS Update - DPP
pcengines-apu3 SeaBIOS->UEFI Transition - DPP
pcengines-apu4 UEFI Update - DPP
pcengines-apu4 SeaBIOS Update - DPP
pcengines-apu4 SeaBIOS->UEFI Transition - DPP
pcengines-apu6 UEFI Update - DPP
pcengines-apu6 SeaBIOS Update - DPP
pcengines-apu6 SeaBIOS->UEFI Transition - DPP
novacustom-ns50mu Initial Deployment - DCR
novacustom-ns50mu UEFI Update - DCR
novacustom-nuc_box-125H Initial Deployment - DCR
novacustom-nuc_box-155H Initial Deployment - DCR
novacustom-nv41mb Initial Deployment - DCR
novacustom-nv41mb UEFI Update - DCR
novacustom-nv41pz Initial Deployment - DCR
novacustom-nv41pz UEFI Update - DCR
novacustom-nv41pz UEFI->Heads Transition - DPP
novacustom-v540tnd Initial Deployment - DCR
novacustom-v540tnd UEFI Update - DCR
novacustom-v540tu Initial Deployment - DCR
novacustom-v540tu UEFI Update - DCR
novacustom-v540tu UEFI->Heads Transition - DPP
novacustom-v560tnd Initial Deployment - DCR
novacustom-v560tnd UEFI Update - DCR
novacustom-v560tne Initial Deployment - DCR
novacustom-v560tne UEFI Update - DCR
novacustom-v560tu Initial Deployment - DCR
novacustom-v560tu UEFI Update - DCR
novacustom-v560tu UEFI->Heads Transition - DPP
odroid-h4-plus Initial Deployment - DPP
odroid-h4-plus UEFI Update - DPP
odroid-h4-plus Dasharo (coreboot+UEFI) to Dasharo (Slim Bootloader+UEFI) Transition - DPP
odroid-h4-plus Dasharo (Slim Bootloader+UEFI) Initial Deployment - DPP
E2EH003.001 Print names of test cases to be generated :: Print out... | PASS |
------------------------------------------------------------------------------
Dts-E2E-Helper                                                        | PASS |
1 test, 1 passed, 0 failed
==============================================================================
Output:  /home/danillklimuk/Projects/DTS/open-source-firmware-validation/output.xml
Log:     /home/danillklimuk/Projects/DTS/open-source-firmware-validation/log.html
Report:  /home/danillklimuk/Projects/DTS/open-source-firmware-validation/report.html

So the platform per DTS workflow list above is outdated.