Changes to migrate to new TUF repository

[lucasmrod]

Changes

• orbit >= 1.38.0, when configured to connect to https://tuf.fleetctl.com (existing fleetd deployments) will now connect to https://updates.fleetdm.com and start using the metadata in path /opt/orbit/updates-metadata.json.
• orbit >= 1.38.0, when configured to connect to some custom TUF (not Fleet's TUFs) will copy /opt/orbit/tuf-metadata.json to /opt/orbit/updates-metadata.json (if it doesn't exist) and start using the latter.
• fleetctl 4.63.0 will now generate artifacts using https://updates.fleetdm.com by default (or a custom TUF if --update-url is set) and generate two (same file) metadata files /opt/orbit/updates-metadata.json and the legacy one to support downgrades /opt/orbit/tuf-metadata.json.
• fleetctl 4.62.0 when configured to use custom TUF (not Fleet's TUF) will generate just the legacy metadata file /opt/orbit/tuf-metadata.json.

User stories

See "User stories" in https://github.com/fleetdm/confidential/issues/8488.

• Update update.defaultRootMetadata and update.DefaultURL when the new repository is ready.
• Changes file added for user-visible changes in changes/, orbit/changes/ or ee/fleetd-chrome/changes.
  See Changes files for more information.
• Added/updated tests
• Manual QA for all new/changed functionality
• For Orbit and Fleet Desktop changes:
  • Orbit runs on macOS, Linux and Windows. Check if the orbit feature/bugfix should only apply to one platform (runtime.GOOS).
  • Manual QA must be performed in the three main OSs, macOS, Windows and Linux.
  • Auto-update manual QA, from released version of component to new version (see tools/tuf/test).

[codecov]

Codecov Report

Attention: Patch coverage is 12.86550% with 149 lines in your changes missing coverage. Please review.

Project coverage is 63.75%. Comparing base (d079b63) to head (aec9d3c).
Report is 64 commits behind head on main.

Files with missing lines	Patch %	Lines
orbit/pkg/update/update.go	16.66%	87 Missing and 3 partials ⚠️
orbit/cmd/orbit/orbit.go	0.00%	42 Missing ⚠️
orbit/pkg/update/runner.go	0.00%	11 Missing and 1 partial ⚠️
orbit/pkg/packaging/packaging.go	50.00%	2 Missing and 1 partial ⚠️
cmd/fleetctl/preview.go	0.00%	1 Missing ⚠️
orbit/cmd/orbit/shell.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #23588      +/-   ##
==========================================
- Coverage   63.84%   63.75%   -0.10%     
==========================================
  Files        1616     1618       +2     
  Lines      153829   154316     +487     
  Branches     4026     4026              
==========================================
+ Hits        98218    98378     +160     
- Misses      47796    48112     +316     
- Partials     7815     7826      +11     

Flag	Coverage Δ
backend	64.61% <12.86%> (-0.11%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[lukeheath]

Adding @getvictor as a reviewer so the MDM product group is in the loop on what's changing.

cc @georgekarrv

[lucasmrod]

I'll convert to draft to attempt to iterate a safer migration and one that would allow rolling back to 1.35.0 if a customer/user wants to.

[getvictor]

Looks good.

[zwass]

The changes lgtm! Let's remember this needs EXTENSIVE testing before going live. If there are issues we risk having to ask users to reinstall agent packages..

[lukeheath]

Converting PR to draft until it's ready to merge.

[lucasmrod]

@zwass We added some changes to support the scenario where 1.38.0 needs an urgent fix (or customer wants to downgrade to 1.37.0) and endpoints cannot access https://updates.fleetdm.com.

To cover that scenario we added HasAccessToNewTUFServer, if it returns false then orbit 1.38.0+ will resort to communicating to the old TUF. Once HasAccessToNewTUFServer succeeds for the first time, then it will always communicate with the new TUF (by storing a /opt/orbit/new-tuf-checked file).

Please re-review this commit.

PS: Was left as draft to not affect our metrics.

[lukeheath]

I'm setting this to "Ready for review" now that it's ready. Because it's so old it would be great to merge by Friday before we report KPIs :)

[zwass]

IIUC the check for access to the new repo only happens while the old update URL is configured. Once the access is detected, the new update URL gets configured and the check stops happening. Is that right? If so it seems reasonable to me.

[lucasmrod]

IIUC the check for access to the new repo only happens while the old update URL is configured. Once the access is detected, the new update URL gets configured and the check stops happening. Is that right? If so it seems reasonable to me.

Correct.

[getvictor]

LGTM