[WIP] 🌱 e2e: fix test for workload-upgrade

[chrischdi]

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

TODOs:

• squashed commits
• includes documentation
• adds unit tests

Release note:

NONE

[netlify]

✅ Deploy Preview for kubernetes-sigs-cluster-api-gcp ready!

Name	Link
🔨 Latest commit	6b098c7
🔍 Latest deploy log	https://app.netlify.com/projects/kubernetes-sigs-cluster-api-gcp/deploys/69162d8d28f65d00089cc8a8
😎 Deploy Preview	https://deploy-preview-1556--kubernetes-sigs-cluster-api-gcp.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chrischdi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [chrischdi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[chrischdi]

/test help

[k8s-ci-robot]

@chrischdi: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test pull-cluster-api-provider-gcp-build

/test pull-cluster-api-provider-gcp-e2e-test

/test pull-cluster-api-provider-gcp-make

/test pull-cluster-api-provider-gcp-test

/test pull-cluster-api-provider-gcp-verify

The following commands are available to trigger optional jobs:

/test pull-cluster-api-provider-gcp-apidiff

/test pull-cluster-api-provider-gcp-capi-e2e

/test pull-cluster-api-provider-gcp-conformance

/test pull-cluster-api-provider-gcp-conformance-ci-artifacts

/test pull-cluster-api-provider-gcp-coverage

/test pull-cluster-api-provider-gcp-e2e-workload-upgrade

Use /test all to run the following jobs that were automatically triggered:

pull-cluster-api-provider-gcp-apidiff

pull-cluster-api-provider-gcp-build

pull-cluster-api-provider-gcp-e2e-test

pull-cluster-api-provider-gcp-make

pull-cluster-api-provider-gcp-test

pull-cluster-api-provider-gcp-verify

In response to this:

/test help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[chrischdi]

/test pull-cluster-api-provider-gcp-conformance-ci-artifacts
/test pull-cluster-api-provider-gcp-e2e-workload-upgrade

[chrischdi]

/test pull-cluster-api-provider-gcp-conformance-ci-artifacts
/test pull-cluster-api-provider-gcp-e2e-workload-upgrade

[cpanato]

@chrischdi those two tests are currently broken, we need to find time to fix it

[chrischdi]

@chrischdi those two tests are currently broken, we need to find time to fix it

That is my plan on this PR :-)

[cpanato]

thanks!

[chrischdi]

/test pull-cluster-api-provider-gcp-e2e-workload-upgrade

[k8s-ci-robot]

@chrischdi: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cluster-api-provider-gcp-conformance-ci-artifacts	a4a5591	link	false	/test pull-cluster-api-provider-gcp-conformance-ci-artifacts
pull-cluster-api-provider-gcp-e2e-workload-upgrade	f2a4d48	link	false	/test pull-cluster-api-provider-gcp-e2e-workload-upgrade

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[chrischdi]

I it is impossible to debug this further with the current CI.

I'd next look at cloud-init / kubelet logs etc for e.g. this machine: k8s-upgrade-and-conformance-fj0afj-control-plane-nxqt8

[chrischdi]

Note: maybe shell out to gcloud compute ssh would be a handy way, or use the sdk to do the same via go code.

Prior art of commands which help to get output:

		captureLogs("kubelet.log",
			"sudo journalctl", "--no-pager", "--output=short-precise", "-u", "kubelet.service"),
		captureLogs("containerd.log",
			"sudo journalctl", "--no-pager", "--output=short-precise", "-u", "containerd.service"),
		captureLogs("cloud-init.log",
			"sudo", "cat", "/var/log/cloud-init.log"),
		captureLogs("cloud-init-output.log",
			"sudo", "cat", "/var/log/cloud-init-output.log"),
		captureLogs("kubeadm-service.log",
			"sudo", "cat", "/var/log/kubeadm-service.log"),
		captureLogs("pod-logs.tar.gz",
			"sudo", "tar", "-czf", "-", "-C", "/var/log", "pods"),

https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/blob/main/test/framework/log/collector.go#L90-L101

[damdo]

Note: maybe shell out to gcloud compute ssh would be a handy way, or use the sdk to do the same via go code.

I think this would be handy to debug things like this for sure!

[chrischdi]

(Note: didn't work ...)