OCPBUGS-63015: QEMU logs are not gathered if there are pending status virt-launcher pods #1164

ncaak · 2025-10-23T08:33:49Z

Bug: In some cases, after a crashloop, the VMs' virt-launcher pods are stuck in Pending status and do not contain a compute container to retrieve the logs from. If any pods are in this situation, the Gatherer receives an error from the API. The Gatherer then returns with the error, but not with logs from the pods that are running normally.
Fix: Only gathering logs from the virt-launcher pods that are in Running status.

Documentation

docs/gathered-data.md

Breaking Changes

No

References

https://issues.redhat.com/browse/OCPBUGS-63015

…nside)

openshift-ci-robot · 2025-10-23T08:33:54Z

@ncaak: This pull request references Jira Issue OCPBUGS-63015, which is invalid:

expected the bug to target the "4.21.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Bug: In some cases, after a crashloop, the VMs' virt-launcher pods are stuck in Pending status and do not contain a compute container to retrieve the logs from. If any pods are in this situation, the Gatherer receives an error from the API. The Gatherer then returns with the error, but not with logs from the pods that are running normally.
Fix: Only gathering logs from the virt-launcher pods that are in Running status.

Categories

Bugfix

Data Enhancement

Feature

Backporting

Others (CI, Infrastructure, Documentation)

Documentation

docs/gathered-data.md

Breaking Changes

No

References

https://issues.redhat.com/browse/OCPBUGS-63015

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2025-10-23T08:34:13Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ncaak

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ncaak]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ncaak · 2025-10-23T08:35:18Z

/jira refresh

openshift-ci-robot · 2025-10-23T08:35:26Z

@ncaak: This pull request references Jira Issue OCPBUGS-63015, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.21.0) matches configured target version for branch (4.21.0)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @BaiyangZhou

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

ncaak · 2025-10-23T08:35:57Z

/verified later @BaiyangZhou

openshift-ci-robot · 2025-10-23T08:36:10Z

@ncaak: This PR has been marked to be verified later by @BaiyangZhou.

In response to this:

/verified later @BaiyangZhou

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

ncaak · 2025-10-23T14:38:16Z

/test e2e-gcp-ovn-techpreview

ncaak · 2025-10-23T15:20:00Z

/test okd-scos-e2e-aws-ovn

ncaak · 2025-10-24T07:26:37Z

/test okd-scos-e2e-aws-ovn

ncaak · 2025-10-24T09:58:10Z

/test okd-scos-e2e-aws-ovn

ncaak · 2025-10-24T11:39:30Z

/test okd-scos-e2e-aws-ovn

opokornyy · 2025-10-24T13:18:58Z

pkg/gatherers/clusterconfig/gather_qemu_kubevirt_launcher_logs.go

 	return common.LogContainersFilter{
 		Namespace:              metav1.NamespaceAll,
 		LabelSelector:          "kubevirt.io=virt-launcher",
+		FieldSelector:          "status.phase=Running",


Did you also consider gathering data from containers that are marked as Succeeded or Failed? Or is the Running phase sufficient for us?

Yeah, that's the work I'm checking on other ticket → OCPBUGS-63396

BaiyangZhou · 2025-10-27T02:11:53Z

/test okd-scos-e2e-aws-ovn

openshift-ci · 2025-10-27T05:21:16Z

@ncaak: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

opokornyy · 2025-10-27T09:23:43Z

/lgtm

BaiyangZhou · 2025-10-27T09:24:51Z

/label qe-approved

openshift-ci-robot · 2025-10-27T09:24:58Z

@ncaak: This pull request references Jira Issue OCPBUGS-63015, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.21.0) matches configured target version for branch (4.21.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @BaiyangZhou

In response to this:

Bug: In some cases, after a crashloop, the VMs' virt-launcher pods are stuck in Pending status and do not contain a compute container to retrieve the logs from. If any pods are in this situation, the Gatherer receives an error from the API. The Gatherer then returns with the error, but not with logs from the pods that are running normally.
Fix: Only gathering logs from the virt-launcher pods that are in Running status.

Categories

Bugfix

Data Enhancement

Feature

Backporting

Others (CI, Infrastructure, Documentation)

Documentation

docs/gathered-data.md

Breaking Changes

No

References

https://issues.redhat.com/browse/OCPBUGS-63015

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2025-10-27T09:27:54Z

@ncaak: Jira Issue OCPBUGS-63015: All pull requests linked via external trackers have merged:

openshift/insights-operator#1164

This pull request has the verified-later tag and will need to be manually moved to VERIFIED after testing. Jira Issue OCPBUGS-63015 has been moved to the MODIFIED state.

In response to this:

Bug: In some cases, after a crashloop, the VMs' virt-launcher pods are stuck in Pending status and do not contain a compute container to retrieve the logs from. If any pods are in this situation, the Gatherer receives an error from the API. The Gatherer then returns with the error, but not with logs from the pods that are running normally.
Fix: Only gathering logs from the virt-launcher pods that are in Running status.

Categories

Bugfix

Data Enhancement

Feature

Backporting

Others (CI, Infrastructure, Documentation)

Documentation

docs/gathered-data.md

Breaking Changes

No

References

https://issues.redhat.com/browse/OCPBUGS-63015

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-merge-robot · 2025-10-30T13:00:06Z

Fix included in accepted release 4.21.0-0.nightly-2025-10-30-060549

ncaak · 2025-11-07T15:24:15Z

/cherry-pick release-4.20

openshift-cherrypick-robot · 2025-11-07T15:25:02Z

@ncaak: new pull request created: #1174

In response to this:

/cherry-pick release-4.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ncaak added 2 commits October 23, 2025 10:24

Add filtering to remove Pending pods (no logs or compute containers i…

d5b6f7d

…nside)

Update with QEMU gatherer last fix

13af60d

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Oct 23, 2025

openshift-ci-robot added the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Oct 23, 2025

openshift-ci bot requested review from katushiik11 and opokornyy October 23, 2025 08:34

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 23, 2025

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Oct 23, 2025

openshift-ci bot requested a review from BaiyangZhou October 23, 2025 08:35

openshift-ci-robot added verified-later verified Signifies that the PR passed pre-merge verification criteria labels Oct 23, 2025

opokornyy reviewed Oct 24, 2025

View reviewed changes

openshift-ci bot assigned opokornyy Oct 27, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Oct 27, 2025

openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label Oct 27, 2025

openshift-merge-bot bot merged commit b84f719 into openshift:master Oct 27, 2025
11 checks passed

openshift-cherrypick-robot mentioned this pull request Nov 7, 2025

[release-4.20] OCPBUGS-64800: QEMU logs are not gathered if there are pending status virt-launcher pods #1174

Open

OCPBUGS-63015: QEMU logs are not gathered if there are pending status virt-launcher pods #1164

OCPBUGS-63015: QEMU logs are not gathered if there are pending status virt-launcher pods #1164

Conversation

ncaak commented Oct 23, 2025

Categories

Documentation

Breaking Changes

References

Uh oh!

openshift-ci-robot commented Oct 23, 2025

Categories

Documentation

Breaking Changes

References

Uh oh!

openshift-ci bot commented Oct 23, 2025

Uh oh!

ncaak commented Oct 23, 2025

Uh oh!

openshift-ci-robot commented Oct 23, 2025

Uh oh!

ncaak commented Oct 23, 2025

Uh oh!

openshift-ci-robot commented Oct 23, 2025

Uh oh!

ncaak commented Oct 23, 2025

Uh oh!

ncaak commented Oct 23, 2025

Uh oh!

ncaak commented Oct 24, 2025

Uh oh!

ncaak commented Oct 24, 2025

Uh oh!

ncaak commented Oct 24, 2025

Uh oh!

opokornyy Oct 24, 2025

Choose a reason for hiding this comment

Uh oh!

ncaak Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

BaiyangZhou commented Oct 27, 2025

Uh oh!

openshift-ci bot commented Oct 27, 2025

Uh oh!

opokornyy commented Oct 27, 2025

Uh oh!

BaiyangZhou commented Oct 27, 2025

Uh oh!

openshift-ci-robot commented Oct 27, 2025

Categories

Documentation

Breaking Changes

References

Uh oh!

Uh oh!

openshift-ci-robot commented Oct 27, 2025

Categories

Documentation

Breaking Changes

References

Uh oh!

openshift-merge-robot commented Oct 30, 2025

Uh oh!

ncaak commented Nov 7, 2025

Uh oh!

openshift-cherrypick-robot commented Nov 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants