oc exec hangs for six hours

[stevekuznetsov]

[INFO] Validating exec
Running test/end-to-end/core.sh:463: executing 'oc exec -p frontend-1-nsqdp id' expecting success and text '1000'...
SUCCESS after 0.266s: test/end-to-end/core.sh:463: executing 'oc exec -p frontend-1-nsqdp id' expecting success and text '1000'
Standard output from the command:
uid=1000050000 gid=0(root) groups=0(root),1000050000

Standard error from the command:
W0406 10:51:16.968491   15548 cmd.go:337] -p POD_NAME is DEPRECATED and will be removed in a future version. Use exec POD_NAME instead.

Running test/end-to-end/core.sh:464: executing 'oc rsh pod/frontend-1-nsqdp id -u' expecting success and text '1000'...
Killed by signal 15.

as seen here

/cc @ncdc

[ncdc]

Sorry, I'm stretched too thin.

…

On Thu, Apr 6, 2017 at 4:06 PM, Steve Kuznetsov ***@***.***> wrote: [INFO] Validating exec Running test/end-to-end/core.sh:463: executing 'oc exec -p frontend-1-nsqdp id' expecting success and text '1000'... SUCCESS after 0.266s: test/end-to-end/core.sh:463: executing 'oc exec -p frontend-1-nsqdp id' expecting success and text '1000' Standard output from the command: uid=1000050000 gid=0(root) groups=0(root),1000050000 Standard error from the command: W0406 10:51:16.968491 15548 cmd.go:337] -p POD_NAME is DEPRECATED and will be removed in a future version. Use exec POD_NAME instead. Running test/end-to-end/core.sh:464: executing 'oc rsh pod/frontend-1-nsqdp id -u' expecting success and text '1000'... Killed by signal 15. as seen here <https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_integration/870/consoleFull> /cc @ncdc <https://github.com/ncdc> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#13662>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAABYj7-WfKPpb7pPR96E8ZQx0j-kTPPks5rtUW1gaJpZM4M2Ggn> .

[stevekuznetsov]

@ncdc can you triage this by passing it on to someone else then

[ncdc]

@derekwaynecarr @mfojtik do you have anyone who can investigate?

[stevekuznetsov]

also: https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_integration/871/

This one is on oc rsh:

Running test/end-to-end/core.sh:488: executing 'oc rsh frontend-1-90m28 ls /tmp/sample-app' expecting success and text 'application-template-stibuild'...
Connection to 172.18.11.10 closed by remote host.

This is a very new regression and seems to be happening often.

[enj]

Seen in #11647 - https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_integration/872/consoleFull#-107583498258b6e51eb7608a5981914356

Excerpt:

Running test/end-to-end/core.sh:464: executing 'oc rsh pod/frontend-1-bn6v4 id -u' expecting success and text '1000'...
Connection to 172.18.0.183 closed by remote host.

Context:

logs: ok
[INFO] Starting build from /tmp/openshift/test-end-to-end/artifacts/stiAppConfig.json with non-existing commit...
Running test/end-to-end/core.sh:456: executing 'oc start-build test --commit=fffffff --wait' expecting failure...
SUCCESS after 0.207s: test/end-to-end/core.sh:456: executing 'oc start-build test --commit=fffffff --wait' expecting failure
There was no output from the command.
Standard error from the command:
Error from server (Forbidden): buildconfigs "test" is forbidden: buildconfigs.build.openshift.io "test" not found

[INFO] Validating exec
Running test/end-to-end/core.sh:463: executing 'oc exec -p frontend-1-bn6v4 id' expecting success and text '1000'...
SUCCESS after 9.809s: test/end-to-end/core.sh:463: executing 'oc exec -p frontend-1-bn6v4 id' expecting success and text '1000'
Standard output from the command:
uid=1000050000 gid=0(root) groups=0(root),1000050000
uid=1000050000 gid=0(root) groups=0(root),1000050000

Standard error from the command:
W0406 12:06:12.530529   31700 cmd.go:337] -p POD_NAME is DEPRECATED and will be removed in a future version. Use exec POD_NAME instead.

Running test/end-to-end/core.sh:464: executing 'oc rsh pod/frontend-1-bn6v4 id -u' expecting success and text '1000'...
Connection to 172.18.0.183 closed by remote host.
++ export status=FAILURE
++ status=FAILURE
+ set +o xtrace
########## FINISHED STAGE: FAILURE: RUN INTEGRATION TESTS ##########
Build step 'Execute shell' marked build as failure

[stevekuznetsov]

Seen in https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_integration/874/consoleFull

[INFO] Starting build from /tmp/openshift/test-end-to-end/artifacts/stiAppConfig.json with non-existing commit...
Running test/end-to-end/core.sh:456: executing 'oc start-build test --commit=fffffff --wait' expecting failure...
SUCCESS after 0.214s: test/end-to-end/core.sh:456: executing 'oc start-build test --commit=fffffff --wait' expecting failure
There was no output from the command.
Standard error from the command:
Error from server (Forbidden): buildconfigs "test" is forbidden: buildconfigs.build.openshift.io "test" not found

[INFO] Validating exec
Running test/end-to-end/core.sh:463: executing 'oc exec -p frontend-1-b0sd0 id' expecting success and text '1000'...
Connection to 172.18.7.107 closed by remote host.

@mfojtik should we be seeing the new API groups right now?

[stevekuznetsov]

Every single build since https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin/604/ has this