Fixing tests which attempted to restart the Surefire booter JVM #3804

jglick · 2018-12-12T03:29:15Z

Recent builds seem to be running this test over and over, finally timing out. I want to know if there is something about this test, or if there is some infrastructure change, etc. (I have not been able to track the issue down to a specific source commit.)

batmat · 2018-12-12T11:49:33Z

@jglick seems like the first run succeeded. I just closed/reopened to retrigger and see if this behaves differently.

How about use the way I used in #3788 and run only this test multiple times and see what it gives?

jglick · 2018-12-12T13:49:12Z

Well build 1 passed indeed, but wound up running JenkinsBuildsAndWorkspacesDirectoriesTest a dozen times. So something is weird with the Surefire runner it seems.

https://ci.jenkins.io/job/Core/job/jenkins/job/master/1179/execution/node/38/log/?consoleFull [WARNING] Corrupted stdin stream in forked JVM 2. See the dump file /home/jenkins/workspace/Core_jenkins_master-CZB5BLBJFXZE2BKR63DSXQKXRZJAOWBX73QP55GU5KMX7AHAF6CQ/test/target/surefire-reports/2018-12-12T10-57-04_077-jvmRun2.dumpstream

jglick · 2018-12-12T15:25:51Z

#3772 suggests that tests calling Jenkins.restart are the issue, as JenkinsBuildsAndWorkspacesDirectoriesTest does. I think I know this could be fixed.

jglick · 2018-12-12T15:34:51Z

Jenkinsfile

                stage("${buildType} Publishing") {
                    if (runTests) {
                        junit healthScaleFactor: 20.0, testResults: '*/target/surefire-reports/*.xml'
+                        archiveArtifacts allowEmptyArchive: true, artifacts: '**/target/surefire-reports/*.dumpstream'


→ jenkinsci/remoting#315

interesting, wasn''t aware of this feature. NIT then reading https://maven.apache.org/surefire/maven-surefire-plugin/faq.html#dumpfiles don't we actually want .dump in case?

I am not sure what *.dump is exactly, but anyway an actual *.dumpstream file led me to a source code issue. Can always improve as needed.

…t seems.

jglick · 2018-12-12T15:43:27Z

I think I know this could be fixed.

Hmm, my idea was reuseForks=false as in jenkinsci/workflow-cps-plugin#178, but test-pom already does that!

jglick · 2018-12-12T15:51:41Z

I cannot reproduce the error noted in #3772 running that single test, but weird behavior appears once you start running other tests too:

mvn -f test surefire:test -Dtest=DisablePluginCommandTest,JenkinsBuildsAndWorkspacesDirectoriesTest,NodesTest,TransientActionFactoryTest -Pall-tests

[INFO] Running hudson.cli.DisablePluginCommandTest
[INFO] Running jenkins.model.JenkinsBuildsAndWorkspacesDirectoriesTest
[INFO] Running hudson.cli.DisablePluginCommandTest
..
[INFO] Results:
[INFO] 
[WARNING] Flakes: 
[WARNING] hudson.cli.DisablePluginCommandTest.canDisableDependentPluginWrongOrderStrategyAll(hudson.cli.DisablePluginCommandTest)
[INFO]   Run 1: PASS
[ERROR]   Run 2: DisablePluginCommandTest.canDisableDependentPluginWrongOrderStrategyAll
[INFO]   Run 3: PASS
[INFO] 
[WARNING] hudson.cli.DisablePluginCommandTest.disablePluginsStrategyAll(hudson.cli.DisablePluginCommandTest)
[ERROR]   Run 1: DisablePluginCommandTest.disablePluginsStrategyAll » Reactor java.lang.Error: ...
[INFO]   Run 2: PASS
[INFO]   Run 3: PASS
[INFO] 
[WARNING] jenkins.model.JenkinsBuildsAndWorkspacesDirectoriesTest.externalBuildDirectorySymlinks(jenkins.model.JenkinsBuildsAndWorkspacesDirectoriesTest)
[ERROR]   Run 1: JenkinsBuildsAndWorkspacesDirectoriesTest.externalBuildDirectorySymlinks
[INFO]   Run 2: PASS
[INFO] 
[INFO] 
[WARNING] Tests run: 65, Failures: 0, Errors: 0, Skipped: 0, Flakes: 3

jglick · 2018-12-12T15:54:23Z

I think the problem is that the test JVM is getting a UnixLifecycle. Using RestartableJenkinsRule is fine since the restarts are purely in application code.

jglick · 2018-12-12T17:24:34Z

As an aside, I just noticed that a59814e puts the burden of calling cleanUp on each Lifecycle implementation, which seems silly—you would expect this to be part of Jenkins.restart/safeRestart.

MRamonLeon

Do you mean use RestartableJenkinsRule in the DisablePluginCommandTest and remove the Ignores @jglick? I didn't use it because EnablePluginCommandTest was so and I didn't need to do things on Jenkins after the restart.

jglick · 2018-12-12T19:22:50Z

Do you mean use RestartableJenkinsRule in the DisablePluginCommandTest

No, because RestartableJenkinsRule is used when the restart is initiated by the test directly (by means of separating it into blocks), whereas these tests are calling some code which in turn calls Jenkins.restart outside of the test’s direct control.

…ducible locally). Amends code added in jenkinsci#2082. java.lang.RuntimeException: Unexpected issues encountered during cleanUp: offset 0, count -1, length 5 at jenkins.model.Jenkins.cleanUp(Jenkins.java:3319) at org.jvnet.hudson.test.JenkinsRule.after(JenkinsRule.java:495) at org.jvnet.hudson.test.JenkinsRule$1.evaluate(JenkinsRule.java:565) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.lang.StringIndexOutOfBoundsException: offset 0, count -1, length 5 at java.base/java.lang.String.checkBoundsOffCount(String.java:3304) at java.base/java.lang.String.<init>(String.java:465) at hudson.TcpSlaveAgentListener$PingAgentProtocol.connect(TcpSlaveAgentListener.java:416) at hudson.TcpSlaveAgentListener.shutdown(TcpSlaveAgentListener.java:196) at jenkins.model.Jenkins._cleanUpShutdownTcpSlaveAgent(Jenkins.java:3548) at jenkins.model.Jenkins.cleanUp(Jenkins.java:3294) ... 6 more

batmat

LGTM, let's merge once we get a green build.

I'll see to be if we can analyze all these through https://issues.jenkins-ci.org/browse/JENKINS-55122 once we get the situation back in control.

batmat · 2018-12-12T23:13:28Z

Jenkinsfile

                stage("${buildType} Publishing") {
                    if (runTests) {
                        junit healthScaleFactor: 20.0, testResults: '*/target/surefire-reports/*.xml'
+                        archiveArtifacts allowEmptyArchive: true, artifacts: '**/target/surefire-reports/*.dumpstream'


interesting, wasn''t aware of this feature. NIT then reading https://maven.apache.org/surefire/maven-surefire-plugin/faq.html#dumpfiles don't we actually want .dump in case?

batmat · 2018-12-12T23:18:59Z

core/src/main/java/hudson/TcpSlaveAgentListener.java

                                    socket.getRemoteSocketAddress(),
                                    new String(ping, "UTF-8"),
-                                    new String(response, 0, responseLength, "UTF-8")
+                                    responseLength > 0 && responseLength <= response.length ?


FTR this fix is for UsageStatisticsTest last failure:

batmat · 2018-12-29T08:14:21Z

@LinuxSuRen in short, if you need that very fix, there's none.
Where are you seeing it? Thanks

LinuxSuRen · 2018-12-29T08:19:18Z

During test junit test of all my plugins. I wonder that shall we should fix this in the older LTS? @batmat

batmat · 2018-12-29T17:35:15Z

You mean when running your plugin's builds using Java 11? Sorry I am trying to make sure I get it.

Anyway if you have a case for backporting this, this will need a JIRA labelled 'lts-candidate' and explanarions as to why it would be backported.

Cheers

LinuxSuRen · 2018-12-30T04:25:40Z

I run my junit test from the IDEA. Java version is 8.

batmat · 2018-12-30T09:35:55Z

Does it also fail on the command line using `mvn test` then? It will help know if it's a general issue or specific to IntelliJ. Le dim. 30 déc. 2018 à 05:25, Zhao Xiaojie <notifications@github.com> a écrit :

…

I run my junit test from the IDEA. Java version is 8. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3804 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AANqbSYQqGkL7i9G-4J0owv7Y-n9pB1zks5u-EBGgaJpZM4ZOqmv> .

LinuxSuRen · 2018-12-30T09:41:12Z

They're the same results. You can test it by using my repo. The link is here https://github.com/LinuxSuRen/alauda-devops-pipeline-plugin/tree/add-test-case.

batmat · 2018-12-30T12:48:31Z

@LinuxSuRen the failure I see locally seems totally unrelated, just the typical issue with a wrong version of a plugin that prevents Jenkins from starting up.

[INFO] Running InjectedTest
[ERROR] Tests run: 23, Failures: 0, Errors: 5, Skipped: 0, Time elapsed: 7.448 s <<< FAILURE! - in InjectedTest
[ERROR] testPluginActive(org.jvnet.hudson.test.PluginAutomaticTestBuilder$OtherTests)  Time elapsed: 0.006 s  <<< ERROR!
java.lang.Error: Plugin workflow-basic-steps failed to start
...

I would recommend the following:

let's stop commenting here as it creates noise for people subcribed here, and if anyway related we should discuss this in the PR, see the next point
file https://github.com/LinuxSuRen/alauda-devops-pipeline-plugin/tree/add-test-case as a PR against https://github.com/jenkinsci/alauda-devops-pipeline-plugin/ so we can check how it behaves there.
** We'll be able to use comments there to discuss your specific issue
** You really should update to the latest parent pom, this issue I'm seeing is now impossible with latest plugin parent pom with RequireUpperBoundDeps rule enabled.

Thanks

Seeing if this helps.

d121e08

jglick added the work-in-progress The PR is under active development, not ready to the final review label Dec 12, 2018

jglick requested a review from batmat December 12, 2018 03:29

batmat closed this Dec 12, 2018

batmat reopened this Dec 12, 2018

batmat mentioned this pull request Dec 12, 2018

Analysis PR to check DisablePluginCommandTest flakiness #3808

Closed

jglick mentioned this pull request Dec 12, 2018

Disable DisablePluginCommandTest #3772

Closed

jglick commented Dec 12, 2018

View reviewed changes

jglick added 3 commits December 12, 2018 10:36

JenkinsBuildsAndWorkspacesDirectoriesTest also has a restart issue, i…

e3b1f32

…t seems.

Merge branch 'master' into timeout-diagnosis

a701c1a

Let us again try all matrix variants.

55ce7a6

Blocking Lifecycle.restart from within JenkinsRule.

208b42a

jglick changed the title ~~[DO NOT MERGE] Timeout diagnosis~~ Fixing tests which attempted to restart the Surefire booter JVM Dec 12, 2018

jglick removed the work-in-progress The PR is under active development, not ready to the final review label Dec 12, 2018

jglick requested review from MRamonLeon and daniel-beck December 12, 2018 16:36

MRamonLeon approved these changes Dec 12, 2018

View reviewed changes

jglick mentioned this pull request Dec 12, 2018

Java 11 test failure: no need to log at WARNING #3811

Closed

batmat approved these changes Dec 12, 2018

View reviewed changes

oleg-nenashev approved these changes Dec 12, 2018

View reviewed changes

batmat added the ready-for-merge The PR is ready to go, and it will be merged soon if there is no negative feedback label Dec 13, 2018

jglick merged commit b826edb into jenkinsci:master Dec 13, 2018

jglick deleted the timeout-diagnosis branch December 13, 2018 16:39

Uh oh!

Fixing tests which attempted to restart the Surefire booter JVM #3804

Fixing tests which attempted to restart the Surefire booter JVM #3804

Uh oh!

Conversation

jglick commented Dec 12, 2018

Uh oh!

batmat commented Dec 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jglick commented Dec 12, 2018

Uh oh!

jglick commented Dec 12, 2018

Uh oh!

jglick Dec 12, 2018

Choose a reason for hiding this comment

Uh oh!

batmat Dec 12, 2018

Choose a reason for hiding this comment

Uh oh!

jglick Dec 13, 2018

Choose a reason for hiding this comment

Uh oh!

jglick commented Dec 12, 2018

Uh oh!

jglick commented Dec 12, 2018

Uh oh!

jglick commented Dec 12, 2018

Uh oh!

jglick commented Dec 12, 2018

Uh oh!

MRamonLeon left a comment

Choose a reason for hiding this comment

Uh oh!

jglick commented Dec 12, 2018

Uh oh!

batmat left a comment

Choose a reason for hiding this comment

Uh oh!

batmat Dec 12, 2018

Choose a reason for hiding this comment

Uh oh!

batmat Dec 12, 2018

Choose a reason for hiding this comment

Uh oh!

batmat commented Dec 29, 2018

Uh oh!

LinuxSuRen commented Dec 29, 2018

Uh oh!

batmat commented Dec 29, 2018

Uh oh!

LinuxSuRen commented Dec 30, 2018

Uh oh!

batmat commented Dec 30, 2018 via email

Uh oh!

LinuxSuRen commented Dec 30, 2018

Uh oh!

batmat commented Dec 30, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

batmat commented Dec 12, 2018 •

edited

Loading

batmat commented Dec 30, 2018 •

edited

Loading