[JENKINS-37341] z/OS Unix System Services enablement

[OldNavi]

Hello all,

i've seen many attempts to enable durable task plugin on native z/OS unix system services, i had it running fine almost for one year - merging base master branch with my changes locally at my site and now decided to offer it. Could you please have a look if this enablement changes can be merged in the plugin codebase or advise what can i amend/change ?

[OldNavi]

Hi @jglick - is there any chance that this PR can be reviewed and accepted ?

[RamchandV]

@jglick waiting for this change , please review, @OldNavi thankyou for helping the community.

[jglick]

Is this intended to supersede #28 and/or #76?

[jglick]

IIUC this is about handling a system in which the native encoding is not a superset of ASCII, which as noted in jenkinsci/jep#173 is explicitly not supported for now. The issue is not about process output, which you could already explicitly request to be transcoded from the EBCDIC in jenkins-log.txt (or output.txt), but about the encoding of script.sh and jenkins-result.txt. Should these files always be written/read in system default encoding? That is probably the right move, but it is unsafe to do so without some sort of automated test—minimally with a commonplace non-UTF-8 encoding and demonstrating that non-ASCII content in script.sh is correctly interpreted; ideally with a non-ASCII-superset encoding and demonstrating that jenkins-result.txt is correctly interpreted, though I am not sure it is even possible to run a Linux Docker container in such an encoding and have it be honored by /bin/sh.

[jglick]

What does this have to do with z/OS support?

[OldNavi]

Well, this is due to weird IBM Java behavior on z/OS. Without flushing stream some of the job output isn't timely appears in a log. No side effects on OpenJDK or Sun Java.

[dwnusbaum]

Without this, does the output not appear at all, or does it just take a long time?

[jglick]

To be clear, I did already try to test UTF-16 as an encoding on a particular sh step using an existing test container, which predictably resulted in complete junk. I did not yet try to create a new test container natively set to UTF-16 and use it to just run regular stuff with ASCII output. Since no Pipeline devs have access to a z/OS environment to test anything interactively, much less in CI, that is the closest we could come.

[OldNavi]

A couple of notices why i had to encode script sh to EBCDIC on the z/OS USS platform. First thing - we couldn't rely on a system default encoding in z/OS Java because it doesn't reflect actual system encoding at all (by default z/OS IBM Java uses UTF8 BTW). Setting it with -Dfile.encoding=EBCDIC code page can avoid transcoding of script.sh but it is global setting and affects a lot of other things in a jenkins agent and plugins behavior. Technically there is a way to have a script.sh in ASCII charset on z/OS USS and execute it but this requires to call a z/OS chtag utility to tag script.sh as file required for transcoding ( https://www.ibm.com/support/knowledgecenter/SSLTBW_2.2.0/com.ibm.zos.v2r2.bpxa500/chtag.htm )
which again means that we need to create some script and execute beforehand. I'm using jenkins to build z/OS legacy code like PL/I and Assembler and another dark side of the moon is an output of programs running on MVS side of z/OS which can be called via script.sh but doesn't accept ASCII encoding in any way.
So in general i've tried to isolate z/OS USS behavior from other platforms.
i've also adopted git-client-plugin to work on z/OS USS - same issues are there.

[OldNavi]

Some comments why setting encoding at sh DSL is not helping on z/OS and i have to force encoding inside plugin (1-st reason is because writing script is only UTF-8 )

So running simple pipeline on z/OS with sh step set to IBM1047 encoding

We are getting the following output

It's clear that jenkins-result.txt can't be parsed because in STATUS_CHECK_INSTANCE invocation we don't do transcoding. in pull request i've changed it to honor charset from FileMonitoringTask

i personally don't like a check i've wrote here - maybe it is better to change a behavior and honor sh encoding setting when writing a script.sh - but it is a bold move, as we aren't going to transcode an output only but an input too
`
String encoding = os == OsType.ZOS ? "IBM1047" : null;
if(encoding != null) charset(Charset.forName(encoding));
ShellController c = new ShellController(ws);

.....

shf.write(s, encoding==null ? "UTF-8" : encoding);
shf.chmod(0755);`

[OldNavi]

For this PR to work it needs JEP-206 implemented so there is a prerequisite of Pipeline Nodes and Processes Plugin > 2.20

[jglick]

BTW it is pointless to work in this area until after #60 is merged.

stale

[OldNavi]

Hi Jesse,

Do you know any timeline when #60 is going to be merged, so i'll be able to continue to adopt my changes on top of it? We have just finished adopting git cli client to work on z/OS USS and merged into master jenkinsci/git-client-plugin#362 so half work of z/OS USS enablement for Jenkins is done. :-)

[RamchandV]

Hi Jesse,
Do you know any timeline when #60 is going to be merged, so i'll be able to continue to adopt my changes on top of it? We have just finished adopting git cli client to work on z/OS USS and merged into master jenkinsci/git-client-plugin#362 so half work of z/OS USS enablement for Jenkins is done. :-)

#60 is merged now

[OldNavi]

Work in progress. Not yet ready for merge.

[OldNavi]

@jglick - I refactored code just to be specific on z/OS platform only, trying to avoid changes which could potentially affect open platforms. Can you have a look if it is okay now ?

[gorbadoc]

I, too, am waiting for this. Any chance it can get merged in the next month or so?

I am the front of a wave that will (hopefully) push a bunch of CI into Jenkins if I can show developers that all they need to do is commit something and their development branch will get built across a bunch of platforms. This is already done on a limited basis for UNIX variants but is only set up for production builds.

What can I do to help expedite this? I used to be a contributor to Eclipse CDT and was a committer on coreboot.

[OldNavi]

Hi Devin,

Is there any chance to have a progress on this PR ?

[dwnusbaum]

@OldNavi Sorry that I have not had a chance to look yet. I will try to review the PR in the next few weeks. Part of the problem is that I am simply not familiar with z/OS, so it is difficult for me to review the changes, and without any tests I am concerned that we will regress the new functionality. When I have time to review I want to try to create a test using a Docker fixture for a container with a non-ASCII superset encoding (similar to tests like this one) so that we can have automated tests for at least some of the behavior.

[gorbadoc]

I have a z/OS system available to me. I have Ivan's patch installed and it works as expected. Perhaps I can run a test for you if you outline the test that needs to be run? I now have a monster makefile with recipes for several languages and am successfully compiling, assembling, and linking several hundred modules successfully in a multibranch pipeline. I have two branches running and will shortly, maybe today, expand to low tens of branches. Does that help?

…

On Tue, Jan 15, 2019, 7:17 AM Devin Nusbaum ***@***.*** wrote: @OldNavi <https://github.com/OldNavi> Sorry that I have not had a chance to look yet. I will try to review the PR in the next few weeks. Part of the problem is that I am simply not familiar with z/OS, so it is difficult for me to review the changes, and without any tests I am concerned that we will regress the new functionality. When I have time to review I want to try to create a test using a Docker fixture for a container with a non-ASCII superset encoding (similar to tests like this one <https://github.com/jenkinsci/durable-task-plugin/blob/d6258dd63384de5ec261e7374028fd6f4a106dd0/src/test/java/org/jenkinsci/plugins/durabletask/BourneShellScriptTest.java#L309>) so that we can have automated tests for at least some of the behavior. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#80 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Ar05EhZyHH2ZIPPa74Y1iiqIyQLb2evpks5vDeLrgaJpZM4V877C> .

[OldNavi]

and without any tests I am concerned that we will regress the new functionality. When I have time to review I want to try to create a test using a Docker fixture for a container with a non-ASCII superset encoding...

I've tried to wrap z/OS logic just not to interfere with open systems at all. As for testing, I was thinking about "emulating" z/OS behavior with Docker - but it seems to be not trivial as for open systems. Playing with System.setProperty("file.encoding", "IBM1047") in runtime has no effect, as it should be only set up at startup time with -Dfile.encoding... but setting that at start up time, completely breaks Jenkins agent behavior...
In general z/OS Java setup in z/OS is kind of mixed behavior in a difference for open systems... JVM itself runs with ASCII charset specified either UTF-8 or ISO8859-1 (which bidi maps to IBM1047). Shell scripts need to be in EBCDIC code pages (IBM1047 by default) so we do a transcoding... Shell return code and output also comes in EBCDIC charset.

[hogstrom]

@dwnusbaum Just a data point. I have a z/OS system on which I installed this fix and it correctly addressed the problem and I was able to build. I'm aware of a large number of z/OS customers that are incorporating Jenkins to improve their CI/CD pipeline and this is a needed fix for them.

I understand that you do not have access to a z/OS system. I offer to get you access to one and to take you through the system to improve your familiarity of the problem and how the platform works. I'm working on Project Zowe and we need this functionality to build open source software on Z as well. The system is on the Internet and we can spend whatever time is needed via WebEx if that would be a benefit to you. Contact information is in my Git Profile.

Thanks for your consideration on bringing this issues to a close.

@OldNavi Thanks for the work here ... very much appreciated.

[jglick]

I do not pretend to follow what this is doing, but with one smallish change, it can be made to not affect non-z/OS platforms at all, and I trust that z/OS users can figure out how to set up some CI system to check for regressions in public releases after the fact.

[jglick]

Thank you, all z/OS specifics finally seem to be fenced off cleanly!

[dwnusbaum]

Now that the z/OS behavior has been separated cleanly from other platforms in the recent changes I am happy to merge the PR since there is nothing that we could test here anyway.

Thanks everyone for the comments and for your interest in Jenkins Pipeline. To clarify my earlier comment, my point was not really about the correctness of this change (I trust that it works for you all), but that without automated testing I can make no guarantee that further changes to this plugin will not break the behavior on z/OS, since I simply do not have any time to dedicate to manually testing specific platforms. As Jesse said, I would highly recommend creating some kind of nightly test run to check for regressions on a z/OS system or similar if you are relying on this functionality.

Thanks for the PR and your patience @OldNavi!

[hogstrom]

@dwnusbaum Devin, as part of the Zowe project that is building z/OS Open Source we are creating instances to be used by open source projects for build testing. Although a small portion of the community, if we could help to provide and maintain an instance for the Jenkins community we'd be interested in helping out. Please let me know if there is interest in including this platform as part of the Jenkins testing.

[dwnusbaum]

@hogstrom I appreciate your offer, and perhaps someone else on this PR would be interested in looking into it, but I am not able to commit to driving that effort myself.

[ghost]

I just ran into this issue and found that the fix does NOT work on intel Jenkins servers. Just Unix based systems....

[OldNavi]

I just ran into this issue and found that the fix does NOT work on intel Jenkins servers. Just Unix based systems....

@nlopez59 - can you share more details ?

[gin-nader]

I'm hitting an issue where the jenkins-result.txt file is getting tagged as ASCII on a z/OS machine. This is causing the StatusCheckWithEncoding(getCharset()) check at

durable-task-plugin/src/main/java/org/jenkinsci/plugins/durabletask/BourneShellScript.java

Line 298 in 0d250a6

status = statusFile.act(new StatusCheckWithEncoding(getCharset()));

to throw an Exception.

Should I open a PR for the fix?