Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove macro redefinitions that cause issues with Open XL #7413

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

Deigue
Copy link
Contributor

@Deigue Deigue commented Jul 18, 2024

Couple instances where the toupper() and tolower() macros are undef and re-defined that cause problems with the native headers. These may be unnecessary and are removed.

@r30shah
Copy link
Contributor

r30shah commented Jul 26, 2024

jenkins build zos

@0xdaryl 0xdaryl self-assigned this Jul 30, 2024
@0xdaryl
Copy link
Contributor

0xdaryl commented Jul 30, 2024

Can you confirm this builds successfully with OpenJ9 (mainly because that is a broader test base than what OMR provides)?

@keithc-ca FYI since it is in DDR code.

@keithc-ca
Copy link
Contributor

That OMR build specifies -DOMR_DDR=OFF, so it doesn't exercise this change.
DDR tooling doesn't use tolower() or toupper(), so the declarations from the compile should be irrelevant.

I did notice that the macros in stricmp.c are just wrong for EBCDIC (letters are not contiguous).

@keithc-ca
Copy link
Contributor

Can you confirm this builds successfully with OpenJ9

I assume this question is for @Deigue.

@0xdaryl
Copy link
Contributor

0xdaryl commented Jul 30, 2024

I assume this question is for @Deigue.

Yes, the author.

@Deigue
Copy link
Contributor Author

Deigue commented Aug 7, 2024

j21 build actually seems like it is not passing with the changes

[2024-08-07T15:30:14.624Z] [  0%] Building CXX object runtime/j9vm31/CMakeFiles/jvm31.dir/jnimisc.cpp.o
[2024-08-07T15:30:15.039Z] "/usr/include/xlocale", line 673.33: CCN5816 (W) Too many arguments are specified for the macro "tolower". The extra arguments are ignored.
[2024-08-07T15:30:15.039Z] "/jenkins/workspace/Build_JDK21_s390x_zos_Personal/omr/cmake/modules/platform/os/../../../../util/a2e/headers/ctype.h", line 97.25: CCN5425 (I) "tolower" is defined on line 97 of "/jenkins/workspace/Build_JDK21_s390x_zos_Personal/omr/cmake/modules/platform/os/../../../../util/a2e/headers/ctype.h".
[2024-08-07T15:30:15.039Z] "/usr/include/xlocale", line 677.33: CCN5816 (W) Too many arguments are specified for the macro "toupper". The extra arguments are ignored.
[2024-08-07T15:30:15.039Z] "/jenkins/workspace/Build_JDK21_s390x_zos_Personal/omr/cmake/modules/platform/os/../../../../util/a2e/headers/ctype.h", line 96.25: CCN5425 (I) "toupper" is defined on line 96 of "/jenkins/workspace/Build_JDK21_s390x_zos_Personal/omr/cmake/modules/platform/os/../../../../util/a2e/headers/ctype.h".
[2024-08-07T15:30:15.039Z] "/usr/include/xlocale", line 806.33: CCN5816 (W) Too many arguments are specified for the macro "tolower". The extra arguments are ignored.
[2024-08-07T15:30:15.039Z] "/jenkins/workspace/Build_JDK21_s390x_zos_Personal/omr/cmake/modules/platform/os/../../../../util/a2e/headers/ctype.h", line 97.25: CCN5425 (I) "tolower" is defined on line 97 of "/jenkins/workspace/Build_JDK21_s390x_zos_Personal/omr/cmake/modules/platform/os/../../../../util/a2e/headers/ctype.h".
[2024-08-07T15:30:15.039Z] "/usr/include/xlocale", line 810.33: CCN5816 (W) Too many arguments are specified for the macro "toupper". The extra arguments are ignored.
[2024-08-07T15:30:15.039Z] "/jenkins/workspace/Build_JDK21_s390x_zos_Personal/omr/cmake/modules/platform/os/../../../../util/a2e/headers/ctype.h", line 96.25: CCN5425 (I) "toupper" is defined on line 96 of "/jenkins/workspace/Build_JDK21_s390x_zos_Personal/omr/cmake/modules/platform/os/../../../../util/a2e/headers/ctype.h".
[2024-08-07T15:30:15.039Z] "/usr/include/xlocale", line 804.23: CCN5063 (S) The text "_C" is unexpected.
[2024-08-07T15:30:15.039Z] [  0%] Linking C static library ../lib/libffi.a
[2024-08-07T15:30:15.039Z] [  0%] Generating NLS files
[2024-08-07T15:30:15.039Z] [  0%] Building CXX object runtime/omr/ddr/test/CMakeFiles/ddrgentest.dir/test.cpp.o
[2024-08-07T15:30:15.422Z] CCN0793(I) Compilation failed for file /jenkins/workspace/Build_JDK21_s390x_zos_Personal/omr/ddr/lib/ddr-ir/ClassType.cpp.  Object file not created.
[2024-08-07T15:30:15.422Z] [  0%] Copying schema/feature-v2.xsd to the build directory
[2024-08-07T15:30:15.422Z] make[6]: *** [runtime/omr/ddr/lib/ddr-ir/CMakeFiles/omr_ddr_ir.dir/build.make:75: runtime/omr/ddr/lib/ddr-ir/CMakeFiles/omr_ddr_ir.dir/ClassType.cpp.o] Error 12
[2024-08-07T15:30:15.422Z] make[5]: *** [CMakeFiles/Makefile2:4689: runtime/omr/ddr/lib/ddr-ir/CMakeFiles/omr_ddr_ir.dir/all] Error 2

@keithc-ca is this being caused by the redefinitions being missing, or is it pointing to something else? wasn't sure what Error 12 meant, and the main error I see is this: [2024-08-07T15:30:15.039Z] "/usr/include/xlocale", line 804.23: CCN5063 (S) The text "_C" is unexpected.

@babsingh
Copy link
Contributor

@Deigue What is the original error that you are trying to resolve with this fix?

@Deigue
Copy link
Contributor Author

Deigue commented Sep 23, 2024

Here is the snippet of the log encountered with Open XL (I had also contacted the cpp team internally and they mentioned that the tolower and toupper are inline functions in C++, and as such the overloading/scoping causes issues)

In file included from /omr/ddr/include/ddr/scanner/dwarf/DwarfScanner.hpp:28:
In file included from /xlc210/usr/lpp/IBM/cnw/v2r1/openxl/bin/../include/c++/v1/map:2193:
In file included from /xlc210/usr/lpp/IBM/cnw/v2r1/openxl/bin/../include/c++/v1/functional:526:
In file included from /xlc210/usr/lpp/IBM/cnw/v2r1/openxl/bin/../include/c++/v1/__functional/boyer_moore_searcher.h:27:
In file included from /xlc210/usr/lpp/IBM/cnw/v2r1/openxl/bin/../include/c++/v1/vector:325:
In file included from /xlc210/usr/lpp/IBM/cnw/v2r1/openxl/bin/../include/c++/v1/__format/formatter_bool.h:20:
In file included from /xlc210/usr/lpp/IBM/cnw/v2r1/openxl/bin/../include/c++/v1/__format/formatter_integral.h:35:
In file included from /xlc210/usr/lpp/IBM/cnw/v2r1/openxl/bin/../include/c++/v1/locale:202:
/xlc210/usr/lpp/IBM/cnw/v2r1/openxl/bin/../include/c++/v1/__locale:501:35: error: expected ')'
  501 |   _LIBCPP_HIDE_FROM_ABI char_type toupper(char_type __c) const { return do_toupper(__c); }
      |                                   ^
/omr/ddr/include/ddr/std/unordered_map.hpp:42:36: note: expanded from macro 'toupper'
   42 | #define toupper(c)     (islower(c) ? (c & _XUPPER_ASCII) : c)
      |                                    ^
/xlc210/usr/lpp/IBM/cnw/v2r1/openxl/bin/../include/c++/v1/__locale:501:35: note: to match this '('
/omr/ddr/include/ddr/std/unordered_map.hpp:42:24: note: expanded from macro 'toupper'
   42 | #define toupper(c)     (islower(c) ? (c & _XUPPER_ASCII) : c)
      |                        ^
  
/xlc210/usr/lpp/IBM/cnw/v2r1/openxl/bin/../include/c++/v1/__locale:501:57: error: expected ';' at end of declaration list
  501 |   _LIBCPP_HIDE_FROM_ABI char_type toupper(char_type __c) const { return do_toupper(__c); }
      |                                                         ^
/xlc210/usr/lpp/IBM/cnw/v2r1/openxl/bin/../include/c++/v1/__locale:503:68: error: too many arguments provided to function-like macro invocation
  503 |   _LIBCPP_HIDE_FROM_ABI const char_type* toupper(char_type* __low, const char_type* __high) const {
      |                                                                    ^
/omr/ddr/include/ddr/std/unordered_map.hpp:42:9: note: macro 'toupper' defined here
   42 | #define toupper(c)     (islower(c) ? (c & _XUPPER_ASCII) : c)
      |         ^

In file included from omr/ddr/lib/ddr-scanner/dwarf/DwarfScanner.cpp:22:
In file included from omr/ddr/include/ddr/scanner/dwarf/DwarfScanner.hpp:28:
In file included from /xlc210/usr/lpp/IBM/cnw/v2r1/openxl/bin/../include/c++/v1/map:2193:
In file included from /xlc210/usr/lpp/IBM/cnw/v2r1/openxl/bin/../include/c++/v1/functional:526:
In file included from /xlc210/usr/lpp/IBM/cnw/v2r1/openxl/bin/../include/c++/v1/__functional/boyer_moore_searcher.h:27:
In file included from /xlc210/usr/lpp/IBM/cnw/v2r1/openxl/bin/../include/c++/v1/vector:325:
In file included from /xlc210/usr/lpp/IBM/cnw/v2r1/openxl/bin/../include/c++/v1/__format/formatter_bool.h:20:
In file included from /xlc210/usr/lpp/IBM/cnw/v2r1/openxl/bin/../include/c++/v1/__format/formatter_integral.h:35:
In file included from /xlc210/usr/lpp/IBM/cnw/v2r1/openxl/bin/../include/c++/v1/locale:202:
/xlc210/usr/lpp/IBM/cnw/v2r1/openxl/bin/../include/c++/v1/__locale:596:68: error: too many arguments provided to function-like macro invocation
  596 |   _LIBCPP_HIDE_FROM_ABI const char_type* tolower(char_type* __low, const char_type* __high) const {
      |                                                                    ^
omr/ddr/include/ddr/std/unordered_map.hpp:43:9: note: macro 'tolower' defined here
   43 | #define tolower(c)     (isupper(c) ? (c | _XLOWER_ASCII) : c)
      |         ^

fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.

@Deigue Deigue force-pushed the openxl-macros branch 2 times, most recently from a8a8919 to 1e0dce1 Compare January 21, 2025 20:20
@Deigue
Copy link
Contributor Author

Deigue commented Jan 21, 2025

Restored back the original code, and added an (__open_xl__) conditional in order to avoid the redefining from taking place in Open XL scenarios, so that this code can work in all scenarios without issues ...
(Confirmed/tested new code is working with the latest open-xl jdk21 build.)

Couple instances where the toupper() and tolower() macros
are undef and re-defined that cause problems with the
native headers. These may be unnecessary and are removed.

Signed-off-by: Gaurav Chaudhari <[email protected]>
@@ -27,13 +27,16 @@
#if defined(J9ZOS390)

#include <ctype.h>
#if !defined(__open_xl__)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think util/a2e/headers/ctype.h should be defining toupper or tolower in which case there should be no need to #undef them anywhere.

A best, this is incomplete as the same pattern also appears in these files:

  • ddr/include/ddr/std/sstream.hpp
  • fvtest/omrGtestGlue/omrTest.h

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am seeing the undefs in (on top of ctype.h)

  • ddr/include/ddr/std/sstream.hpp
  • fvtest/omrGtestGlue/omrTest.h
  • ddr/include/ddr/std/string.hpp
  • ddr/include/ddr/std/unordered_map.hpp
  • fvtest/omrGtest.cpp
  • fvtest/omrGtestGlue

Is not needing the definition coming from the fact that you have noticed these redefinitions are not being used anywhere (like no usages seen in c files?)

What would be the best direction, removing the snippet from ctype.h here, and follow up with just removing the undefs as no-longer having clash with usr/include/xlocale ?

/*
 *  In ASCII, upper case characters have the bit off    ibm@4345
 */
#define toupper(c)     (islower(c) ? (c & _XUPPER_ASCII) : c)
#define tolower(c)     (isupper(c) ? (c | _XLOWER_ASCII) : c)

Or alternatively, consistently add the open-xl conditionals across these undef areas?

@Deigue
Copy link
Contributor Author

Deigue commented Mar 13, 2025

I tried removing all the redefinitions and undefs from everywhere,
see commit: 96985df

But even with Open XL , I am seeing some errors regarding xtrace options now:

JVMTRCE003E Error processing trace option, detail: Trace level required without an integer level specifier
JVMTRCE003E Error processing trace option, detail: Syntax error encountered at offset 8 in: MAXIMAL=all{level1}
JVMTRCE002E Error processing trace option: -Xtrace:MAXIMAL=all{level1}

+ exitcode=1
+ /bin/cp /jit/team/gauravc/repos/openj9-openjdk-jdk21-zos/build/zos-s390x-server-release/jdk/_optimize_image_exec.log /jit/team/gauravc/repos/openj9-openjdk-jdk21-zos/build/zos-s390x-server-release/make-support/failure-logs/jdk__optimize_image_exec.log
+ /bin/cp /jit/team/gauravc/repos/openj9-openjdk-jdk21-zos/build/zos-s390x-server-release/jdk/_optimize_image_exec.cmdline /jit/team/gauravc/repos/openj9-openjdk-jdk21-zos/build/zos-s390x-server-release/make-support/failure-logs/jdk__optimize_image_exec.cmdline
+ exit 1
make[3]: *** [ExplodedImageOptimize.gmk:41: /jit/team/gauravc/repos/openj9-openjdk-jdk21-zos/build/zos-s390x-server-release/jdk/_optimize_image_exec.marker] Error 1
make[3]: Leaving directory '/jit/team/gauravc/repos/openj9-openjdk-jdk21-zos/make'
make[2]: *** [make/Main.gmk:492: exploded-image-optimize] Error 1
make[2]: Leaving directory '/jit/team/gauravc/repos/openj9-openjdk-jdk21-zos'
+ exitcode=2
+ printf '\nERROR: Build failed for target '\''images'\'' in configuration '\''zos-s390x-server-release'\'' (exit code 2) \n'
+ /bin/tee -a /jit/team/gauravc/repos/openj9-openjdk-jdk21-zos/build/zos-s390x-server-release/build.log

ERROR: Build failed for target 'images' in configuration 'zos-s390x-server-release' (exit code 2)

When I had only removed from string.hpp and unordered_map.hpp the compilation had no issues, was wondering if I made any erroneous edits in my commit above, or if there is actually a problem removing one of these definitions, that is leading to some option here not being parsed in the expected manner.

(Expecting it to certainly fail with the xlc builds if the open-xl build is having issues in the first place too, so trying to resolve this first before cherry picking over to my xlc local build)

@keithc-ca
Copy link
Contributor

JVMTRCE002E Error processing trace option: -Xtrace:MAXIMAL=all{level1}

This is a sign that isdigit() is not behaving as expected (see trccomponent.c).

The compiler should be providing an implementation of isdigit() that is consistent with the interpretation of character and string literals (e.g. if *tempstr == '{' considers the brace in ASCII, then so should isdigit() consider its argument to be ASCII). This is an issue probably best addressed by the compiler team.

@Deigue
Copy link
Contributor Author

Deigue commented Mar 19, 2025

Just adding a small update here, while I am testing changes offline.

After seeing that the isdigit function is not operating correctly after removing all the #define and redefs in the a2e headers, the compiler team has recommended utilizing the __isdigit_a() function, which requires #define _AE_BIMODAL . This function is ascii version of isdigit that can correctly recognize the ascii byte decimals.

Tests build was done with XLC Compiler as well, which also failed when removing all the defines.
As such the changes/suggestions above apply for both compilers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants