Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to use JPype on IBM I series #719

Open
Devender2806 opened this issue Apr 29, 2020 · 117 comments
Open

Unable to use JPype on IBM I series #719

Devender2806 opened this issue Apr 29, 2020 · 117 comments
Labels
question Question on how to use JPype

Comments

@Devender2806
Copy link

Dear Team,

We are attempting to install python 3.6 64 bit on IBM I server (PACE,AIX) so that we can connect to Oracle database. I started to work with JDBC thin driver concept first hence downloaded and installed jaydebeapi and JPype1 using the pip3 install command. The libraries installed fine, but when i initiate a connection i get core dump. I tried reading the core file with java tools and all it tells me is no JRE.

I can see java and javac executable files in the one of the directories and when i type java or javac it displays list of options. I also tried giving the path of libjvm.so file but doesn't help. I setup the JAVA_HOME and LIBPATH and PATH variables accordingly but i cannot get past the core dump. We have almost 7 JVMs running on the I series box currently.
@Thrameos
Copy link
Contributor

Thrameos commented Apr 29, 2020 •
edited
Loading

I haven't done much with that platform but lets first go through the basics.

  1. Write a simple hello world just starts JPype by itself without anything else.
import jpype
print (jpype.getDefaultJVMPath())
jpype.startJVM()
print(jpype.java.lang.System.getProperty('java.class.path'))

  1. Lets assume it crashes here. Okay the problem is with the shared library. So we need to verify that the shared library and the Python have the same architecture (both 64 bit). Assuming that is correct...

  2. Next we need to look for a dependency problem. Java has to find all of its required shared libraries when it loads. If something is missing then the JVM will fail to load. On some systems like Window 16 server that means adding the bin directory to the path. Use a library tool to verify that all libraries required are able to be found.

  3. If JPype does load by itself, the problem then the issue would have to be loading up a resource for JVM loading a native library. If this is the case then we would have to look at what resource that are loaded with that native library.

This is still pretty far from specific.

  1. What JVM vendor and version?
  2. Does JPype pass its self test?
  3. Can you produce a backtrace from the core file. (Please note that the JVM will scramble the core file on some systems. On those systems it is necessary to run the debugger rather than analyzing the core).
  4. What do you mean by "initiate a connection"? Is that after the JVM is started or when the JDBC driver is loaded?
  5. Is there a mismatch in shared library architecture between the JVM, Python and the JDBC library?
  6. If JPype does load the JVM what happens if you call forName on the JDBC driver?
  7. What version of JPype is being run?

@Thrameos Thrameos added the question Question on how to use JPype label Apr 29, 2020
@Devender2806
Copy link
Author

Devender2806 commented Apr 29, 2020 via email

@Thrameos
Copy link
Contributor

The error message reads like a bug in 0.7.2, but it may be a different problem. Are you currently up to date with the latest (0.7.4)?

I have a fairly good idea what is going on here. But we would need to get a bit more diagnostics before I can be sure. I have never tried to run on an IBM JVM. Most likely one of the commands in the file jp_context startJVM is producing an unexpected result. That is yielding an exception in a place where we do not yet have the ability to handle it cleanly.

So what we need to do is compile with --enable-tracing. On my systems I used the following

  • Download a clean copy from git
  • Run python setup.py --enable-tracing build_makefile. This should give you a makefile version which will allow you to quickly compile.
  • Run make to create a version of the code in place.
  • Run python setup.py test_java which creates the test bench for later steps.
  • Run to snippet.

Find the point in the trace with the fail. If there is no instrumentation in the area, add JP_TRACE() commands to isolate the failed call. This will hopefully get us to the source.

@Devender2806
Copy link
Author

Devender2806 commented Apr 29, 2020 via email

@Thrameos
Copy link
Contributor

Not much help I can be with that error. It appears to be site problem. Assuming that you can get python working I will try to be of further assistance.

@Devender2806
Copy link
Author

Devender2806 commented Apr 30, 2020 via email

@Thrameos
Copy link
Contributor

Thrameos commented Apr 30, 2020 via email

@Devender2806
Copy link
Author

Devender2806 commented Apr 30, 2020 via email

@Thrameos
Copy link
Contributor

Thrameos commented Apr 30, 2020 via email

@Thrameos
Copy link
Contributor

Lets start with something simple...

When I run the script to test tracing

import _jpype
_jpype.trace(0)
import jpype
_jpype.trace(255)
jpype.startJVM()
_jpype.trace(0)

The tracing program should be run at the top level of the jpype repo that you have build using
python setup.py --enable-tracing build_makefile; make.

I get an output like. Run this same command on your system and we can see what we get.

id="< PyInit__jpype
> PyJPModule_startup
  > JPPyUnicode::asStringUTF8
    PY: JPPyUnicode::asStringUTF8: pyref new(claim) 0x7fb36afedab0 1 bytes
    PY: JPPyUnicode::asStringUTF8: pyref dtor(dec) 0x7fb36afedab0 1 bytes
  < JPPyUnicode::asStringUTF8
  PyJPModule_startup: vmpath /usr/lib/jvm/java-11-openjdk-amd64/lib/server/libjvm.so
  PY: PyJPModule_startup: pyref new(inc) 0x7fb36e840040 10555 tuple
  PY: PyJPModule_startup: pyref new(claim) 0x7fb36c2cd360 21 module
  PY: PyJPModule_startup: pyref new(claim) 0x7fb36c2a5140 4 list
  PY: PyJPModule_startup: pyref new(claim) 0x7fb36d41aef0 3 builtin_function_or_method
  PY: PyJPModule_startup: pyref dtor(dec) 0x7fb36d41aef0 4 builtin_function_or_method
  PY: PyJPModule_startup: pyref dtor(dec) 0x7fb36c2a5140 4 list
  PY: PyJPModule_startup: pyref dtor(dec) 0x7fb36c2cd360 21 module
  > JPContext::startJVM
    + JNI: JPContext::startJVM: JavaFrame
      > JPException_init
        JNI: JPException_init: JPJavaFrame::FindClass
        JNI: JPException_init: JPJavaFrame::GetMethodID
        JNI: JPException_init: JPJavaFrame::FindClass
        JNI: JPException_init: JPJavaFrame::GetMethodID
        JNI: JPException_init: JPJavaFrame::GetMethodID
        JNI: JPException_init: JPJavaFrame::GetMethodID
        JNI: JPException_init: JPJavaFrame::GetMethodID

@Devender2806
Copy link
Author

Devender2806 commented Apr 30, 2020 via email

@Thrameos
Copy link
Contributor

Excellent. So now we have the problem isolated to this block of code. I am not sure why it segfaulted (but given this code is very hard to trigger it doesn't surprise me).

I am going to create a PR with some modifications with some additional diagnostics, so we can pin down the exact source of the error. You will need to

  • Repoint the origin of the git repo to the Thrameos repo version
  • Checkout the branch ibm
git remote set-url origin [email protected]:Thrameos/jpype.git
git fetch
git checkout ibm
  • Run python setup.py build_ext --makefile
  • Modify the makefile to remove dependency includes again.
  • You may need to delete the build directory as we don't have working dependencies .
  • Run make
  • Rerun the trace test program.

And please bare with me as we may take one or shots to pin down the issue.

	JP_TRACE_IN("JPContext::startJVM");

	m_ConvertStrings = convertStrings;

	// Get the entry points in the shared library
	loadEntryPoints(vmPath);

	// Pack the arguments
	JavaVMInitArgs jniArgs;
	jniArgs.options = NULL;

	// prepare this ...
	jniArgs.version = USE_JNI_VERSION;
	jniArgs.ignoreUnrecognized = ignoreUnrecognized;

	jniArgs.nOptions = (jint) args.size();
	jniArgs.options = (JavaVMOption*) malloc(sizeof (JavaVMOption) * jniArgs.nOptions);
	memset(jniArgs.options, 0, sizeof (JavaVMOption) * jniArgs.nOptions);
	for (int i = 0; i < jniArgs.nOptions; i++)
	{
		jniArgs.options[i].optionString = (char*) args[i].c_str();
	}

	// Launch the JVM
	JNIEnv* env;
	CreateJVM_Method(&m_JavaVM, (void**) &env, (void*) &jniArgs);
	free(jniArgs.options);

	if (m_JavaVM == NULL)
	{
		JP_TRACE("Unable to start");
		JP_RAISE(PyExc_RuntimeError, "Unable to start JVM");
	}

@Devender2806
Copy link
Author

Devender2806 commented Apr 30, 2020 via email

@Thrameos
Copy link
Contributor

Thrameos commented Apr 30, 2020 via email

@Devender2806
Copy link
Author

Devender2806 commented Apr 30, 2020 via email

@Devender2806
Copy link
Author

Devender2806 commented Apr 30, 2020 via email

@Thrameos
Copy link
Contributor

Yes you can down load a branch from Thrameos/jpype.

Just select the branch and then hit download.

image

@Devender2806
Copy link
Author

Devender2806 commented Apr 30, 2020 via email

@Devender2806
Copy link
Author

Devender2806 commented Apr 30, 2020 via email

@Thrameos
Copy link
Contributor

Thrameos commented Apr 30, 2020 •
edited
Loading

You still seem to have basic platform issues.

For example the line below shows that you have the JAVA_HOME pointing to a JRE not a JDK. JRE is the runtime not the build location. I would guess that jdk80 or jdk80/64bit is your JAVA_HOME, but without viewing the system it is only a guess.

"/QOpenSys/QIBM/ProdData/JavaVM/jdk80/64bit/jre/lib/ppc64/j9vm" does not
provide jni.h
' JAVA_HOME "%s" does not provide jni.h' % java_home)

Java home is expected to contain something like: (important pieces are bin, include, and lib

bin  conf  docs  include  jmods  legal  lib  man  release

The JVM binary is looking for $JAVA_HOME/lib/server which should contain the shared library with the import piece being libjvm.so

Xusage.txt  classes.jsa  libjsig.so  libjvm.so

The PATH must contain $JAVA_HOME/bin which should contain something like where javac is required

jaotc      javac    jdb        jimage  jmap        jshell  keytool  rmiregistry
jar        javadoc  jdeprscan  jinfo   jmod        jstack  pack200  serialver
jarsigner  javap    jdeps      jjs     jps         jstat   rmic     unpack200
java       jcmd     jhsdb      jlink   jrunscript  jstatd  rmid

The header for java is pulled from $JAVA_HOME/include which contains something like, where jni.h is needed

classfile_constants.h  jdwpTransport.h  jni.h  jvmti.h  jvmticmlr.h  linux

If your system is vastly different in terms of the structure then you will need to modify the scripts. Unfortunately, without being able to see the structure of your JDK install I wouldn't be able to tell you how to begin.

@Devender2806
Copy link
Author

Devender2806 commented Apr 30, 2020 via email

@Thrameos
Copy link
Contributor

Isn't it at their web site. Unfortunately I don't know anything about AIX so I can't be of assistance.
My first guess given your path would be /QOpenSys/QIBM/ProdData/JavaVM/jdk80/ but that is only a guess.

https://developer.ibm.com/javasdk/support/aix-download-service/

@Devender2806
Copy link
Author

Devender2806 commented Apr 30, 2020 via email

@Thrameos
Copy link
Contributor

Sounds like I need to know what sys.platform is reporting. You will then need to go to setupext/platform.py and add a new section with the details for your machine type. You may be able to use linux as a sheleton.

elif sys.platform.startswith('linux'):
    platform_specific['libraries'] = ['dl']
    platform_specific['extra_compile_args'] = ['-g3', '-std=c++11']
    jni_md_platform = 'linux'

@Devender2806
Copy link
Author

Devender2806 commented Apr 30, 2020 via email

@Thrameos
Copy link
Contributor

So start by adding a section in setupext/platform.py

elif sys.platform.startswith('aix7'):
    platform_specific['libraries'] = ['dl']
    platform_specific['extra_compile_args'] = ['-g3', '-std=c++11']
    jni_md_platform = 'aix7'

(I am assuming it is similar to linux)

@Devender2806
Copy link
Author

Devender2806 commented May 1, 2020 via email

@Devender2806
Copy link
Author

Devender2806 commented May 1, 2020 via email

@Thrameos
Copy link
Contributor

Thrameos commented May 1, 2020 via email

@Devender2806
Copy link
Author

Devender2806 commented May 4, 2020 via email

@Devender2806
Copy link
Author

Devender2806 commented May 4, 2020 via email

@Thrameos
Copy link
Contributor

Thrameos commented May 4, 2020 via email

@Devender2806
Copy link
Author

Devender2806 commented May 4, 2020 via email

@Thrameos
Copy link
Contributor

Thrameos commented May 4, 2020 via email

@Devender2806
Copy link
Author

Devender2806 commented May 8, 2020 via email

@Devender2806
Copy link
Author

Devender2806 commented May 10, 2020 via email

@Thrameos
Copy link
Contributor

It could. You can try to connect to the JVM using a simple demo program. I know that the diagnostic information on a linked library is much better than the dlopen method. Perhaps it will give you some insight into what is going wrong in the library loading process.

@Devender2806
Copy link
Author

Devender2806 commented May 10, 2020 via email

@Thrameos
Copy link
Contributor

Thrameos commented May 10, 2020 via email

@Thrameos
Copy link
Contributor

I am going to try to set up the same script for Unix that I did in #729. Perhaps that would help find the issue.

@Thrameos
Copy link
Contributor

I added a similar debugging procedure in #739. Perhaps that would assist you in identifying the issue.

@Devender2806
Copy link
Author

Devender2806 commented May 10, 2020 via email

@Devender2806
Copy link
Author

Devender2806 commented May 10, 2020 via email

@Thrameos
Copy link
Contributor

Thrameos commented May 10, 2020 via email

@Devender2806
Copy link
Author

Devender2806 commented May 11, 2020 via email

@Devender2806
Copy link
Author

Devender2806 commented May 12, 2020 via email

@Devender2806
Copy link
Author

Devender2806 commented May 13, 2020 via email

@Thrameos
Copy link
Contributor

Thrameos commented May 13, 2020 via email

@Devender2806
Copy link
Author

Devender2806 commented May 13, 2020 via email

@Devender2806
Copy link
Author

Devender2806 commented May 13, 2020 via email

@Thrameos
Copy link
Contributor

Thrameos commented May 13, 2020 via email

@Devender2806
Copy link
Author

Devender2806 commented May 19, 2020 via email

@Thrameos
Copy link
Contributor

Not that I am aware of. My only suggestion would be to copy the JVM and required libraries into a user space so that you can try to assemble a working version. But that may not possible on AIX.

@LeoBarbosa76
Copy link

Hi, did you find a way to work? I’m having the same issue. python code trying to use jpype to write a simple hello world didnt work.

I’m on an AIX 7.1 box.
There is a jdk160_35 installed and lot of other java.
usr/java7_64 too

I could start the JVM machine but this simple code doesnt work and dump:
Java.lang.System.out.println(“hello”)

How should I set the vars: java_home, libpath, classpath? Any other var needs to be set?
Any suggestions?

@Thrameos
Copy link
Contributor

Unfortunately, Java 6 and Java 7 are both past service life for the majority of computers. The oldest version of Java that we currently support is Java 8. Many features of the Java were needed to achieve JPype and these older platforms are now very difficult to test and support. If you can upgrade to Java 8 then it should work. JPype 0.6.3 is likely the last version that will operate on older Java versions.

https://www.oracle.com/java/technologies/java-se-support-roadmap.html

@LeoBarbosa76
Copy link

I’ve downgrade the jpype to this version, but it didnt work either.
I will try to install a newer java to test.

How could I set the env vars?

@Thrameos
Copy link
Contributor

Thrameos commented Aug 1, 2020

Only variable is JAVA_HOME which should be set to the base directory for the JRE.

@LeoBarbosa76
Copy link

LeoBarbosa76 commented Aug 7, 2020 via email

@Thrameos
Copy link
Contributor

Thrameos commented Aug 7, 2020

We don't have a lot of testing with the IBM JVM or on AIX, so it may be a fairly simple problem. But you will likely need to debug it yourself as I lack access to that platform.

The failure appears to be an unhandled exception. There should be very few (if any places) that don't have an exception handling frame which should have directed the error to the console. I would start by launching jpype with startJVM to see if we can isolate where it went wrong. I would say that it is likely in the code in jp_context.cpp having to do with starting, launching or loading resources. Unfortunately that is only a guess until I have further information to isolate it. The process starts in Python in startJVM() in jpype/_core then proceeds to pyjp_module then to jp_context. It all of that succeeds it returns to jpype/_core to load all the resources.

About all I can do to help would be to scan through and see if there are any likely spots that lack an exception frame. As far as I am aware they all do. But perhaps this is a double failure. In such a case something unexpected fails during the start, and it arrives to handle the fail but gets another exception as the resources are not set up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question Question on how to use JPype
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Thrameos @Devender2806 @LeoBarbosa76