Start jpype from Java if only to improve the workflow

[apatrida]

When developing with jpype and you are working on both sides of the fence, having the Python side start the JVM each time is nice in the way that both resync at the same time to new versions of code. But, it is bad in terms of workflow because it requires all the Java code to be packaged up each build (into a JAR), and the classpath to be known (already is active on the Java side) which means extracting that from a Maven or Gradle build to avoid hard coding it.

The overall workflow of the Java code becomes tiresome in this model. Starting jpype from the Java side and using the existing JVM fixes this and removes both the packaging and classpath issues.

Is this on the radar for anytime soon, I see it mentioned in the FAQ of course.

[Thrameos]

Yes, it is on the work flow. The general plan is to start Python form within Java like you would any other scripting engine. There are a few items that are on the work plan that were considered to be prerequisites.

• Export the Python API to Java. This is a task of just wrapping everything from the Python Object API as a JNI call. Mostly autogenerated code. (I have done a bit of this in a test branch, but nothing ready to ship)
• Remove the bootstrap loader and instead install org.jpype.jar with the internal module shared library. The concept of using the thunks was a good way to help with packaging but it has its limits. Since the JPype jar is not exposed it is hard to make it useable from the outside. This is more a bit of packaging magic rather than coding.
• Construct the attach point for the module. For some odd reason we have had an attach to JVM command in here since the beginning, but there was never any way to execute it. The whole code surrounding it was just broken do it have been excised. A suitable replacement needs to be added, but as it is only called from JVM side it doesn't need to be in the Python API.
• Last deal with module portability. With Python as the script engine, you can start multiple interpreters, which means the central module has to be written in a particular fashion (avoiding globals and the like.) I have been tried this twice before, but each time I have had to back off because it prevents progress for the overall core rewrite. However, that rewrite is now complete so it can likely once again advance. The good news is I have plenty of abandoned copies of this to pull from, but until the other piece are in place it won't do much good.

I have been hoping that Project Panama would be included in the JVM which would make this task much simpler but appears still to be on the horizon far out of reach. Thus I am just going to have to bite the bullet and got at it with JNI.

The current plan is to attack each of these in order until full support is complete. So it is definitely on the radar likely sometime after the next release.

[Thrameos]

(there is some similarity to our development and that of insect wings. The intermediate stages have to be something useful or other priorities just eat up the time and you never reach the sky.)

[apatrida]

There are limited versions of the approach that could work:

Don't bite off all of the JSR scripting engine support at once, like line by line evaluation. You can get it to work as a direct API, and wrap as a script engine later. (We did this with Kotlin as a scripting engine)

Don't worry about more than one instance in a VM on day 1, if people need this, maybe they can live with one instance of Python (I can for example). Later they can get multiple instances. With threads and separation into different class instances, not sure what you could not find a way to workaround.

You know your codebase way better than me, I know the JVM, but things outside and native libraries I haven't touched in forever (I used JNI the first years it existed and not since).

You have a good library, keep it up! If I can contribute, I will, but this particular area is a bit outside of what would be productive for me.

[Thrameos]

I made a good push on this one last week. I resolved about 80% of the issues for setting a up a reverse bridge and even managed to get an IPython console running from within Java. But the magnitude of typing that would be required for integrated code at least 2 months out and well past the 0.8 release. I will post a task list for this once I have finished the 0.8 release.

[pelson]

This topic came up today for me, and I'm curious if you've made any further progress on it @Thrameos? It would indeed be very interesting to start a CPython interpreter from within the JVM. Do you have a rough idea of the timeline we might be talking about to see such a feature (not wanting to apply any pressure here, just curious!)

[Thrameos]

There is a "mostly" working copy of this on the epypj fork of my repo. Unfortunately it is a "shit-ton" of typing to get every module of Python tested.

Current state:

• reverse bridge starting of an interactive shell WORKS
• automatic building of Python class wrappers WORKS
• FFI calling mechanism WORKS but still some random crashes on bad entry points that need to be resolved.
• Testing framework is in place but not connected to CI, many tests need to be added and then debugged.
• Still need to resolve future proofing issues. How do we expose whole modules like numpy beyond simple objects? How should exceptions be creatable from within Java (stack frame merging issues) How do we avoid memory loops on complex structures?

I have been putting about 2 days in each month towards this. But the size of the work is pretty large and unless my boss at work says I can perform the work during day hours (or some nice user wants to help me push it forward ) it will be another few months.

If the third and fourth item are resolved it would be usable though I really need a lot of user input in terms of review and documentation. The complicating factor is which jpype is production quality, this fork is closer to alpha. Getting to the same level will take time. I debated putting epypj as a project fork which exists just for testing purposes until I get enough feedback to push it to the same quality level prior to merging it back in, but the path is very unclear.

[cowwoc]

Hello from December 2022. I'm also very interested in this feature. Has there been any further progress?

[cowwoc]

I'm curious what the difference is between this feature request and "JPype for Android". Specifically, shouldn't we be able to launch Java then (somehow) invoke Python code that accepts a reference to the existing JVM?

[Thrameos]

There is certainly a difference. In JPype for android we are just making access to the existing JVM which was started by the system. The request here is for Java to launch Python which means locating Python, creating the Python kernel then transferring control.

[rsjeffers]

Hello, I'm also very interested in this feature, has more progress been made since the last comment? Thank you in advance :)