hook OSGi/eclipse bundle loaders into jpype.imports ?

[froh]

Hi,

I'm successfully and happily so controlling an eclipse application from python with JPype :-)

Now Eclipse uses and wraps OSGi, which has its own idea of class loading. Specifically the JVM class loader only knows about the absolute bare necessities (org.eclipse.equinox.launcher_*jar and a tiny backgroundstarter-*.jar) and the eclipse and OSGi set up their own "bundle" search mechanisms and their own per-bundle class loaders.

That leads to ugly code like this to load some classes:

bundleSymbolicName = "...."
B = Platform.getBundle(bundleSymbolicName)
packageName = "..."
SomeClass= jpype.JClass(
    B.loadClass(packageName + ".SomeClass")
)

...

x=SomeClass()
x.doSomething() ...

now I wonder if someone has ever struggled with this before and has figured how to inject OSGi bundles and their loaders into jpype.imports

any pointers appreciated

[Thrameos]

I can't say much about OSGi. Though one would need to tap into some kind of directory structure like found in a jar file (the jar doesn't need to be loaded to be accessable, we just need to be able to get to the zip directory to know what classes are available).

The issue of course would be that one would need to install hooks in jpype jar to recognize the concept of adding a bundle (using reflection so that it can operate without the jar dependency) and then publish this such that the jpype imports sees the directory when in recieves a dir call.

[froh]

bundles do have some instrospection support, like enumerating 'ressources' in there, especially resources loadable via the (hidden) class loader.

So I could indeed hook into that, that's good news.

However im a bit at a loss how to handle the bundle namespaces. each bundle has it's own namespace and it's own class loader.

so I'd need to actually do something like

# note: in no universe is this valid python syntax yet:
from some.bundle.SymbolicName import some.package.SomeClass [as ...]

as that can't work, I could go this route:

# note: in no universe is this valid python syntax yet:

from jpype.imports.OSGi import OSGiLoader

OSGiLoader.switchTo("some.bundle.SymbolicName")
import some.package.SomeClass

hm

or I could do a two step import, akin to this:

# note: in no universe is this valid python syntax yet:

from jpype.imports.OSGi import OSGiLoader

import bundles.some.bundle.SymbolicName # this enters that namespace for subsequent package imports
import packages.some.package.SomeClass

# or maybe even

import bundles.some.bundle.SymbolicName as da_bundle

import da_bundle.some.package.SomeClass
from da_bundle.some.package import SomeOtherClass, MORE_STUFF, even_more

btw I need to have a similar conversation about the fantastic stubgenj with @michi42. there, too, I need to figure how to make it work with OSGi bundles. fun :-)

anyhow, what I get is that so far to your radar this hasn't been tried and I'm on uncharted territory?

[michi42]

For what concerns stubgenj, I rely on the the JPype.JPackage objects and the fact they expose an iterable API to enumerate the contained JClasses to generate the type stubs.

Once you manage to obtain a JPackage object from your bundle, you can do

pkg = JPackage(...)

from stubgenj import generateJavaStubs
generateJavaStubs([pkg])

[Thrameos]

Jpype imports is just a front end of JPackage. Thus if JPackage is aware of bundles then imports would be as well.

[froh]

@michi42, @Thrameos, this is very helpful thanks!

It leaves me with a few questions, most of them towards @Thrameos I think

class JPackage is fundamentally empty. the implementation sits in _JPackage in c++, right?

and that immediately deep dives through the JNI bridge and resolves the package on the Java side with the built in Java class loader.

if that is correct, then here is my first question :-) do you have some suggestions how I could "inject" the bundle class loader into JPackage, such that i can say package = JPackage(loader, name)?

I have found a stanza which gives me a class loader for a bundle, and I could provide that.

loader = bundle.adapt(BundleWiring.class).getClassLoader()

---

I'd then start with something similar to the "old" JPackage syntax:

ClassXY = JBundle("bundle_name").path.to.package.ClassXY

for this to work, JBundle(...) would return a 'special' JPackageLoader object (with the ... bundle class loader underneath)

and only further down the road, when and if that works, I'd then think about something along the lines of

from ... import bundles # <-- registers a 'bundles' top level sys.meta_path finder
import bundles.path.to.bundle.aaa  # <-- registers top level aaa with the JPype importer, with it's bundle class loader
import bundles.path.to.other.bundle.aaa as a2
import aaa.some.package.name.ClassX # <-- uses the ' aaa' bundle importer...
import a2.some.package.name.ClassY

---

I think my alternative is to replicate JPackage 'light', and create this JBundle and the JPackageLoader bi meticulously studying jpype.JPackage.

opinions?

[Thrameos]

I would recommend just using the same JPackage mechanism. As you noted it isn't a real class, just a front end for Python that quickly reaches Java side. If you add a hook for adding a bundle into the Java side pulling from an environmental variable or talking directly. If you provide the directory info, and we tell the class lookup (also in Java) then bundles become automatic. If you can prototype it then we just have to convert it to by reflection (which breaks it from being dependent) then I can include it.

[Thrameos]

More specifically, have getPackage check with a BundleManager to see if the path goes to a bundle. Make JPypeBundle implement the same interface or inherit from JPypePackage. The Bundle9Manager checks for the Bundle classes by reflection , if the Bundle support is missing then it just will returns. Else it calls the probe methods again using reflection. Then all the mechanisms in JPype should just work.

There is one other place the hook has to connect to. That is the find class by name hook. That would see the name is in the Bundle and look up the classloader rather than contacting the Dynamic class loader.

The changes are entirely in the Java side. You need no C++ code. If you have to talk to theJPypeBundleManager, you just fetch org.jpype.bundle.JPypeBundleMsnager.getInstance() from Python. No need for custom hooks.

[astrelsky]

However im a bit at a loss how to handle the bundle namespaces. each bundle has it's own namespace and it's own class loader.

For the namespace problem you could probably try something like this.

https://github.com/dod-cyber-crime-center/pyhidra/blob/c878e91b53498f65f2eb0255e22189a6d172917c/pyhidra/launcher.py#L75-L91

You can then use sys.meta_path.append and then import it however you decide. If I remember correctly, if you can get the java.lang.Package from java then you'll have a JPackage to use.

I think you might be able to use this to completely sidestep the need to tinker with JPackage all together. I do not fully comprehend the issue at hand enough to know for sure though.

[Thrameos]

Messing with a few classes in org.jpype will make it work transparently to the user. Workarounds are also an option, it is just a matter of how much effort one wants to put in.