Refactor JImplements as a metaclass

[KOLANICH]

def _Implements(className: str, parents: typing.Tuple[typing.Type, ...], attrs: typing.Dict[str, typing.Any]):
	interface = parents[0]
	res = type(className, (), attrs)
	dec = jpype.JImplements(interface)
	return dec(res)

It is not quite a true metaclass (true metaclass has caused an error), but is interfaced like it:

so it can be used as

class ClassName(InterfaceName, metaclass=_Implements):
    ....

Why is it needed? I feel like it may allow some compatibility to the impls (i.e. Jython and probably GraalVM in future, currently it cannot implement Java interfaces or inherit Java classes) where implementing an interface is done via simple inheritance from it. So, to change an impl, one has only to change the set of classes and metaclasses. I have a lib https://github.com/KOLANICH/JAbs.py for this.

[Thrameos]

I plan to review this once I have support to class extensions. I am not sure that meta class is a good idea as it implies something else. I was planning to make it work with

class MyClass(java.lang.Object, implements=[interface1, interface2]):
   pass

This is much more clear as the Python class would have a presence as an Object and would get the interfaces. We could also do

class MyClass(JProxy, implements=[interface1, interface2]):
   pass

with some effort.

But I believe we can't do so until we drop support for Python 3.5. We have a meta class (_jpype._JClass) which can process the implements list and create the correct proxy implementation.

[KOLANICH]

Won't just

class MyClass(interface1, interface2):
    ...

be better?

[Thrameos]

I am not sure that I can do that. It is true that anything that inherits from a Java object gets the _JClass meta. I am not sure that I can properly enforce the extends first implements second.

But perhaps you are correct. I will play with this when I have the ability to extends a class. Perhaps I can make it that simple.

[Thrameos]

I am going to put this on the long term enhancement list as I can't really do much with it prior to the release.

[Thrameos]

Looking at the refactoring for extensions I believe I will be able to grant this one. Though there are some implications that I should run by you.

Currently when we implement an interface the object that we are creating is based in Python and thus when passed as proxy unpacks to a Python object as "self"

@JImplements(java.lang.Serializable)
class MyImpl(object):
     def __init__(self):  # self is a Python object with Python rules, open 
         pass 

If we implement a Java class (or extend one), then the instance is just stealing the methods and thus any instances that we create will be Java handles which point to Python methods. This means that the first argument is a Java object instance. There is no actual Python object associated with it. It is simply a Java class that will call Python methods. The class will be closed and any slots must be declared with a type and access restrictions.

class MyImpl(java.lang.Object):
     @JPublic
     def __init__(this):  # this is a Java object with Java rules
         pass 

Does this seem usable?

[KOLANICH]

Does this seem usable?

IDK for now.

First of all,

Any slots must be declared with a type and access restrictions.

is completely OK. And probably that can be done implicitly by defaulting to public.

But I don't quite understand what

The class will be closed

means. Does it mean that it would be impossible to create a class with additional fields/methods? If so, it feels like that such classes will almost always be useless (though it may be possible to workaround that with much perversions, such as a global dict with ids as keys).

[Thrameos]

I put down a pretty good explanation of this in the "doc/extensions.rst". By closed I mean just like Java classes they cannot be altered after they are defined. Thus the methods and fields that they have can only be defined when the class definition is parsed as opposed to Python classes where attributes and methods can be added even after the class if defined (or even on the fly as it is used.)

Here is a quick example

def pyfunc(s):
   pass

class MyExtension(JObject):
    # Define the slots
    JPublic(JString, a=None, b=None)  # We have a public field a and b of type java String
    JPublic(object, c=None)  # TypeError Java can't hold Python objects directly so this will fail. 
    JPublic(JString, d="hello") # ValueError Only primitives can be given a default value.  (I hope to allow String to also be initialized)
    
    # Define the methods
    @JPublic
    def __init__(this, s:String):
        this.a = s
        this.foo = s   # AttributeError "MyExtension has no 'foo' attribute."  'foo' was not defined as a slot so we can't add it later.
        this.call(s)   
        this.bar(s)  # AttibuteError "MyExtension has no bar attribute."  'bar' was not annotated so it doesn't get copied.
        pyfunc(this) # This succeeds as we can call Python code from within Java subject to visibility rules.
    
    @JPublic 
    def call(this, s:JString) -> JString:  # This will create a new method in the extension class.
       out = this.a
       this.a = s
       return out

     @JPublic
     def fail1(this, s):  # TypeError, all functions must be annotated so we can support overloading
          pass

     @JPublic
     def call(this, s:JString) -> JObject:  # TypeError, overloading with the same arguments is prohibited.
          pass

    @JPublic
    def call(this, s1:JString, s2:JString) -> None:
         pass   # this works because we are making a Java class rather than a Python one so it follows Java rules.

    def bar(self, s):  # This method is not properly annotated and thus won't get copied into Java.
       self.b = s 

    @JPublic
    def getSomething(this, a:JString="value) -> None: # TypeError defaults can not be specified.
       pass

print(type(MyExtension)) # This will print "<java class 'dynamic.MyExtension$1'>" because it is a dynamically generated Java class.

me = MyExtension()  # This produced as Java handle not a Python object

me.call("foo")  # This will only match just like a normal Java function.
me.call("yes", "no")  # This will only match just like a normal Java function.

me.bar("foo")  # AttributeError "MyExtension has no 'bar' attribute"
me.bar = a      # AttributeError "MyExtension has no field 'bar' "
MyExtension.bar = a # AttributeError MyExtension has no static field 'bar'

Basically this allows you to make a new Java class from within Python with the added capability that it automatically proxies so you can call Python code directly. It just compiles it on the fly into a new Java class.

I can try to give it a dictionary, add storage for Python fields, or generate more native like behavior later, but those will have significant down sides in terms of potential memory leaks and other dangerous side effects.

Do these restrictions appear reasonable?

[KOLANICH]

I put down a pretty good explanation of this in the "doc/extensions.rst".

strangely, I haven't found this doc. Also, IDK where epypj repo resides.

closed I mean just like Java classes they cannot be altered after they are defined.

That is OK as long as a user can add a collection to store the data it doesn't know beforehand.

It just compiles it on the fly into a new Java class.

Does it mean that we will have inheritance too?

I can try to give it a dictionary, add storage for Python fields, or generate more native like behavior later, but those will have significant down sides in terms of potential memory leaks and other dangerous side effects.

The primary here is storage for python-native objects. It seems it can be implemented even without built-in support, if id is stable for the same object, and can be cleaned if __del__ works for java-classes too:

storage = {}

class MyExtension(JObject):
    @JPublic
    def __init__(self, s:String):
        storage[id(self)] = {"s": s}

    @JPublic
    def __del__(self):
        del storage[id(self)]

[Thrameos]

The extension and epypj are branches on the Thrameos/jpype fork. I do a lot of my advanced work there are then push it over when it is ready. Of course if I haven't pushed it, it likely does not work. The doc/extension.rst is attached to the PR for extensions.

That is OK as long as a user can add a collection to store the data it doesn't know beforehand.

Well I can potentially let someone do something like

class MyExtension(JObject):
     JPrivate(dict, storage = None)

     @JPublic
     def __init__(this):
           this.storage = {}
           this.storage['stuff'] = "put"

But that requires that I implement enough of epypj to make is so that the Java type PyDict is recognizable. You can also just use JPrivate(java.lang.HashMap, storage=None) but then you will only be able to store Java objects directly.

The danger comes if someone attempts the following. this.storage['stuff'] = this. Python will place the Java handle holds a hard reference to the Java object. As Java can't see into Python space and Python cannot traverse Java space we have an irresolvable memory loop.

Does it mean that we will have inheritance too?

Yes, you can inherit from one extension to another in Python. You can also use reflecting in Java to call any of the newly defined methods. It automatically names classes using the $# notation so that there can never be conflicts no matter how many times you define the same class name. Unfortunately you still can't unload them.

Even if I don't define it explicitly you can always add a dictionary or store arbitrary data using the existing Proxy. The interfaces 'java.io.Serializable' (or any interface with no methods) can be used to export Python object into Java. JProxy with the dict keyword will store a dictionary in Java.

I plan to hook that up to the cast operation for "(java)@obj" which would look at the type. If it is already Java then nothing happens it just returns the object. But it it is a Python object you would get a Java PyObject (inherited from java.lang.Object) instance which holds a reference to the Python object. You can then unwrap it to get back the Python instance. (it unpacks automatically when accessing as a field or argument in a proxy). This is useful to do things like putting Python objects onto Java collection like ArrayList. It is currently supported using some magic with JProxy but I need to formalize it into a usable feature.

Again there are dangers for memory leaks until we add support to allow Python traverse to pass though Java classes. This takes a lot of magic (like serious weeks of planning) as we would need an agent to load all Java classes (including bootloaded classes) to have a new internal method so that the GC play nicely together. It is doable (potentially assume no technical road blocks).

Of course porting PyPy using libffi and JAS to implement a full CPython API and add reference counting support into Java using an agent so that Python can run at native Java speeds and recycle memory for reasonable performance is also possible but I doubt anyone has the time to pull such a thing off. I suspect getting Graalvm up to par would be less effort.