Add possibility to extract all packages with extractPackages

[timrid]

I am currently working on debugging support for Android applications with VSCode and debugpy (beeware/briefcase#2351).

Briefcase is using the remote debugging feature of debugpy for VSCode debugging. For this to work correctly, the Python source code must be available on both, the host PC and the Android device. These are linked via path mappings. However, this requires that the .py files are also available on the Android device. Otherwise, it is not possible to set breakpoints on the host PC in VSCode, which are then transferred to the Android device.

Extracting the .py files is currently possible in chaquopy via extractPackages, but you have to explicitly specify the modules to extract. So it is not possible to debug the whole application. You always have to specify which files you want to debug. This would be quite annoying for users of Briefcase.

Therefore, there should be a way to extract all .py files of all modules. I suggest that extractPackages should interpret the value * that all modules should be unpacked.

[timrid]

I tested debugging with VSCode a little further and noticed that there is another problem when debugging with VSCode: Chaquopy apparently only extracts the .py files during the first import and not all of them at startup. This seems to be problematic if you have already set breakpoints in VSCode before starting the debug session. These breakpoints are transferred to the app when you first connect to debugpy. At this point, however, the modules have not yet been imported and therefore not yet extracted. I need to investigate this further and switch this PR to a draft.

[timrid]

I have extended the code so that when using the wildcard *, the packages are extracted directly at startup. This fixes the problem with the breakpoints in VSCode.

[mhsmith]

These breakpoints are transferred to the app when you first connect to debugpy. At this point, however, the modules have not yet been imported and therefore not yet extracted.

What happens in this case? Does it give an error, or does it just silently fail to set the breakpoint? And is it the same behavior for Python files which were imported (but not extracted) before setting the breakpoint?

[mhsmith]

It would be unfortunate if debugpy requires this, but if there's no other way to get it to work, then I guess this is the best solution.

Is it still necessary to extract the files even when pyc.src and pyc.pip are set to false?

[mhsmith]

Please also add a change note in product/runtime/docs/sphinx/changes.

[timrid]

These breakpoints are transferred to the app when you first connect to debugpy. At this point, however, the modules have not yet been imported and therefore not yet extracted.

What happens in this case? Does it give an error, or does it just silently fail to set the breakpoint?

It fails to set the breakpoint. You can see it when the debugger starts and you want to set a breakpoint, but the breakpoint is grey and not turning red. An error message is not shown.

And is it the same behavior for Python files which were imported (but not extracted) before setting the breakpoint?

Yes, the same behavior.

It would be unfortunate if debugpy requires this, but if there's no other way to get it to work, then I guess this is the best solution.

Is it still necessary to extract the files even when pyc.src and pyc.pip are set to false?

I investigated further if the extraction is required for debugging.

Basically, VSCode also supports debugging modules that are stored in .zip files and that were imported via zipimport. I tested this locally. Renaming a .zip to .imy also works without any problems.

In my test I created the following .imy manually:

my_pkg.imy
  -> helloworld
    -> __init__.py

With an __init__.py of:

def print_hello_world():
    print("Hello World!")

And tested it via

import sys
sys.path.insert(0, "my_pkg.imy")

import helloworld

helloworld.print_hello_world()  # you can step into this function with VSCode debugger.

print(helloworld.__file__)  
# Output: my_pkg.imy/helloworld/__init__.py

I noticed that the __file__ path of the imported module helloworld includes the name of the zip file my_pkg.imy.

As I understand chaquopy, the source code of the application I am running with briefcase is located at:

/data/user/0/com.example.helloworld/files/chaquopy/app.imy

So I would expect the __file__ path in an Android app would be:

/data/user/0/com.example.helloworld/files/chaquopy/app.imy/helloworld/__init__.py

But when running it (with pyc.src = false and without extraction), I get:

import helloworld

print(helloworld.__file__)  
# Output: /data/data/com.example.helloworld/files/chaquopy/AssetFinder/app/helloworld/__init__.py

So the path does not contain app.imy and also the file it points to does not exist. So how should debugpy get the correct location to the file?

If Chaquopy would be modified so that the __file__ paths correctly refer to the app.imy, I could imagine that debugging would also be possible. However, I don't yet understand the import logic in Chaquopy well enough to understand why it was done this way. This seems like a tricky part. Maybe you can explain this?

[mhsmith]

Thanks for the explanation. Yes, this is a complicated area, but I've found it's necessary in order to get good performance.

The most portable approach, both for data files and for debuggers, is to not assume that anything exists as an individual file, and use APIs like importlib.resources, pkgutil.get_data or pkg_resources.ResourceManager. But since 99% of Python installations do use separate files, many packages instead write code like open(f"{dirname(__file__)}/data.txt"). Obviously this wouldn't work if some component of __file__ was a ZIP.

So our current approach is:

• The .imy file and the .apk file are both actually ZIPs. The .apk exists as a real file on the target device, but the .imy doesn't; it's accessed from within the .apk using the AssetFile class. The .imy extension was chosen to prevent a second layer of compression, as explained in Common.java.
• Any data files (i.e. files whose extension is NOT .py or .pyc) are extracted when their containing package is imported.
• For performance, .py and .pyc files are not extracted unless listed in extractPackages, but their __file__ attribute is set to the location they would have if they existed, to allow code to find its data files using a relative path.

Even though debugpy understands zipimport, it looks like it's not quite flexible enough to handle what our importer does, so let's continue with this wildcard approach.

[mhsmith]

It's not possible to test more than one extractPackages configuration with the unit tests, because this is fixed when the app is built, but I've tested this manually and it works fine.

All that remains is to add an integration test to the ExtractPackages class of test_gradle_plugin.py. This file has changed quite a bit in the last week, so I suggest you merge from the master branch before you edit it. You'll also need to update the checkZip function to recognize the wildcard syntax, similarly to what you've already done in PythonTasks.kt.

Instructions for running the integration tests are in product/gradle-plugin/README.md, and you might also find it useful to look at how GitHub Actions runs them in .github/workflows/ci.yml.

Please also add a change note in product/runtime/docs/sphinx/changes.

[timrid]

I added the integration tests and the changelog. So this PR is ready for a new review.

[mhsmith]

For future reference:

• Please don't force-push a PR that's already been reviewed, as it makes it much more difficult to see the changes since the previous review. Just do a normal merge commit.
• "added" isn't one of the change note categories listed in pyproject.toml; it should have been "feature".

Thanks for the contribution!