Properly support permissions and symlinks in Zip files (500USD Bounty)

[lihaoyi]

---

From the maintainer Li Haoyi: I'm putting a 500USD bounty on this issue, payable by bank transfer on a merged PR fixing this.

See https://github.com/orgs/com-lihaoyi/discussions/6 for other bounties and the terms and conditions that bounties operate under

---

@kiendang hit some issues using os.zip in Mill (com-lihaoyi/mill#4503), some of which are permissions related. It seems that since 2019 the JDK's zip file APIs should be able to support permissions (https://bugs.java.com/bugdatabase/view_bug?bug_id=8213031), so we should make sure OS-Lib takes advantage to preserve permissions when zipping and unzipping files and folders

[kiendang]

Turned out the problem was all permission related. It worked after executable perm was given to files inside bin and libexec.

Anw can see 2 problems with os.unzip: It does not perserve file permissions and symlinks. After being unzipped from cosmocc.zip (https://cosmo.zip/pub/cosmocc/) with either os.unzip or os.unzip.stream, the files are stripped of permissions (files in bin no longer have executable permission) and all the original symlinks are replaced with copies of the destination files.

[sideeffffect]

https://unix.stackexchange.com/a/509337

Info-Zip 3.0 SUPPORTS preserving files/dirs UNIX permissions and UID/GID ownership data. zip stores it by default but you need to use unzip in an special way to restore them:

• unzip must be used with the -X flag.
• unzip must run as root to set the files/dirs UID/GID. If you run it as a normal user then the UID will be always the one of the current user and the GID will be restored ONLY IF the current user belongs to that group.

Note: you can also use unzip with the -K flag to also restore SUID/SGID/Sticky bits.

[kiendang]

Extra fields have been used to store information about POSIX permissions and symlink though. jdk.zipfs does support POSIX permissions from JDK 14.

os-lib uses java.util.zip.ZipEntry which does not expose this information though. It's stored in the private ZipEntry.extraAttributes field. There's talk on making it accessible but I don't think it's gonna be resolved any time soon judging on the lack of activity.

Apache Commons and Apache Ant's own implementation of ZipEntry does support permissions (but obviously we're not adding external deps here).

So for a quick fix I think can use jdk.zipfs to support permissions for os.zip and os.unzip (only for Path, not .stream).

[kiendang]

@sideeffffect java.util.zip is based on the Info-ZIP format anyway. Just that the field that contains the information is unexposed...

[lihaoyi]

For extraAttributes, can we get at them via reflection? Would that allow us to support permissions on JDK <14?

Otherwise only supporting permissions for os.zip and os.unzip for os.Paths is an alternative. Would that require JDK >= 14? That might be fine if raise the minimum JDK requirement to 17, as has been proposed for Scala itself, and I think a PR that does that would be enough to satisfy this ticket

Adding a third party dependency is an option, but it would be great if we could find a solution just using the native JDK

[kiendang]

Otherwise only supporting permissions for os.zip and os.unzip for os.Paths is an alternative. Would that require JDK >= 14?

Yup JDK >= 14 is required. The thing is this is not without hiccups either. Using jdk.zipfs, a default set of permissions is defined. Zip entries without permission info would be given this set of default permissions, while the more appropriate thing to do would be to follow umask? The API doesn't provide a way to know which entry actually have or doesn't have permission info either. We would have to first figure out umask (by creating a blank file?) then set it as the default permissions?

Third party dependency is also not perfect. Apache Commons Compress doesn't support symlink (edit: it does).

I think extraAttributes can be obtained with reflection, and if it's straightforward from there this would be the best solution. Available since JDK 11. Just that the attribute has been renamed a couple times so need to dance around that.

[kiendang]

Ah looks like using reflection to access private field for java.lang. classes is not permitted anymore since JDK 17. Would throw InaccessibleObjectException.

[kiendang]

Looks like using Apache Commons Compress is the cleanest solution for now? Need to experiment though since their ZipArchiveEntry only exposes certain things to ZipFile, not stream.

[lihaoyi]

I think using a third party library is fine. We can shade it so it is totally encapsulated and collide with other versions on the classpath

[lefou]

I don't think we should shade just because we can. Is there even a known risk, that anything can collide with other version of commons-compress? We could build against the oldest version we can/want support, so that binary compatibility isn't an issue, and add the newest known version to our runtime deps.

[lefou]

commons-compress is most likely not available for the scala-native version, I guess.

[kiendang]

I've played a bit with Apache Commons Compress and found that it's not possible to make unzip.stream preserve symlinks and permissions since zip files store this information in the central directory at the end of the file, not together with each zip entry. zip, unzip and zip.stream can do this though. Would still need Apache Commons Compress for zip.stream to support this.

On the Apache Commons Compress dependency thing may I suggest another approach? It's not a good idea to include it as a dependency. It is quite heavy since it also includes support for several other file formats, plus the Scala native issue as @lefou mentioned (edit: this was wrong). I was thinking of just including the few source files that we need as unmanaged sources. However, Apache Commons Compress zip even depends on Commons IO.

jdeps --multi-release 61 -cp commons-compress-1.27.1.jar org/apache/commons/compress/archivers/zip/ZipArchiveOutputStream.class

ZipArchiveOutputStream.class -> commons-compress-1.27.1.jar
ZipArchiveOutputStream.class -> java.base
ZipArchiveOutputStream.class -> not found
   org.apache.commons.compress.archivers.zip          -> java.io                                            java.base
   org.apache.commons.compress.archivers.zip          -> java.lang                                          java.base
   org.apache.commons.compress.archivers.zip          -> java.nio                                           java.base
   org.apache.commons.compress.archivers.zip          -> java.nio.channels                                  java.base
   org.apache.commons.compress.archivers.zip          -> java.nio.charset                                   java.base
   org.apache.commons.compress.archivers.zip          -> java.nio.file                                      java.base
   org.apache.commons.compress.archivers.zip          -> java.util                                          java.base
   org.apache.commons.compress.archivers.zip          -> java.util.zip                                      java.base
   org.apache.commons.compress.archivers.zip          -> org.apache.commons.compress.archivers              commons-compress-1.27.1.jar
   org.apache.commons.compress.archivers.zip          -> org.apache.commons.compress.utils                  commons-compress-1.27.1.jar
   org.apache.commons.compress.archivers.zip          -> org.apache.commons.io                              not found

Apache Commons Compress zip was actually lifted from Apache Ant org.apache.tools.zip. org.apache.tools.zip provides what we need and is a self-sufficient package, only depending on java standard libs

jdeps -cp ant-1.10.15.jar org/apache/tools/zip/ZipOutputStream.class

ZipOutputStream.class -> java.base
   org.apache.tools.zip                               -> java.io                                            java.base
   org.apache.tools.zip                               -> java.lang                                          java.base
   org.apache.tools.zip                               -> java.nio                                           java.base
   org.apache.tools.zip                               -> java.nio.file                                      java.base
   org.apache.tools.zip                               -> java.util                                          java.base
   org.apache.tools.zip                               -> java.util.zip                                      java.base

We could just include src/main/org/apache/tools/zip as unmanaged sources (and attribute properly with respect to ant's license of course). We're not exposing any of this to the APIs anyway.