[JEP-202] Extending VirtualFile #3302

jglick · 2018-02-19T23:04:01Z

See JENKINS-49635 and also JENKINS-26810, and JEP-202.

basic API sketch
local test of toExternalURL
VirtualFile.mode() (needed by [JENKINS-22637] ArtifactManager support on new core copyartifact-plugin#100)
VirtualFile.readLink() (needed by [JENKINS-22637] ArtifactManager support on new core copyartifact-plugin#100)
VirtualFile.list(String, String, boolean) (needed by [JENKINS-22637] ArtifactManager support on new core copyartifact-plugin#100) (could also be used to simplify Add support for download of zipped artifacts compress-artifacts-plugin#1)
test of toExternalURL in [JENKINS-22637] ArtifactManager support on new core copyartifact-plugin#100

Reference implementation is in the artifact-manager-s3 plugin.

@reviewbybees @jenkinsci/code-reviewers

…ternal artifact storage.

…ly defined semantics.

…ring, boolean) which is also more convenient for some implementations.

…ows, contrary to Javadoc.

…or, which is normally handled deep within DirectoryScanner.

jglick · 2018-02-21T23:30:12Z

core/src/main/java/hudson/Util.java

     */
    @CheckForNull
-    public static String resolveSymlink(@Nonnull File link) throws InterruptedException, IOException {
+    public static String resolveSymlink(@Nonnull File link) throws IOException {


Binary compatible simplification—the body never actually threw that exception.

Not source compatible IIRC (javac fails when there is a catch block for impossible exception), may impact PCT. There are usages of it externally, e.g. in the Support Core plugin: https://github.com/search?p=1&q=org%3Ajenkinsci+resolveSymlink&type=Code

I would propose to detach it to a separate PR if you feel it's important enough

@oleg-nenashev Jesse already disputed this in #3340 (comment)

Right, PCT should be unaffected, though it could impede

buildPlugin(jenkinsVersions: [null, '2.222']) // or whatever

Why the Java language treats an exception which cannot be thrown as a fatal error rather than a warning (ditto unreachable statements etc.), I do not know.

The simple workaround for the buildPlugin case would be to just catch (Exception I guess.

would be better to fix it tho

jglick · 2018-02-21T23:30:57Z

core/src/main/java/hudson/model/DirectoryBrowserSupport.java

        try (ZipOutputStream zos = new ZipOutputStream(outputStream)) {
            zos.setEncoding(System.getProperty("file.encoding")); // TODO JENKINS-20663 make this overridable via query parameter
-            for (String n : dir.list(glob.length() == 0 ? "**" : glob)) {
+            for (String n : dir.list(glob.isEmpty() ? "**" : glob, null, /* TODO what is the user expectation? */true)) {


Retaining default excludes functionality here to maintain semantics but I am unsure whether users would want this or not.

jglick · 2018-02-21T23:32:42Z

core/src/main/java/jenkins/util/VirtualFile.java

+        }
+        return r.stream().filter(p -> {
+            TokenizedPath path = new TokenizedPath(p.replace('/', File.separatorChar));
+            return includePatterns.stream().anyMatch(patt -> patt.matchPath(path, true)) && !excludePatterns.stream().anyMatch(patt -> patt.matchPath(path, true));


VirtualFileTest.Ram checks all this implementation.

jglick · 2018-02-21T23:34:07Z

core/src/main/java/jenkins/util/VirtualFile.java

+     * <p>Generally this will be harder to implement than {@link #asRemotable},
+     * which would have the opportunity to perform arbitrary preparation for {@link #open}
+     * such as negotiating session authentication.
+     * @return an externally usable URL like {@code https://gist.githubusercontent.com/ACCT/GISTID/raw/COMMITHASH/FILE}, or null if there is no such support


Just the most familiar download URL example I could come up with—certainly not a choice for artifact storage!

Whoever implements this as a weekend project will get 🍻 from me.

Adding some 🍺 to the trophy

olivergondza · 2018-03-03T11:42:04Z

core/src/main/java/jenkins/util/VirtualFile.java

+ * <p>Since some implementations may in fact use external file storage,
+ * callers may request optional APIs to access those services more efficiently.
+ * Otherwise, for example, a plugin copying a file
+ * previously saved by {@link ArtifactManager} to an external storage servuce


s/servuce/service/

oleg-nenashev

Looks good so far though I would still prefer InterruptedException. Accmod is still waiting for the release by @kohsuke

oleg-nenashev · 2018-04-02T14:39:26Z

core/src/main/java/hudson/Util.java

     */
    @CheckForNull
-    public static String resolveSymlink(@Nonnull File link) throws InterruptedException, IOException {
+    public static String resolveSymlink(@Nonnull File link) throws IOException {


would be better to fix it tho

oleg-nenashev · 2018-04-02T14:40:18Z

core/src/main/java/hudson/model/DirectoryBrowserSupport.java

     */
    private static List<List<Path>> patternScan(VirtualFile baseDir, String pattern, String baseRef) throws IOException {
-            String[] files = baseDir.list(pattern);
+            Collection<String> files = baseDir.list(pattern, null, /* TODO what is the user expectation? */true);


Note that there was a JEP-200 regression in the similar code recently

Yes, in DirScanner.Glob IIRC, which is called by some list implementations.

oleg-nenashev · 2018-04-02T15:28:17Z

core/src/main/java/hudson/model/Run.java

+    /** {@link Run.ArtifactList} without the implicit link to {@link Run} */
+    private static final class SerializableArtifactList extends ArrayList<SerializableArtifact> {
+        private static final long serialVersionUID = 1L;
+        private LinkedHashMap<SerializableArtifact, String> tree = new LinkedHashMap<>();


Even for a private class not looks a bit dangerous since you do not override modification methods to put data to both containers (array and tree)

This code was already a mess and I am trying to touch it as little as possible. The problem here was that Artifact and ArtifactList were, surprisingly, nonstatic—they held a Run reference—which is unsupportable when they must be constructed inside a callable which might run remotely. (If using the FilePath-based VirtualFile, for example. In fact this is not currently used by any ArtifactManager, only workspace browsing, but the API does not preclude the possibility, so…)

oleg-nenashev · 2018-04-02T15:31:43Z

core/src/main/java/jenkins/util/VirtualFile.java

+ * only on the node on which the object was created.
+ *
+ * <p>Since some implementations may in fact use external file storage,
+ * callers may request optional APIs to access those services more efficiently.


I still feel that we grow a technical debt in API, but Ican live with that.

oleg-nenashev · 2018-04-02T15:44:20Z

core/src/main/java/hudson/util/DirScanner.java

-                throw (IOException) new InterruptedIOException().initCause(e);
+                target = Util.resolveSymlink(f);
+            } catch (IOException x) { // JENKINS-13202
+                target = null;


carlossg · 2018-04-13T15:35:12Z

this is stable now for final review

carlossg · 2018-04-13T16:34:13Z

@daniel-beck can you take a look at the new changes to see if your positive review still stands?

carlossg

This is tested with the https://github.com/jenkinsci/artifact-manager-s3-plugin and works fine

oleg-nenashev

🐝 The diff from the previous review looks good to me. Since all APIs are annotated with Beta, we can reconsider APIs in the worst case

oleg-nenashev · 2018-04-16T17:19:51Z

@reviewbybees done

carlossg · 2018-04-20T16:06:06Z

Given the JEP is in place, no objections have been raised and all apis are marked as @beta I think this is ready to be merged for the benefit of downstream consumers, both the initial s3 implementation and any other interested implementers, and provide feedback

daniel-beck · 2018-04-20T20:24:48Z

core/src/main/java/jenkins/util/VirtualFile.java

+    /**
+     * @deprecated use {@link #list(String, String, boolean)} instead
+     */
+    @Deprecated


This deprecation points to a Beta API. This seems unhelpful.

Well, the only callers will be updated anyway.

daniel-beck · 2018-04-20T20:30:25Z

FTR, agree with @carlossg regarding the eligibility for this to be merged. Beta annotations allow this to evolve if needed without breaking existing API commitments.

jglick · 2018-04-20T22:08:27Z

FTR jenkinsci/jep#96

[JENKINS-49635] Defining new VirtualFile methods to better support ex…

3f01a77

…ternal artifact storage.

jglick added the work-in-progress The PR is under active development, not ready to the final review label Feb 19, 2018

jglick requested review from carlossg and oleg-nenashev February 19, 2018 23:04

daniel-beck self-requested a review February 19, 2018 23:51

jglick mentioned this pull request Feb 20, 2018

[JENKINS-22637] ArtifactManager support jenkinsci/copyartifact-plugin#99

Merged

7 tasks

[JENKINS-26810] Added VirtualFile.mode.

8e603a5

jglick mentioned this pull request Feb 21, 2018

[JENKINS-22637] ArtifactManager support on new core jenkinsci/copyartifact-plugin#100

Merged

jglick changed the title ~~[JENKINS-49635] Defining new VirtualFile methods to better support external artifact storage~~ [JENKINS-49635] [JENKINS-26810] Extending VirtualFile API Feb 21, 2018

jglick changed the title ~~[JENKINS-49635] [JENKINS-26810] Extending VirtualFile API~~ [JENKINS-49635] [JENKINS-26810] Extending VirtualFile Feb 21, 2018

jglick added 9 commits February 21, 2018 14:17

New VirtualFile.list overload with more capabilities and more precise…

e773bdb

…ly defined semantics.

More compatible default implementation of VirtualFile.list(String, St…

4911c5c

…ring, boolean) which is also more convenient for some implementations.

VirtualFile.readLink

e3e6d57

Forgotten since tags.

2e070ca

Compilation error.

9735a34

Mistake caught by DirectoryBrowserSupportTest.zipDownload.

a9430d3

list(String, String, boolean) was returning \-separated paths on Wind…

e0fab1a

…ows, contrary to Javadoc.

SelectorUtils.tokenizePath apparently expects the native path separat…

9ccffba

…or, which is normally handled deep within DirectoryScanner.

Minor comments and simplifications.

7131067

jglick commented Feb 21, 2018

View reviewed changes

jglick added on-hold This pull request depends on another event/release, and it cannot be merged right now and removed work-in-progress The PR is under active development, not ready to the final review labels Feb 21, 2018

Merge branch 'master' into VirtualFile-JENKINS-49635

9b9f81d

olivergondza reviewed Mar 3, 2018

View reviewed changes

jglick added 3 commits March 5, 2018 10:02

Merge branch 'master' into VirtualFile-JENKINS-49635

23f12fd

Typo noticed by @olivergondza.

63885e6

Merge branch 'master' into VirtualFile-JENKINS-49635

64ee6fd

Using new beta annotation.

2090468

jglick mentioned this pull request Mar 27, 2018

Allow easy consumption of beta restriction jenkinsci/plugin-pom#98

Merged

oleg-nenashev reviewed Apr 2, 2018

View reviewed changes

jglick added 2 commits April 2, 2018 14:09

Merge branch 'master' into VirtualFile-JENKINS-49635

a345051

Removing VirtualFile.asRemotable in favor of toExternalURL.

547fd6f

jglick mentioned this pull request Apr 2, 2018

asRemotable has been deleted jenkinsci/artifact-manager-s3-plugin#2

Merged

access-modifier.version=1.14

a54d49b

Merge branch 'master' into VirtualFile-JENKINS-49635

d25c0b2

Vlatombe self-requested a review April 16, 2018 08:42

oleg-nenashev self-requested a review April 16, 2018 17:10

carlossg approved these changes Apr 16, 2018

View reviewed changes

jglick requested review from carlossg and oleg-nenashev and removed request for oleg-nenashev April 16, 2018 17:11

oleg-nenashev approved these changes Apr 16, 2018

View reviewed changes

jglick changed the title ~~[JENKINS-49635] [JENKINS-26810] Extending VirtualFile~~ [JEP-202] Extending VirtualFile Apr 16, 2018

Merge branch 'master' into VirtualFile-JENKINS-49635

8f5d1d3

carlossg approved these changes Apr 18, 2018

View reviewed changes

jglick added ready-for-merge The PR is ready to go, and it will be merged soon if there is no negative feedback and removed on-hold This pull request depends on another event/release, and it cannot be merged right now labels Apr 20, 2018

daniel-beck approved these changes Apr 20, 2018

View reviewed changes

jglick merged commit 51cac7d into jenkinsci:master Apr 20, 2018

jglick deleted the VirtualFile-JENKINS-49635 branch April 20, 2018 22:08

bitwiseman mentioned this pull request Apr 21, 2018

[on-hold] Accepting JEP-202 jenkinsci/jep#96

Merged

Uh oh!

[JEP-202] Extending VirtualFile #3302

[JEP-202] Extending VirtualFile #3302

Uh oh!

Conversation

jglick commented Feb 19, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

oleg-nenashev left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlossg commented Apr 13, 2018

Uh oh!

carlossg commented Apr 13, 2018

Uh oh!

carlossg left a comment

Choose a reason for hiding this comment

Uh oh!

oleg-nenashev left a comment

Choose a reason for hiding this comment

Uh oh!

oleg-nenashev commented Apr 16, 2018

Uh oh!

carlossg commented Apr 20, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daniel-beck commented Apr 20, 2018

Uh oh!

jglick commented Apr 20, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

jglick commented Feb 19, 2018 •

edited

Loading