[JENKINS-22637] ArtifactManager support #99

jglick · 2018-02-20T23:54:34Z

JENKINS-22637: supporting custom ArtifactManagers via VirtualFile. Unlike #71 the goal here is to make just this change with as little impact on existing installations as possible.

basic refactoring
integration test with the compress-artifacts plugin (14 votes)
restore support for explicit excludes (1 vote)
restore support for symlinks (2 votes): [JENKINS-22637] ArtifactManager support on new core #100
restore support for file attribute handling (11 votes): [JENKINS-22637] ArtifactManager support on new core #100
restore support for default excludes (1 vote): [JENKINS-22637] ArtifactManager support on new core #100

To handle symlinks and file attributes may require new core APIs—TBD. Really if you care about platform-specific stuff like this you should create a tarball and archive that instead, since you cannot guarantee that the Jenkins master will preserve these nuances. Similarly, suppressing of default excludes is desirable but might just be treated as a core bug fix, and again it is going to be safer and much more efficient to create a ZIP or tarball of all your little files.

I also plan to

support direct download as per [JEP-202] Extending VirtualFile jenkins#3302: [JENKINS-22637] ArtifactManager support on new core #100

but this will certainly require a new core baseline and might better be filed as an add-on PR.

Note that I am dropping the Copier extension point here. In practice I doubt this would be incompatible for anyone:

There are no known external implementations.
This never really behaved like a regular extension point to begin with: the user cannot select an implementation! In fact FilePathCopyMethod was dead code—even as of 3cf80e5, it was always overridden by FingerprintingCopyMethod. And that one was rather misnamed as of JENKINS-12134 - Make fingerprinting artifacts optional. #11—it may or may not do fingerprinting!
JENKINS-7753 claimed as justification

I would like to implement a plug-in which provides CopyArtifact with rsync capabilities when copying workspaces between jobs. This will decrease load on the master and most likely increase IO-throughput.

which is sensible enough but better handled via ArtifactManager, especially as of JENKINS-49635.

I am also making an API change to BuildSelector. In this case there are two known external implementations but neither override the getSourceDirectory method. In fact, other than WorkspaceSelector, it is hard to imagine any other useful override. I am making a cursory effort to retain compatibility with any external overrides just in case.

Subsumes #98.

@reviewbybees

… least.

ikedam

Looks good to me.

you should create a tarball and archive that instead

+1
Users are recommended to use zip, unzip steps or call tar command via sh step.
Unfortunately, zip step doesn't support exclude filters...

ikedam · 2018-02-22T23:38:45Z

src/main/java/hudson/plugins/copyartifact/Copier.java

+ * @deprecated No longer used.
 */
+@Deprecated
 public abstract class Copier implements ExtensionPoint {


We can remove them.
It's useless to preserving binary compatibilities as they're completely no longer used,
and exceptions should occur if someone use them.

Well, no “exception” would result from using them somehow. I do not have a strong opinion either way.

ikedam · 2018-02-23T00:15:21Z

src/main/java/hudson/plugins/copyartifact/CopyArtifact.java

                throw new AbortException(message);
            }
        }
-        FilePath targetDir = workspace, baseTargetDir = targetDir;


Note for me: baseTargerDir looks not used at all. Completely useless.

Right, AFAICT it was never used anyway.

jglick · 2018-02-24T13:18:30Z

@ikedam so I am leaving it up to your judgment what to do here. I noted the number of JIRA votes for various affected issues, to the extent that is meaningful data. This PR in isolation implements a feature with 14 votes (and no good workaround), and regresses three bug reports with a combined total of 14 votes (and the workaround of archiving a tarball rather than individual files). #100 implements the feature (and more) without regressing any of the prior bugs, but will require a new core baseline. 🤷‍♂️

One option not previously discussed would be to merge this but release only as a -beta version, allowing users interested in JENKINS-22637 to get the implementation from the experimental update center; and then release #100 if and when the corresponding core PR is released (the update center is smart enough not to offer that update to anyone using older LTS lines).

The hope is that once JEP-301 is deployed, Essentials users would get new stuff that just works, right away. Obviously a lot needs to happen before that is a reality.

oleg-nenashev

Looks good in general. Some code could be moved to the core or API plugin if there are use-case in other plugins. 🐝

oleg-nenashev · 2018-03-14T01:01:38Z

src/main/java/hudson/plugins/copyartifact/CopyArtifact.java

-                for (FilePath file : list)
-                    copier.copyOne(file, new FilePath(targetDir, file.getName()), isFingerprintArtifacts());
-                cnt = list.length;
+            targetDir.mkdirs();  // Create target if needed


would be nice to switch it to Java7 Files API while you are around

oleg-nenashev · 2018-03-14T01:04:11Z

src/main/java/hudson/plugins/copyartifact/CopyArtifact.java

+            try {
+                md5 = MessageDigest.getInstance("MD5");
+            } catch (NoSuchAlgorithmException x) {
+                throw new AssertionError(x);


does not really matter, but maybe just IOException? There were some discussions about disabling MD5 at all to harden the security. I'd guess it is not realistic, but I would rather prefer to avoid errors in the production code

To do that we would first need to reimplement fingerprints in Jenkins core, which would surely force changes here as well anyway.

oleg-nenashev · 2018-03-14T01:05:33Z

src/main/java/hudson/plugins/copyartifact/CopyArtifact.java

+        }
+    }
+
+    private static List<String> scan(VirtualFile root, String expandedFilter, @CheckForNull String expandedExcludes) throws IOException {


Maybe it should be moved to core utils eventually. Looks to be generic enough

This code is deleted in #100.

oleg-nenashev · 2018-03-14T01:09:25Z

src/main/java/hudson/plugins/copyartifact/CopyArtifact.java

+            targetDir.mkdirs();  // Create target if needed
+            List<String> list = scan(srcDir, expandedFilter, expandedExcludes);
+            for (String file : list) {
+                copyOne(src, dst, fingerprints, srcDir.child(file), new FilePath(targetDir, isFlatten() ? file.replaceFirst(".+/", "") : file), md5);


passing isFingerprintArtifacts() looked better, but OK

But now we need to pass in the map.

oleg-nenashev · 2018-03-14T01:12:57Z

src/main/java/hudson/plugins/copyartifact/CopyArtifact.java

-            copier.end();
+            if (fingerprints != null) {
+                for (Run<?, ?> r : new Run<?, ?>[] {src, dst}) {
+                    if (fingerprints.size() > 0) {


It is not modified inside the cycle IIUC, maybe makes sense to just move to the outer check

Does not really matter—the loop has only two iterations.

oleg-nenashev · 2018-03-14T01:13:52Z

src/main/java/hudson/plugins/copyartifact/CopyArtifact.java

+        return pattern;
+    }
+
+    private static void copyOne(Run<?,?> src, Run<?,?> dst, Map<String, String> fingerprints, VirtualFile s, FilePath d, MessageDigest md5) throws IOException, InterruptedException {


same, does not seem to be plugin-specific

It is pretty specific to copyartifact I think.

ikedam · 2018-03-14T06:19:28Z

src/main/java/hudson/plugins/copyartifact/CopyArtifact.java

+            targetDir.mkdirs();  // Create target if needed
+            List<String> list = scan(srcDir, expandedFilter, expandedExcludes);
+            for (String file : list) {
+                copyOne(src, dst, fingerprints, srcDir.child(file), new FilePath(targetDir, isFlatten() ? file.replaceFirst(".+/", "") : file), md5);


Note for me: snipping ".+/" is equivalent to the current implementation calling FilePath#getName() in FingerprintingCopyMethod.
(I'm not sure this is a good behavior as reported in JENKINS-16987)

No, this replaceFirst call just implements the flatten mode. It has nothing to do with fingerprinting. The behavior you mention is retained in copyOne as

fingerprints.put(s.getName(), digest); // ^^^^^^^^^^

Oops, I misplaced the comment!
https://github.com/jenkinsci/copyartifact-plugin/pull/100/files#r174750602

ikedam · 2018-03-14T06:36:24Z

@jglick Sorry for my late response.

Releasing to the experimental update center at first
Releasing to the actual update center when [JEP-202] Extending VirtualFile jenkins#3302 is released

This option looks good to me.

Let me have a little more time to work for https://issues.jenkins-ci.org/browse/JENKINS-47905 and https://issues.jenkins-ci.org/browse/JENKINS-47074.
So releases:

copyartifact-1.40: JENKINS-47905 , JENKINS-47074
copyartifact-1.41-beta: JENKINS-22637
copyartifact-1.41: JENKINS-22637

ikedam · 2018-04-01T01:33:04Z

As JENKINS-47074 will be resolved as Won't fix and JENKINS-47905 provides no new features, versions are:

copyartifact-1.39.1: JENKINS-47905
copyartifact-1.40-beta: JENKINS-22637
copyartifact-1.40: JENKINS-22637

ikedam · 2018-04-29T06:14:02Z

Really sorry for long absence.
I resume this task.
Though the extension of VirtualFile is already released as Jenkins2.108, I believe it still make sense to release 1.40-beta for LTS users.

ikedam · 2018-04-29T06:20:19Z

Hmm... It seems that releasing 1.40 will mask 1.40-beta as described in https://jenkins.io/doc/developer/publishing/releasing-experimental-updates/.
I might no longer make sense to release 1.40-beta.

ikedam · 2018-04-29T07:30:15Z

Released 1.40-beta-1 with this change.
I couldn't update the wiki page as login failure (looks like a system error). I'll try it later.

jglick · 2018-05-01T13:48:29Z

releasing 1.40 will mask 1.40-beta

AFAIK not, since the UC for LTS users should only be offering 1.39.

ikedam · 2018-05-05T12:55:35Z

Updates center provides several versions of plugin lists for core versions by redirection.

But the experimental update center provides only one version, and will result 1.40-beta-1 inaccessible via the update center.

Anyway, that doesn't matter so much as users can download the beta version directly from http://updates.jenkins-ci.org/download/plugins/copyartifact/1.40-beta-1/copyartifact.hpi .

[JENKINS-22637] Refactored from FilePath to VirtualFile.

6333853

jglick added the work-in-progress label Feb 20, 2018

jglick requested review from ikedam and oleg-nenashev February 20, 2018 23:54

jglick added 2 commits February 20, 2018 20:56

Demonstrating effectiveness with CompressingArtifactManagerFactory at…

f384826

… least.

Tracked down the problem with explicit excludes.

b3e6d53

jglick mentioned this pull request Feb 21, 2018

[JENKINS-22637] ArtifactManager support on new core #100

Merged

ikedam approved these changes Feb 23, 2018

View reviewed changes

jglick removed the work-in-progress label Feb 24, 2018

Merge branch 'master' into VirtualFile-JENKINS-22637

731f1fd

oleg-nenashev approved these changes Mar 14, 2018

View reviewed changes

ikedam reviewed Mar 14, 2018

View reviewed changes

jglick changed the title ~~[JENKINS-22637] ArtifactManager support~~ [on-hold] [JENKINS-22637] ArtifactManager support Mar 14, 2018

Merge branch 'master' into VirtualFile-JENKINS-22637

0dcb411

ikedam changed the title ~~[on-hold] [JENKINS-22637] ArtifactManager support~~ [JENKINS-22637] ArtifactManager support Apr 29, 2018

ikedam merged commit 1f762e5 into jenkinsci:master Apr 29, 2018

jglick deleted the VirtualFile-JENKINS-22637 branch May 1, 2018 13:48

Uh oh!

[JENKINS-22637] ArtifactManager support #99

[JENKINS-22637] ArtifactManager support #99

Uh oh!

Conversation

jglick commented Feb 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ikedam left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jglick commented Feb 24, 2018

Uh oh!

oleg-nenashev left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ikedam commented Mar 14, 2018

Uh oh!

ikedam commented Apr 1, 2018

Uh oh!

ikedam commented Apr 29, 2018

Uh oh!

ikedam commented Apr 29, 2018

Uh oh!

ikedam commented Apr 29, 2018

Uh oh!

jglick commented May 1, 2018

Uh oh!

ikedam commented May 5, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jglick commented Feb 20, 2018 •

edited

Loading