Skip to content

Commit 9b6592f

Browse files
tsibleytsibley
committed
entrypoint-aws-batch: Remove existing /nextstrain/{augur,auspice,fauna}/ for overlays
Makes these overlays work the same on AWS Batch as they do in local containers (Docker or Singularity): the overlays fully replace the image's copy. This addresses a caveat noted the Nextstrain CLI commit adding AWS Batch overlay support¹ that was further discussed in review.² ZIP archive directory members always end in "/", so we test only for the presence of ../{augur,auspice,fauna}/ in the archive (or the presence of equivalent relative paths adjusted for a non-default workdir). This works consistently for workdir archives with overlays produced by Nextstrain CLI, but it isn't foolproof for archives produced other ways. For example, it is possible for a ../augur/x/y/z path to be present in the archive without ../augur/ also being present. In that case, nothing will be deleted. This could be seen as a bug or a feature. ¹ <nextstrain/cli@99168ac> ² <nextstrain/cli#419 (comment)>
1 parent 654459f commit 9b6592f

4 files changed

+202
-0
lines changed

‎entrypoint-aws-batch

+10
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,16 @@ fi
1010
case "$NEXTSTRAIN_AWS_BATCH_WORKDIR_URL" in
1111
s3://*.zip)
1212
aws s3 cp --no-progress "$NEXTSTRAIN_AWS_BATCH_WORKDIR_URL" "$PWD.zip"
13+
14+
for dir in /nextstrain/{augur,auspice,fauna}; do
15+
relative_dir="$(realpath "$dir" --relative-to="$PWD")"/
16+
17+
if zipinfo -1 "$PWD.zip" "$relative_dir" &>/dev/null; then
18+
echo "removing $dir because workdir ZIP contains $relative_dir overlay"
19+
rm -rf "$dir"
20+
fi
21+
done
22+
1323
unzip -: -o "$PWD.zip"
1424
;;
1525
s3://*)

‎tests/aws-batch-workdir-url.t

+192
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,192 @@
1+
#!/usr/bin/env cram
2+
3+
Setup.
4+
5+
$ [[ -n "$AWS_ACCESS_KEY_ID" && -n "$AWS_SECRET_ACCESS_KEY" ]] || exit 80
6+
7+
$ : "${IMAGE:=localhost:5000/nextstrain/base:latest}"
8+
$ (docker image inspect "$IMAGE" || docker image pull "$IMAGE") &>/dev/null
9+
10+
$ export NEXTSTRAIN_AWS_BATCH_VERBOSE=0
11+
12+
Workdir ZIP archive is downloaded and extracted.
13+
14+
$ export NEXTSTRAIN_AWS_BATCH_WORKDIR_URL="s3://nextstrain-tmp/$(python3 -c 'import uuid; print(uuid.uuid4())').zip"
15+
16+
$ aws s3 cp --quiet "$TESTDIR/data/workdir-without-overlays.zip" "$NEXTSTRAIN_AWS_BATCH_WORKDIR_URL"
17+
18+
$ docker run --rm --env=NEXTSTRAIN_AWS_BATCH_{WORKDIR_URL,VERBOSE} --env=AWS_{ACCESS_KEY_ID,SECRET_ACCESS_KEY,SESSION_TOKEN} "$IMAGE" \
19+
> /sbin/entrypoint-aws-batch bash -euo pipefail -xc 'ls -l'
20+
download: s3://nextstrain-tmp/*.zip to ../build.zip (glob)
21+
Archive: /nextstrain/build.zip
22+
extracting: reticulating
23+
extracting: splines
24+
+ ls -l
25+
total 0
26+
-rw-rw-r-- 1 nextstrain nextstrain 0 Mar 10 21:46 reticulating
27+
-rw-rw-r-- 1 nextstrain nextstrain 0 Mar 10 21:46 splines
28+
upload: ../build.zip to s3://nextstrain-tmp/*.zip (glob)
29+
30+
/nextstrain/{augur,auspice} are removed when the workdir ZIP contains overlays.
31+
32+
$ export NEXTSTRAIN_AWS_BATCH_WORKDIR_URL="s3://nextstrain-tmp/$(python3 -c 'import uuid; print(uuid.uuid4())').zip"
33+
34+
$ aws s3 cp --quiet "$TESTDIR/data/workdir-with-augur-auspice-overlays.zip" "$NEXTSTRAIN_AWS_BATCH_WORKDIR_URL"
35+
36+
$ docker run --rm --env=NEXTSTRAIN_AWS_BATCH_{WORKDIR_URL,VERBOSE} --env=AWS_{ACCESS_KEY_ID,SECRET_ACCESS_KEY,SESSION_TOKEN} "$IMAGE" \
37+
> /sbin/entrypoint-aws-batch bash -euo pipefail -xc 'ls -lR . ../augur ../auspice'
38+
download: s3://nextstrain-tmp/*.zip to ../build.zip (glob)
39+
removing /nextstrain/augur because workdir ZIP contains ../augur/ overlay
40+
removing /nextstrain/auspice because workdir ZIP contains ../auspice/ overlay
41+
Archive: /nextstrain/build.zip
42+
extracting: reticulating
43+
extracting: splines
44+
creating: ../augur/
45+
creating: ../augur/a/
46+
creating: ../augur/a/b/
47+
creating: ../augur/a/b/c/
48+
extracting: ../augur/a/b/c/world.txt
49+
extracting: ../augur/a/b/c/hello.txt
50+
creating: ../augur/augur/
51+
extracting: ../augur/augur/__init__.py
52+
extracting: ../augur/README.md
53+
creating: ../auspice/
54+
+ ls -lR . ../augur ../auspice
55+
.:
56+
total 0
57+
-rw-rw-r-- 1 nextstrain nextstrain 0 Mar 10 21:46 reticulating
58+
-rw-rw-r-- 1 nextstrain nextstrain 0 Mar 10 21:46 splines
59+
60+
../augur:
61+
total 12
62+
-rw-rw-r-- 1 nextstrain nextstrain 22 Mar 10 21:45 README.md
63+
drwxrwxr-x 3 nextstrain nextstrain 4096 Mar 10 21:32 a
64+
drwxrwxr-x 2 nextstrain nextstrain 4096 Mar 10 21:46 augur
65+
66+
../augur/a:
67+
total 4
68+
drwxrwxr-x 3 nextstrain nextstrain 4096 Mar 10 21:32 b
69+
70+
../augur/a/b:
71+
total 4
72+
drwxrwxr-x 2 nextstrain nextstrain 4096 Mar 10 21:33 c
73+
74+
../augur/a/b/c:
75+
total 8
76+
-rw-rw-r-- 1 nextstrain nextstrain 6 Mar 10 21:32 hello.txt
77+
-rw-rw-r-- 1 nextstrain nextstrain 6 Mar 10 21:32 world.txt
78+
79+
../augur/augur:
80+
total 4
81+
-rw-rw-r-- 1 nextstrain nextstrain 34 Mar 10 21:46 __init__.py
82+
83+
../auspice:
84+
total 0
85+
upload: ../build.zip to s3://nextstrain-tmp/*.zip (glob)
86+
87+
…even when the workdir is not /nextstrain/build, e.g. when it's a sibling.
88+
89+
$ docker run --rm --env=NEXTSTRAIN_WORKDIR=/nextstrain/abc --env=NEXTSTRAIN_AWS_BATCH_{WORKDIR_URL,VERBOSE} --env=AWS_{ACCESS_KEY_ID,SECRET_ACCESS_KEY,SESSION_TOKEN} "$IMAGE" \
90+
> /sbin/entrypoint-aws-batch bash -euo pipefail -xc 'ls -lR . ../augur ../auspice'
91+
download: s3://nextstrain-tmp/*.zip to ../abc.zip (glob)
92+
removing /nextstrain/augur because workdir ZIP contains ../augur/ overlay
93+
removing /nextstrain/auspice because workdir ZIP contains ../auspice/ overlay
94+
Archive: /nextstrain/abc.zip
95+
extracting: reticulating
96+
extracting: splines
97+
creating: ../augur/
98+
creating: ../augur/a/
99+
creating: ../augur/a/b/
100+
creating: ../augur/a/b/c/
101+
extracting: ../augur/a/b/c/world.txt
102+
extracting: ../augur/a/b/c/hello.txt
103+
creating: ../augur/augur/
104+
extracting: ../augur/augur/__init__.py
105+
extracting: ../augur/README.md
106+
creating: ../auspice/
107+
+ ls -lR . ../augur ../auspice
108+
.:
109+
total 0
110+
-rw-rw-r-- 1 nextstrain nextstrain 0 Mar 10 21:46 reticulating
111+
-rw-rw-r-- 1 nextstrain nextstrain 0 Mar 10 21:46 splines
112+
113+
../augur:
114+
total 12
115+
-rw-rw-r-- 1 nextstrain nextstrain 22 Mar 10 21:45 README.md
116+
drwxrwxr-x 3 nextstrain nextstrain 4096 Mar 10 21:32 a
117+
drwxrwxr-x 2 nextstrain nextstrain 4096 Mar 10 21:46 augur
118+
119+
../augur/a:
120+
total 4
121+
drwxrwxr-x 3 nextstrain nextstrain 4096 Mar 10 21:32 b
122+
123+
../augur/a/b:
124+
total 4
125+
drwxrwxr-x 2 nextstrain nextstrain 4096 Mar 10 21:33 c
126+
127+
../augur/a/b/c:
128+
total 8
129+
-rw-rw-r-- 1 nextstrain nextstrain 6 Mar 10 21:32 hello.txt
130+
-rw-rw-r-- 1 nextstrain nextstrain 6 Mar 10 21:32 world.txt
131+
132+
../augur/augur:
133+
total 4
134+
-rw-rw-r-- 1 nextstrain nextstrain 34 Mar 10 21:46 __init__.py
135+
136+
../auspice:
137+
total 0
138+
upload: ../abc.zip to s3://nextstrain-tmp/*.zip (glob)
139+
140+
…but not when the workdir is somewhere completely different.
141+
142+
$ docker run --rm --env=NEXTSTRAIN_WORKDIR=/nextstrain/x/y/z --env=NEXTSTRAIN_AWS_BATCH_{WORKDIR_URL,VERBOSE} --env=AWS_{ACCESS_KEY_ID,SECRET_ACCESS_KEY,SESSION_TOKEN} "$IMAGE" \
143+
> /sbin/entrypoint-aws-batch bash -euo pipefail -xc 'ls -lR . ../augur ../auspice; realpath ../augur ../auspice'
144+
download: s3://nextstrain-tmp/*.zip to ../z.zip (glob)
145+
Archive: /nextstrain/x/y/z.zip
146+
extracting: reticulating
147+
extracting: splines
148+
creating: ../augur/
149+
creating: ../augur/a/
150+
creating: ../augur/a/b/
151+
creating: ../augur/a/b/c/
152+
extracting: ../augur/a/b/c/world.txt
153+
extracting: ../augur/a/b/c/hello.txt
154+
creating: ../augur/augur/
155+
extracting: ../augur/augur/__init__.py
156+
extracting: ../augur/README.md
157+
creating: ../auspice/
158+
+ ls -lR . ../augur ../auspice
159+
.:
160+
total 0
161+
-rw-rw-r-- 1 nextstrain nextstrain 0 Mar 10 21:46 reticulating
162+
-rw-rw-r-- 1 nextstrain nextstrain 0 Mar 10 21:46 splines
163+
164+
../augur:
165+
total 12
166+
-rw-rw-r-- 1 nextstrain nextstrain 22 Mar 10 21:45 README.md
167+
drwxrwxr-x 3 nextstrain nextstrain 4096 Mar 10 21:32 a
168+
drwxrwxr-x 2 nextstrain nextstrain 4096 Mar 10 21:46 augur
169+
170+
../augur/a:
171+
total 4
172+
drwxrwxr-x 3 nextstrain nextstrain 4096 Mar 10 21:32 b
173+
174+
../augur/a/b:
175+
total 4
176+
drwxrwxr-x 2 nextstrain nextstrain 4096 Mar 10 21:33 c
177+
178+
../augur/a/b/c:
179+
total 8
180+
-rw-rw-r-- 1 nextstrain nextstrain 6 Mar 10 21:32 hello.txt
181+
-rw-rw-r-- 1 nextstrain nextstrain 6 Mar 10 21:32 world.txt
182+
183+
../augur/augur:
184+
total 4
185+
-rw-rw-r-- 1 nextstrain nextstrain 34 Mar 10 21:46 __init__.py
186+
187+
../auspice:
188+
total 0
189+
+ realpath ../augur ../auspice
190+
/nextstrain/x/y/augur
191+
/nextstrain/x/y/auspice
192+
upload: ../z.zip to s3://nextstrain-tmp/*.zip (glob)

‎tests/data/workdir-with-augur-auspice-overlays.zip

1.51 KB
Binary file not shown.

‎tests/data/workdir-without-overlays.zip

292 Bytes
Binary file not shown.

0 commit comments

Comments
 (0)