You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: pages/nextflow.md
+2-6Lines changed: 2 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,10 +11,8 @@ Nextflow runs each workflow process in its own environment. When using container
11
11
pulls the required image automatically. On BioShell, you can override this behaviour by
12
12
creating a config file that points processes to the existing CVMFS images.
13
13
14
-
> **Note:** Nextflow always prioritises a `-config` file passed on the command line over the
15
-
> workflow's built-in `nextflow.config`.
14
+
{% include callout.html type="note" content="Nextflow always prioritises a `-config` file passed on the command line over the workflow's built-in `nextflow.config`." %}
16
15
17
-
---
18
16
19
17
## Using containers directly from CVMFS {#cvmfs-containers}
20
18
@@ -29,7 +27,6 @@ process {
29
27
}
30
28
```
31
29
32
-
---
33
30
34
31
## Using an installed SHPC module {#shpc-module}
35
32
@@ -44,9 +41,8 @@ process {
44
41
}
45
42
```
46
43
47
-
> **Note:**The `module` value must match exactly what appears in `module avail`.
44
+
{% include callout.html type="note" content="The `module` value must match exactly what appears in `module avail`." %}
Paste the output into the [BioShell Onboarding Portal](AUTHOR TO SUPPLY — add URL).
44
45
45
-
---
46
46
47
47
## Prerequisites {#prerequisites}
48
48
@@ -52,13 +52,13 @@ Paste the output into the [BioShell Onboarding Portal](AUTHOR TO SUPPLY — add
52
52
| Linux |`openssh-client` — install with `sudo apt install openssh-client` or `sudo dnf install openssh`|
53
53
| Windows | OpenSSH Client — built-in on Windows 10 (1809+) and Windows 11; enable via **Settings > Optional Features** if missing |
54
54
55
-
---
56
55
57
56
## Step 1: Generate the key pair {#generate}
58
57
59
58
The recommended algorithm is **Ed25519** — it is fast, secure, and compact. Use RSA 4096
60
59
only if a service requires it.
61
60
61
+
62
62
### macOS and Linux
63
63
64
64
```bash
@@ -110,7 +110,6 @@ Enter passphrase (empty for no passphrase):
110
110
A passphrase encrypts your private key at rest. It is recommended for any key that will
111
111
access production systems. Press **Enter** to skip.
112
112
113
-
---
114
113
115
114
## Step 2: Verify the key files {#verify}
116
115
@@ -121,6 +120,7 @@ After generation, two files are created:
121
120
|`id_ed25519`| Private key — never share this |
122
121
|`id_ed25519.pub`| Public key — share this with servers and services |
123
122
123
+
124
124
### macOS and Linux
125
125
126
126
```bash
@@ -133,7 +133,6 @@ ls -la ~/.ssh/
133
133
dir $env:USERPROFILE\.ssh\
134
134
```
135
135
136
-
---
137
136
138
137
## Step 3: Add the key to ssh-agent (optional) {#ssh-agent}
139
138
@@ -146,14 +145,12 @@ Run the relevant command once after login:
146
145
| Linux |`ssh-add ~/.ssh/id_ed25519`|
147
146
| Windows (PowerShell) |`ssh-add $env:USERPROFILE\.ssh\id_ed25519`|
148
147
149
-
> **Note:** On macOS the key is stored in Keychain and persists across reboots. On Linux
150
-
> you may need to re-add the key after each session unless you configure your shell to start
151
-
> the agent automatically.
148
+
{% include callout.html type="note" content="On macOS the key is stored in Keychain and persists across reboots. On Linux you may need to re-add the key after each session unless you configure your shell to start the agent automatically." %}
Copy file name to clipboardExpand all lines: pages/training-development.md
+21-11Lines changed: 21 additions & 11 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,18 +3,18 @@ title: Training development instructions
3
3
description: How to set up and prepare a BioShell training environment for the BioCommons Training Cooperative, including VM configuration, CVMFS use, trainee directory layout, and pre-snapshot requirements.
4
4
---
5
5
6
-
> **Draft:** These instructions are a work in progress. If anything is unclear, reach out to Mitchell directly. Feedback is very welcome.
6
+
{% include callout.html type="warning" content="**Draft:** These instructions are a work in progress. If anything is unclear, reach out to Mitchell directly. Feedback is very welcome." %}
7
7
8
8
This guide is for **training developers** (`tdevNN` users) who are setting up BioShell training environments for the BioCommons Training Cooperative. You will configure tools, build training materials, and prepare a template that gets automatically applied to trainee accounts when VMs are provisioned. When a VM is built from your snapshot by Giorgia or Mitchell, the provisioning script runs automatically: the trainee's username will be `userNN` (for example `user1`), their password is generated deterministically from their username and is unique per VM, the contents of `/etc/skel/` at snapshot time become the trainee's home directory, and `/etc/skel/` is cleared afterwards to reduce VM image size.
9
9
10
-
---
11
10
12
11
## VM setup and access {#vm-setup}
13
12
14
13
### Launch a VM instance
15
14
16
15
VM instances are provisioned by Giorgia or Mitchell. Each dev machine is named with the prefix `D` followed by a number (for example `D1`).
17
16
17
+
18
18
### Log in via SSH
19
19
20
20
The training team will provide you with a username and IP address. Your username follows the format `tdevNN`, where `NN` matches your VM's prefix number.
@@ -34,16 +34,16 @@ Once you have an active SSH connection, open your browser and go to:
34
34
35
35
Replace `<IP_Address>` with the IP address provided by the training team.
36
36
37
-
---
38
37
39
38
## Developing training materials {#developing-materials}
40
39
41
40
Set up your home directory (`/home/tdevNN`) exactly as you want trainees to experience it. Build and test your workflows there, using CVMFS containers and references throughout. The goal is a working, self-contained environment that a trainee could follow from start to finish.
42
41
43
-
> **Keep data small:** When the Australian BioCommons dataset repository is ready, it will be the main access point for training data. Like CVMFS, data stored there will not count against VM size. Until then, use the minimum input data needed to demonstrate the workflow. If your ideal dataset is too large to copy to every trainee home directory, see [If resources are not available on CVMFS](#non-cvmfs) for how to pull it at boot time instead.
42
+
{% include callout.html type="tip" content="**Keep data small:** When the Australian BioCommons dataset repository is ready, it will be the main access point for training data. Like CVMFS, data stored there will not count against VM size. Until then, use the minimum input data needed to demonstrate the workflow. If your ideal dataset is too large to copy to every trainee home directory, see [If resources are not available on CVMFS](#non-cvmfs) for how to pull it at boot time instead." %}
44
43
45
44
Once everything works end-to-end, your home directory becomes the template.
46
45
46
+
47
47
### Disk budget {#disk-budget}
48
48
49
49
The base VM template takes up approximately **12 GB** on a **30 GB** disk, leaving roughly **16 GB** of usable space. That space has to cover:
@@ -52,7 +52,7 @@ The base VM template takes up approximately **12 GB** on a **30 GB** disk, leavi
52
52
- CVMFS-cached container layers and reference data (written to the CVMFS cache on first use)
53
53
- Trainee working files generated during the session
54
54
55
-
> **Watch the disk:** If the disk fills up during user creation, the `useradd` step will fail and the trainee account will not be created. Keep `/etc/skel/` as lean as possible.
55
+
{% include callout.html type="tip" content="**Watch the disk:** If the disk fills up during user creation, the `useradd` step will fail and the trainee account will not be created. Keep `/etc/skel/` as lean as possible." %}
56
56
57
57
Check how much space you have with:
58
58
@@ -115,7 +115,8 @@ This creates a symlink in your current directory pointing to the CVMFS source. Y
> **Symlinks must use relative paths or CVMFS paths.** If you create a symlink pointing to an absolute path inside your home directory (for example `/home/tdev01/data/reference.fa`), that path will not exist on trainee VMs and the link will be broken. Always point symlinks either at a CVMFS path (absolute paths under `/cvmfs/` are safe because CVMFS is available on all VMs) or at a relative path within the directory structure that will be copied to `/etc/skel/`.
118
+
{% include callout.html type="note" content="**Symlinks must use relative paths or CVMFS paths.** If you create a symlink pointing to an absolute path inside your home directory (for example `/home/tdev01/data/reference.fa`), that path will not exist on trainee VMs and the link will be broken. Always point symlinks either at a CVMFS path (absolute paths under `/cvmfs/` are safe because CVMFS is available on all VMs) or at a relative path within the directory structure that will be copied to `/etc/skel/`." %}
119
+
119
120
120
121
### RStudio and JupyterLab environments {#interactive-envs}
R writes packages to `~/R/x86_64-pc-linux-gnu-library/<version>/` by default. This directory will be picked up automatically when you copy your home directory to `/etc/skel/`.
134
135
135
-
> **Watch the size:** R packages for workflows like scRNA-seq (Seurat, Bioconductor suites) can be several gigabytes. Check the size before copying:
136
+
{% include callout.html type="tip" content="**Watch the size:** R packages for workflows like scRNA-seq (Seurat, Bioconductor suites) can be several gigabytes." %}
137
+
138
+
Check the size before copying:
136
139
137
140
```bash
138
141
du -sh ~/R/
@@ -156,6 +159,7 @@ du -sh ~/.local/
156
159
157
160
If a required container or reference dataset is not on CVMFS and is too large to include directly in the trainee directory, it needs to be fetched at boot time via the VM provisioning script. Prepare the required scripts and share them with Giorgia or Mitchell.
158
161
162
+
159
163
#### Adding a pull step to the provisioning script
160
164
161
165
Downloads should be added before the `useradd` block, targeting the relevant folder inside `/etc/skel/`:
Resources pulled this way will appear in the correct location in the trainee's home directory, matching the layout described in [Trainee directory layout](#directory-layout).
174
178
175
-
> **Note:** Boot-time pulls count toward your disk budget. Factor their size into your 30 GB total.
179
+
{% include callout.html type="note" content="Boot-time pulls count toward your disk budget. Factor their size into your 30 GB total." %}
180
+
176
181
177
182
### Trainee directory layout {#directory-layout}
178
183
@@ -221,16 +226,17 @@ Once you are confident the template is correct, you can clean up your own home d
221
226
rm -rf ~/*
222
227
```
223
228
224
-
> **Tip:**Ask Giorgia or Mitchell to take a snapshot at key checkpoints if you want a backup before making significant changes.
229
+
{% include callout.html type="tip" content="Ask Giorgia or Mitchell to take a snapshot at key checkpoints if you want a backup before making significant changes." %}
225
230
226
-
---
227
231
228
232
## Pre-snapshot requirements {#pre-snapshot}
229
233
234
+
230
235
### Cloud-init datasource check {#cloud-init}
231
236
232
237
Running `apt upgrade` during VM setup can silently change cloud-init configuration, which causes all VMs built from your snapshot to inherit the wrong hostname. Please check and fix this before asking the VM to be snapshot.
233
238
239
+
234
240
#### Step 1: Check for the offending apt file
235
241
236
242
```bash
@@ -257,13 +263,16 @@ datasource_list:
257
263
- OpenStack
258
264
```
259
265
266
+
260
267
#### Step 3: Clean cloud-init state
261
268
262
269
```bash
263
270
sudo cloud-init clean --logs
264
271
```
265
272
266
-
> **Important:**`cloud-init clean` deletes `/etc/cloud/ds-identify.cfg`. You will need to recreate it:
273
+
{% include callout.html type="important" content="`cloud-init clean` deletes `/etc/cloud/ds-identify.cfg`." %}
0 commit comments