Skip to content

Commit 54ebe4e

Browse files
committed
Merge branch 'main' of https://github.com/fhdsl/FH_Cluster201 into main
2 parents bd62ca5 + 98b078b commit 54ebe4e

8 files changed

+215
-148
lines changed

docs/02-chapter_of_course.md

+2-59
Original file line numberDiff line numberDiff line change
@@ -1,64 +1,7 @@
11

2-
There are many ways to leverage a local cluster and cloud resources for computing.
32

43

5-
# Interactive Modes
4+
# Data Storage
65

7-
## Rhino (Login Nodes)
6+
Much about how you structure your computing work depends on where and how your data are stored. We'll discuss a bit about how data storage might influence your computing choices in this chapter.
87

9-
10-
- see what software you can use that's already installed
11-
- do small, interactive jobs using software not installed on your local computer
12-
- test small jobs prior to scaling up
13-
- can run cronjobs from here to schedule things to happen regularly
14-
15-
16-
### `Grabnode`
17-
18-
- runs one big job with limited configuration options
19-
20-
# Non-Interactive Modes
21-
22-
## SLURM (S-commands)
23-
24-
SLURM job submission to gizmo:
25-
26-
- making jobs resilient to connectivity issues (grabnode sessions end your job if you lose connectivity)
27-
- no need to babysit the job this way
28-
- generates job logs for troubleshooting
29-
- uses the SLURM scheduler and better lab job allocation (helps you get resources sooner and in coordination with the rest of your lab and the Hutch users more broadly)
30-
- srun, sbatch, etc. gives you more control and options for configuration that let your jobs run better in many ways
31-
32-
- SLURM array jobs
33-
34-
35-
### Workflow managers
36-
37-
- good when you have multi-step processes that need to occur together
38-
- when your multi-step processes have dependencies that conflict: allows task isolation and management
39-
- good when you're working on multiple different workflows at the same time
40-
- want to leverage the cluster allocation very efficiently by breaking jobs up
41-
- want to prepare a workflow for better reproducibility, sharing, and expanding to the cloud
42-
43-
#### Cromwell
44-
45-
## Cloud computing
46-
47-
- once a workflow has been optimized (likely using a workflow manager above), leveraging the cloud for large, production workflows can help your lab's usage of the cluster while also encouraging reproducibility
48-
- Workflows can ideally be run by others, not just those with access to the Hutch resources
49-
50-
- great when you do not have access to a local cluster or when you want to use a commercial platform for running computational work.
51-
52-
53-
# Sharing and Collaboration
54-
55-
Issues like sharable code, sharable software environments and compute resource optimization become more important when you want to share and collaborate.
56-
57-
58-
- GitHub
59-
- Dockerhub/Quay.io
60-
- Dockstore
61-
62-
- Terra
63-
- DNA Nexus
64-
- other people's resources (like their non-SLURM cluster or maybe they use Azure or Google or AWS primarily?)

docs/404.html

+2-17
Original file line numberDiff line numberDiff line change
@@ -78,24 +78,9 @@
7878
<li class="chapter" data-level="0.2" data-path="index.html"><a href="index.html#target-audience"><i class="fa fa-check"></i><b>0.2</b> Target Audience</a></li>
7979
<li class="chapter" data-level="0.3" data-path="index.html"><a href="index.html#curriculum"><i class="fa fa-check"></i><b>0.3</b> Curriculum</a></li>
8080
</ul></li>
81-
<li class="chapter" data-level="1" data-path="interactive-modes.html"><a href="interactive-modes.html"><i class="fa fa-check"></i><b>1</b> Interactive Modes</a>
82-
<ul>
83-
<li class="chapter" data-level="1.1" data-path="interactive-modes.html"><a href="interactive-modes.html#rhino-login-nodes"><i class="fa fa-check"></i><b>1.1</b> Rhino (Login Nodes)</a></li>
84-
<li class="chapter" data-level="1.2" data-path="interactive-modes.html"><a href="interactive-modes.html#grabnode"><i class="fa fa-check"></i><b>1.2</b> <code>Grabnode</code></a></li>
85-
</ul></li>
86-
<li class="chapter" data-level="2" data-path="non-interactive-modes.html"><a href="non-interactive-modes.html"><i class="fa fa-check"></i><b>2</b> Non-Interactive Modes</a>
87-
<ul>
88-
<li class="chapter" data-level="2.1" data-path="non-interactive-modes.html"><a href="non-interactive-modes.html#slurm-s-commands"><i class="fa fa-check"></i><b>2.1</b> SLURM (S-commands)</a></li>
89-
<li class="chapter" data-level="2.2" data-path="non-interactive-modes.html"><a href="non-interactive-modes.html#workflow-managers"><i class="fa fa-check"></i><b>2.2</b> Workflow managers</a>
90-
<ul>
91-
<li class="chapter" data-level="2.2.1" data-path="non-interactive-modes.html"><a href="non-interactive-modes.html#cromwell"><i class="fa fa-check"></i><b>2.2.1</b> Cromwell</a></li>
92-
<li class="chapter" data-level="2.2.2" data-path="non-interactive-modes.html"><a href="non-interactive-modes.html#nextflow"><i class="fa fa-check"></i><b>2.2.2</b> Nextflow</a></li>
93-
</ul></li>
94-
<li class="chapter" data-level="2.3" data-path="non-interactive-modes.html"><a href="non-interactive-modes.html#cloud-computing"><i class="fa fa-check"></i><b>2.3</b> Cloud computing</a></li>
95-
</ul></li>
96-
<li class="chapter" data-level="3" data-path="sharing-and-collaboration.html"><a href="sharing-and-collaboration.html"><i class="fa fa-check"></i><b>3</b> Sharing and Collaboration</a></li>
81+
<li class="chapter" data-level="1" data-path="data-storage.html"><a href="data-storage.html"><i class="fa fa-check"></i><b>1</b> Data Storage</a></li>
9782
<li class="chapter" data-level="" data-path="about-the-authors.html"><a href="about-the-authors.html"><i class="fa fa-check"></i>About the Authors</a></li>
98-
<li class="chapter" data-level="4" data-path="references.html"><a href="references.html"><i class="fa fa-check"></i><b>4</b> References</a></li>
83+
<li class="chapter" data-level="2" data-path="references.html"><a href="references.html"><i class="fa fa-check"></i><b>2</b> References</a></li>
9984
<li class="divider"></li>
10085
<p style="text-align:center;"> <a href="https://github.com/jhudsl/OTTR_Template" target="blank" > This content was published with</a> <a href="https://bookdown.org/" target="blank"> bookdown by: </a> </p>
10186
<p style="text-align:center;"> <a href="https://hutchdatascience.org/"> The Fred Hutch Data Science Lab </a></p>

docs/about-the-authors.html

+4-19
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@
3030
<meta name="apple-mobile-web-app-status-bar-style" content="black" />
3131

3232
<link rel="shortcut icon" href="assets/favicon.ico" type="image/x-icon" />
33-
<link rel="prev" href="sharing-and-collaboration.html"/>
33+
<link rel="prev" href="data-storage.html"/>
3434
<link rel="next" href="references.html"/>
3535
<script src="libs/header-attrs-2.10/header-attrs.js"></script>
3636
<script src="libs/jquery-3.6.0/jquery-3.6.0.min.js"></script>
@@ -78,24 +78,9 @@
7878
<li class="chapter" data-level="0.2" data-path="index.html"><a href="index.html#target-audience"><i class="fa fa-check"></i><b>0.2</b> Target Audience</a></li>
7979
<li class="chapter" data-level="0.3" data-path="index.html"><a href="index.html#curriculum"><i class="fa fa-check"></i><b>0.3</b> Curriculum</a></li>
8080
</ul></li>
81-
<li class="chapter" data-level="1" data-path="interactive-modes.html"><a href="interactive-modes.html"><i class="fa fa-check"></i><b>1</b> Interactive Modes</a>
82-
<ul>
83-
<li class="chapter" data-level="1.1" data-path="interactive-modes.html"><a href="interactive-modes.html#rhino-login-nodes"><i class="fa fa-check"></i><b>1.1</b> Rhino (Login Nodes)</a></li>
84-
<li class="chapter" data-level="1.2" data-path="interactive-modes.html"><a href="interactive-modes.html#grabnode"><i class="fa fa-check"></i><b>1.2</b> <code>Grabnode</code></a></li>
85-
</ul></li>
86-
<li class="chapter" data-level="2" data-path="non-interactive-modes.html"><a href="non-interactive-modes.html"><i class="fa fa-check"></i><b>2</b> Non-Interactive Modes</a>
87-
<ul>
88-
<li class="chapter" data-level="2.1" data-path="non-interactive-modes.html"><a href="non-interactive-modes.html#slurm-s-commands"><i class="fa fa-check"></i><b>2.1</b> SLURM (S-commands)</a></li>
89-
<li class="chapter" data-level="2.2" data-path="non-interactive-modes.html"><a href="non-interactive-modes.html#workflow-managers"><i class="fa fa-check"></i><b>2.2</b> Workflow managers</a>
90-
<ul>
91-
<li class="chapter" data-level="2.2.1" data-path="non-interactive-modes.html"><a href="non-interactive-modes.html#cromwell"><i class="fa fa-check"></i><b>2.2.1</b> Cromwell</a></li>
92-
<li class="chapter" data-level="2.2.2" data-path="non-interactive-modes.html"><a href="non-interactive-modes.html#nextflow"><i class="fa fa-check"></i><b>2.2.2</b> Nextflow</a></li>
93-
</ul></li>
94-
<li class="chapter" data-level="2.3" data-path="non-interactive-modes.html"><a href="non-interactive-modes.html#cloud-computing"><i class="fa fa-check"></i><b>2.3</b> Cloud computing</a></li>
95-
</ul></li>
96-
<li class="chapter" data-level="3" data-path="sharing-and-collaboration.html"><a href="sharing-and-collaboration.html"><i class="fa fa-check"></i><b>3</b> Sharing and Collaboration</a></li>
81+
<li class="chapter" data-level="1" data-path="data-storage.html"><a href="data-storage.html"><i class="fa fa-check"></i><b>1</b> Data Storage</a></li>
9782
<li class="chapter" data-level="" data-path="about-the-authors.html"><a href="about-the-authors.html"><i class="fa fa-check"></i>About the Authors</a></li>
98-
<li class="chapter" data-level="4" data-path="references.html"><a href="references.html"><i class="fa fa-check"></i><b>4</b> References</a></li>
83+
<li class="chapter" data-level="2" data-path="references.html"><a href="references.html"><i class="fa fa-check"></i><b>2</b> References</a></li>
9984
<li class="divider"></li>
10085
<p style="text-align:center;"> <a href="https://github.com/jhudsl/OTTR_Template" target="blank" > This content was published with</a> <a href="https://bookdown.org/" target="blank"> bookdown by: </a> </p>
10186
<p style="text-align:center;"> <a href="https://hutchdatascience.org/"> The Fred Hutch Data Science Lab </a></p>
@@ -236,7 +221,7 @@ <h1>About the Authors</h1>
236221
</div>
237222
</div>
238223
</div>
239-
<a href="sharing-and-collaboration.html" class="navigation navigation-prev " aria-label="Previous page"><i class="fa fa-angle-left"></i></a>
224+
<a href="data-storage.html" class="navigation navigation-prev " aria-label="Previous page"><i class="fa fa-angle-left"></i></a>
240225
<a href="references.html" class="navigation navigation-next " aria-label="Next page"><i class="fa fa-angle-right"></i></a>
241226
</div>
242227
</div>

docs/data-storage.html

+194
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,194 @@
1+
<!DOCTYPE html>
2+
<html lang="" xml:lang="">
3+
<head>
4+
5+
<meta charset="utf-8" />
6+
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
7+
<title>Chapter 1 Data Storage | FH Cluster 102</title>
8+
<meta name="description" content="A guide for effective use of high performance computing resources for research." />
9+
<meta name="generator" content="bookdown 0.24 and GitBook 2.6.7" />
10+
11+
<meta property="og:title" content="Chapter 1 Data Storage | FH Cluster 102" />
12+
<meta property="og:type" content="book" />
13+
14+
15+
<meta property="og:description" content="A guide for effective use of high performance computing resources for research." />
16+
17+
18+
<meta name="twitter:card" content="summary" />
19+
<meta name="twitter:title" content="Chapter 1 Data Storage | FH Cluster 102" />
20+
21+
<meta name="twitter:description" content="A guide for effective use of high performance computing resources for research." />
22+
23+
24+
25+
26+
<meta name="date" content="2022-12-30" />
27+
28+
<meta name="viewport" content="width=device-width, initial-scale=1" />
29+
<meta name="apple-mobile-web-app-capable" content="yes" />
30+
<meta name="apple-mobile-web-app-status-bar-style" content="black" />
31+
32+
<link rel="shortcut icon" href="assets/favicon.ico" type="image/x-icon" />
33+
<link rel="prev" href="index.html"/>
34+
<link rel="next" href="about-the-authors.html"/>
35+
<script src="libs/header-attrs-2.10/header-attrs.js"></script>
36+
<script src="libs/jquery-3.6.0/jquery-3.6.0.min.js"></script>
37+
<script src="https://cdn.jsdelivr.net/npm/[email protected]/dist/fuse.min.js"></script>
38+
<link href="libs/gitbook-2.6.7/css/style.css" rel="stylesheet" />
39+
<link href="libs/gitbook-2.6.7/css/plugin-table.css" rel="stylesheet" />
40+
<link href="libs/gitbook-2.6.7/css/plugin-bookdown.css" rel="stylesheet" />
41+
<link href="libs/gitbook-2.6.7/css/plugin-highlight.css" rel="stylesheet" />
42+
<link href="libs/gitbook-2.6.7/css/plugin-search.css" rel="stylesheet" />
43+
<link href="libs/gitbook-2.6.7/css/plugin-fontsettings.css" rel="stylesheet" />
44+
<link href="libs/gitbook-2.6.7/css/plugin-clipboard.css" rel="stylesheet" />
45+
46+
47+
48+
49+
50+
51+
52+
53+
<link href="libs/anchor-sections-1.0.1/anchor-sections.css" rel="stylesheet" />
54+
<script src="libs/anchor-sections-1.0.1/anchor-sections.js"></script>
55+
56+
57+
58+
59+
<link rel="stylesheet" href="assets/style.css" type="text/css" />
60+
</head>
61+
62+
<body>
63+
64+
65+
66+
<div class="book without-animation with-summary font-size-2 font-family-1" data-basepath=".">
67+
68+
<div class="book-summary">
69+
<nav role="navigation">
70+
71+
<ul class="summary">
72+
<a href="https://hutchdatascience.org/" target="_blank"><img src="assets/big-dasl-stacked.png" style="width: 80%; padding-left: 34px; padding-top: 8px;"</a>
73+
74+
<li class="divider"></li>
75+
<li class="chapter" data-level="" data-path="index.html"><a href="index.html"><i class="fa fa-check"></i>About this Book</a>
76+
<ul>
77+
<li class="chapter" data-level="0.1" data-path="index.html"><a href="index.html#motivation"><i class="fa fa-check"></i><b>0.1</b> Motivation</a></li>
78+
<li class="chapter" data-level="0.2" data-path="index.html"><a href="index.html#target-audience"><i class="fa fa-check"></i><b>0.2</b> Target Audience</a></li>
79+
<li class="chapter" data-level="0.3" data-path="index.html"><a href="index.html#curriculum"><i class="fa fa-check"></i><b>0.3</b> Curriculum</a></li>
80+
</ul></li>
81+
<li class="chapter" data-level="1" data-path="data-storage.html"><a href="data-storage.html"><i class="fa fa-check"></i><b>1</b> Data Storage</a></li>
82+
<li class="chapter" data-level="" data-path="about-the-authors.html"><a href="about-the-authors.html"><i class="fa fa-check"></i>About the Authors</a></li>
83+
<li class="chapter" data-level="2" data-path="references.html"><a href="references.html"><i class="fa fa-check"></i><b>2</b> References</a></li>
84+
<li class="divider"></li>
85+
<p style="text-align:center;"> <a href="https://github.com/jhudsl/OTTR_Template" target="blank" > This content was published with</a> <a href="https://bookdown.org/" target="blank"> bookdown by: </a> </p>
86+
<p style="text-align:center;"> <a href="https://hutchdatascience.org/"> The Fred Hutch Data Science Lab </a></p>
87+
<p style="text-align:center; font-size: 12px;"> <a href="https://github.com/rstudio4edu/rstudio4edu-book/"> Style adapted from: rstudio4edu-book </a> <a href ="https://creativecommons.org/licenses/by/2.0/"> (CC-BY 2.0) </a></p>
88+
<p style="padding-left: 40px;"><div class="trapezoid" style = "padding-left: 40px;"><span> <a href="https://forms.gle/W6Mg4rzuMK6Yk3Am8"> Click here to provide feedback</a> <img src="assets/itcr_arrow.png" style=" width: 10%" ></span></div></p>
89+
</ul>
90+
91+
</nav>
92+
</div>
93+
94+
<div class="book-body">
95+
<div class="body-inner">
96+
<div class="book-header" role="navigation">
97+
<h1>
98+
<i class="fa fa-circle-o-notch fa-spin"></i><a href="./">FH Cluster 102</a>
99+
</h1>
100+
</div>
101+
102+
<div class="page-wrapper" tabindex="-1" role="main">
103+
<div class="page-inner">
104+
105+
<section class="normal" id="section-">
106+
<head>
107+
<meta name="viewport" content="width=device-width,minimum-scale=1.0,maximum-scale=10.0,initial-scale=1.0">
108+
<!--script src="https://kit.fontawesome.com/6a26f47516.js"></script-->
109+
<!--<script src="assets/hideOutput.js"></script>-->
110+
<link href="assets/style.css" rel="stylesheet">
111+
</head>
112+
113+
114+
115+
<div class="hero-image-container">
116+
<img class= "hero-image" src="assets/dasl_thin_main_image.png">
117+
</div>
118+
<div id="data-storage" class="section level1" number="1">
119+
<h1><span class="header-section-number">Chapter 1</span> Data Storage</h1>
120+
<p>Much about how you structure your computing work depends on where and how your data are stored. We’ll discuss a bit about how data storage might influence your computing choices in this chapter.</p>
121+
122+
</div>
123+
<hr>
124+
<center>
125+
<div class="footer">
126+
All illustrations <a href="https://creativecommons.org/licenses/by/4.0/">CC-BY. </a>
127+
<br>
128+
All other materials <a href= "https://creativecommons.org/licenses/by/4.0/"> CC-BY </a> unless noted otherwise.
129+
<a href="https://hutchdatascience.org/" target="_blank"><img src="https://hutchdatascience.org/images/crazy-idea-wide.png" style="width: 80%; padding-left: 15px; padding-top: 8px;"</a>
130+
</div>
131+
</center>
132+
</section>
133+
134+
</div>
135+
</div>
136+
</div>
137+
<a href="index.html" class="navigation navigation-prev " aria-label="Previous page"><i class="fa fa-angle-left"></i></a>
138+
<a href="about-the-authors.html" class="navigation navigation-next " aria-label="Next page"><i class="fa fa-angle-right"></i></a>
139+
</div>
140+
</div>
141+
<script src="libs/gitbook-2.6.7/js/app.min.js"></script>
142+
<script src="libs/gitbook-2.6.7/js/clipboard.min.js"></script>
143+
<script src="libs/gitbook-2.6.7/js/plugin-search.js"></script>
144+
<script src="libs/gitbook-2.6.7/js/plugin-sharing.js"></script>
145+
<script src="libs/gitbook-2.6.7/js/plugin-fontsettings.js"></script>
146+
<script src="libs/gitbook-2.6.7/js/plugin-bookdown.js"></script>
147+
<script src="libs/gitbook-2.6.7/js/jquery.highlight.js"></script>
148+
<script src="libs/gitbook-2.6.7/js/plugin-clipboard.js"></script>
149+
<script>
150+
gitbook.require(["gitbook"], function(gitbook) {
151+
gitbook.start({
152+
"sharing": {
153+
"github": false,
154+
"facebook": true,
155+
"twitter": true,
156+
"linkedin": false,
157+
"weibo": false,
158+
"instapaper": false,
159+
"vk": false,
160+
"whatsapp": false,
161+
"all": ["facebook", "twitter", "linkedin", "weibo", "instapaper"]
162+
},
163+
"fontsettings": {
164+
"theme": "white",
165+
"family": "sans",
166+
"size": 2
167+
},
168+
"edit": {
169+
"link": null,
170+
"text": null
171+
},
172+
"history": {
173+
"link": null,
174+
"text": null
175+
},
176+
"view": {
177+
"link": null,
178+
"text": null
179+
},
180+
"download": null,
181+
"search": {
182+
"engine": "fuse",
183+
"options": null
184+
},
185+
"toc": {
186+
"collapse": "section"
187+
}
188+
});
189+
});
190+
</script>
191+
192+
</body>
193+
194+
</html>

0 commit comments

Comments
 (0)