forked from datacarpentry/cloud-genomics
-
Notifications
You must be signed in to change notification settings - Fork 1
/
0.cloud-introduction.Rmd
367 lines (314 loc) · 12.4 KB
/
0.cloud-introduction.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
---
title: "Cloud Introduction"
author: "Data Carpentry"
date: "Tuesday, March 24, 2015"
---
## Learning Objectives
* Describe reasons why using a remote computer system is useful.
* Understand terms related to cloud and high-performance (HPC) computing.
* Describe the differences between HPC and cloud computing.
* Access resources related to HPC and cloud computing.
### What are some of reasons to access a remote computer system?
* Your computer does not have enough resources to run the desired analysis (memory, processors, disk space, network bandwidth).
* You want to produce results faster than your computer can.
* You cannot install software in your computer (application does not have support for your operating system, conflicts with other existing applications)
### Main Computational Resources
* Find how much disk space is available in your system. Check that you have enough space to get and extract the input file of :
```{r, eval=FALSE}
df -h
wget http://www.hpa-bioinformatics.org.uk/lgp/resource/Sample280.fastq.gz
ls -lah Sample280.fastq.gz
gunzip Sample280.fastq.gz
ls -lah Sample280.fastq
```
The output on iPlant Atmosphere:
```{r, eval=FALSE}
--2015-03-25 09:40:33-- http://www.hpa-bioinformatics.org.uk/lgp/resource/Sample280.fastq.gz
Resolving www.hpa-bioinformatics.org.uk... 194.74.226.185
Connecting to www.hpa-bioinformatics.org.uk|194.74.226.185|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 215949113 (206M) [application/x-gzip]
Saving to: \u201cSample280.fastq.gz\u201d
100%[==========================================================================================>] 215,949,113 213K/s in 16m 48s
2015-03-25 09:57:21 (209 KB/s) - \u201cSample280.fastq.gz\u201d saved [215949113/215949113]
-rw-r--r-- 1 dnasubway01 iplant-everyone 206M Jun 29 2011 Sample280.fastq.gz
-rw-r--r-- 1 dnasubway01 iplant-everyone 627M Jun 29 2011 Sample280.fastq
```
The output on Amazon EC2:
```{r, eval=FALSE}
--2015-03-25 15:30:23-- http://www.hpa-bioinformatics.org.uk/lgp/resource/Sample280.fastq.gz
Resolving www.hpa-bioinformatics.org.uk (www.hpa-bioinformatics.org.uk)... 194.74.226.185
Connecting to www.hpa-bioinformatics.org.uk (www.hpa-bioinformatics.org.uk)|194.74.226.185|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 215949113 (206M) [application/x-gzip]
Saving to: \u2018Sample280.fastq.gz\u2019
Sample280.fastq.gz 100%[===============================================================>] 205.94M 376KB/s in 9m 23s
2015-03-25 15:39:46 (375 KB/s) - \u2018Sample280.fastq.gz\u2019 saved [215949113/215949113]
-rw-rw-r-- 1 ec2-user ec2-user 206M Jun 29 2011 Sample280.fastq.gz
-rw-rw-r-- 1 ec2-user ec2-user 627M Jun 29 2011 Sample280.fastq
```
The output on Azure:
```{r, eval=FALSE}
--2015-03-25 14:37:18-- http://www.hpa-bioinformatics.org.uk/lgp/resource/Sample280.fastq.gz
Resolving www.hpa-bioinformatics.org.uk... 194.74.226.185
Connecting to www.hpa-bioinformatics.org.uk|194.74.226.185|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 215949113 (206M) [application/x-gzip]
Saving to: \u201cSample280.fastq.gz\u201d
100%[===========================================================================================================================================>] 215,949,113 216K/s in 16m 26s
2015-03-25 14:53:44 (214 KB/s) - \u201cSample280.fastq.gz\u201d saved [215949113/215949113]
-rw-r--r--. 1 root root 206M Jun 29 2011 Sample280.fastq.gz
-rw-r--r--. 1 root root 627M Mar 25 14:57 Sample280.fastq
```
* Find how many cores and processors are available in your system:
```{r, eval=FALSE}
cat /proc/cpuinfo
```
```{r, eval=FALSE}
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Westmere E56xx/L56xx/X56xx (Nehalem-C)
stepping : 1
cpu MHz : 2400.084
cache size : 4096 KB
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good unfair_spinlock pni pclmulqdq vmx ssse3 cx16 pcid sse4_1 sse4_2 popcnt aes hypervisor lahf_lm
bogomips : 4800.16
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Westmere E56xx/L56xx/X56xx (Nehalem-C)
stepping : 1
cpu MHz : 2400.084
cache size : 4096 KB
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good unfair_spinlock pni pclmulqdq vmx ssse3 cx16 pcid sse4_1 sse4_2 popcnt aes hypervisor lahf_lm
bogomips : 4800.16
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
```
On an Amazon EC2 instance, the following output shows availability of 2 Intel Xeon processors:
```{r, eval=FALSE}
cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 62
model name : Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
stepping : 4
microcode : 0x415
cpu MHz : 2494.060
cache size : 25600 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm xsaveopt fsgsbase smep erms
bogomips : 4988.12
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 62
model name : Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
stepping : 4
microcode : 0x415
cpu MHz : 2494.060
cache size : 25600 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
apicid : 2
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm xsaveopt fsgsbase smep erms
bogomips : 4988.12
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
```
On an Microsoft Azure instance, the following output shows availability of 2 Intel Xeon processors:
```{r, eval=FALSE}
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2698B v3 @ 2.00GHz
stepping : 2
cpu MHz : 1995.365
cache size : 40960 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good unfair_spinlock pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm xsaveopt fsgsbase bmi1 avx2 smep bmi2 erms
bogomips : 3990.73
clflush size : 64
cache_alignment : 64
address sizes : 42 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2698B v3 @ 2.00GHz
stepping : 2
cpu MHz : 1995.365
cache size : 40960 KB
physical id : 0
siblings : 4
core id : 1
cpu cores : 4
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good unfair_spinlock pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm xsaveopt fsgsbase bmi1 avx2 smep bmi2 erms
bogomips : 3990.73
clflush size : 64
cache_alignment : 64
address sizes : 42 bits physical, 48 bits virtual
power management:
processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2698B v3 @ 2.00GHz
stepping : 2
cpu MHz : 1995.365
cache size : 40960 KB
physical id : 0
siblings : 4
core id : 2
cpu cores : 4
apicid : 2
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good unfair_spinlock pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm xsaveopt fsgsbase bmi1 avx2 smep bmi2 erms
bogomips : 3990.73
clflush size : 64
cache_alignment : 64
address sizes : 42 bits physical, 48 bits virtual
power management:
processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2698B v3 @ 2.00GHz
stepping : 2
cpu MHz : 1995.365
cache size : 40960 KB
physical id : 0
siblings : 4
core id : 3
cpu cores : 4
apicid : 3
initial apicid : 3
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good unfair_spinlock pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm xsaveopt fsgsbase bmi1 avx2 smep bmi2 erms
bogomips : 3990.73
clflush size : 64
cache_alignment : 64
address sizes : 42 bits physical, 48 bits virtual
power management:
```
* Find how many processes are running in your system, and how much resources is each consuming:
```{r, eval=FALSE}
top
```
* Find how long it takes to run an application:
```{r, eval=FALSE}
time <your_application>
```
* For example, let's measure the time it takes to separate the R1 sequences into a separate file:
```{r, eval=FALSE}
time awk "/\/1$/{print; getline; print}" Sample280.fastq > Sample280_1.fastq
```
The output on iPlant Atmosphere, Amazon EC2, and Microsoft Azure are shown respectively below, indicating that EC2 took half of the time of Azure.
```{r, eval=FALSE}
real 0m20.893s
user 0m19.573s
sys 0m1.309s
real 0m14.856s
user 0m14.260s
sys 0m0.592s
real 0m36.294s
user 0m12.762s
sys 0m0.432s
```
### Distributed System Definitions and stacks:
(Note that many definitions exist for these terms)
* Distributed application: an application that can be executed on a distributed system platform (e.g., mpiBLAST)
* Distributed system platform: software layers that facilitates coordination and management of a distributed system (e.g., queue-based system, and MapReduce)
* Distributed system:
* High Performance Computing (HPC): large assemble of physical machines and a homogeneous operating system (e.g., your institutions' HPC, XSEDE's HPC)
* Cloud Computing: virtual machines, distributed platforms and/or applications offered as a service (e.g., Amazon Web Services, Microsoft Azure, Google Cloud Computing)
* Virtual machine (VM): software computer that like a physical computer can run an operating system and applications
* Operating system (OS): the basic software layer that allows execution and management of applications
* Physical machine: the hardware (processors, memory, disk and network)
### HPC vs. Cloud:
| HPC | Cloud |
|:----|:------|
| User account on the system | root account on the system |
| Limited control of the system | Full control of the system |
| Central shared file system | Local file system |
| Jobs submitted into a queue | Jobs executed on each resource |
| Account-based isolation | OS-based isolation |
| Batch-oriented execution of applications | support for batch or interactive applications |
| Request for resource and time allocation | Pay-per-use |
| etc. | etc.|
### Resources:
* Cloud computing offerings:
* Amazon EC2: http://aws.amazon.com/ec2/
* Microsoft Azure: https://azure.microsoft.com/en-us/
* Google Cloud Platform: https://cloud.google.com/
* iPlant's Atmosphere: http://www.iplantcollaborative.org/ci/atmosphere
* iPlant's help page: https://pods.iplantcollaborative.org/wiki/display/atmman/Atmosphere+Manual+Table+of+Contents
* HPC offerings:
* XSEDE: https://www.xsede.org/high-performance-computing