-
Notifications
You must be signed in to change notification settings - Fork 24
/
minio-erasure-code-quickstart-guide
527 lines (471 loc) · 27.7 KB
/
minio-erasure-code-quickstart-guide
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="author" content="MinIO">
<link rel="apple-touch-icon" sizes="180x180" href="https://min.io/resources/favs/apple-icon-180x180.png">
<link rel="icon" type="image/png" sizes="32x32" href="https://min.io/resources/favs/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="96x96" href="https://min.io/resources/favs/favicon-96x96.png">
<link rel="icon" type="image/png" sizes="16x16" href="https://min.io/resources/favs/favicon-16x16.png">
<title>MinIO | Learn more about MinIO's Erasure Code Implementation</title>
<meta name="description" content="The MinIO Erasure Code Quickstart Guide offers examples, code and insight on this powerful data protection approach">
<meta name="keywords" content="Erasure coding, inline, forward error correction, reed-solomon code, silent data corruption, RAID6, BitRot protection, healing, highway hash">
<link href="https://cdn.jsdelivr.net/npm/docsearch.js@2/dist/cdn/docsearch.min.css" rel="stylesheet">
<link href="/docs/css/gluegun.min.css" rel="stylesheet">
<link href="/docs/vendors/highlight.js/default.min.css" rel="stylesheet">
<link href="https://fonts.googleapis.com/css?family=Open+Sans:400,600" rel="stylesheet">
<style type="text/css">
.algolia-autocomplete {
width: 100%;
left: 0; }
.algolia-autocomplete .ds-dropdown-menu {
min-width: 500px;
max-width: 800px; }
@media (max-width: 991px) {
.algolia-autocomplete .ds-dropdown-menu {
min-width: 200px; } }
.algolia-autocomplete .ds-dropdown-menu:before {
display: none !important; }
</style>
</head>
<body>
<aside class="sidebar">
<div class="sidebar__inner">
<a href="" class="sidebar__close">
<svg class="sidebar__close__icon">
<polygon points="15.8333324 5.34166622 14.6583325 4.16666642 9.99999942 8.82499958 5.34166622 4.16666642 4.16666642 5.34166622 8.82499958 9.99999942 4.16666642 14.6583325 5.34166622 15.8333324 9.99999942 11.1749992 14.6583325 15.8333324 15.8333324 14.6583325 11.1749992 9.99999942"></polygon>
</svg>
</a>
<a href="https://min.io" class="logo">
<img src="https://min.io/resources/img/logo/MINIO_wordmark.png" alt="">
</a>
<ul class="doc-link">
<li class='doc-link__category toggled'>
<a href="" class="doc-link__header">
MinIO Server </a>
<ul class="doc-link__items" style='display: block'>
<li>
<a href="/docs/minio-quickstart-guide.html">
MinIO Quickstart Guide </a>
</li>
<li>
<a href="/docs/minio-docker-quickstart-guide.html">
MinIO Docker Quickstart Guide </a>
</li>
<li>
<a href="/docs/minio-erasure-code-quickstart-guide.html" class="doc-link__active">
MinIO Erasure Code Quickstart Guide </a>
</li>
<li>
<a href="/docs/distributed-minio-quickstart-guide.html">
Distributed MinIO Quickstart Guide </a>
</li>
<li>
<a href="/docs/disaggregated-spark-and-hadoop-hive-with-minio.html">
Disaggregated Spark and Hadoop Hive with MinIO </a>
</li>
<li>
<a href="/docs/how-to-secure-access-to-minio-server-with-tls.html">
How to secure access to MinIO server with TLS </a>
</li>
<li>
<a href="/docs/minio-bucket-notification-guide.html">
MinIO Bucket Notification Guide </a>
</li>
<li>
<a href="/docs/minio-server-limits-per-tenant.html">
MinIO Server Limits Per Tenant </a>
</li>
<li>
<a href="/docs/minio-server-configuration-guide.html">
MinIO Server Configuration Guide </a>
</li>
<li>
<a href="/docs/multi-tenant-minio-deployment-guide.html">
Multi-tenant MinIO Deployment Guide </a>
</li>
<li>
<a href="/docs/minio-gateway-for-azure.html">
MinIO Gateway for Azure </a>
</li>
<li>
<a href="/docs/minio-gateway-for-nas.html">
MinIO Gateway for NAS </a>
</li>
<li>
<a href="/docs/minio-gateway-for-s3.html">
MinIO Gateway for S3 </a>
</li>
<li>
<a href="/docs/minio-gateway-for-hdfs.html">
MinIO Gateway for HDFS </a>
</li>
<li>
<a href="/docs/minio-disk-cache-guide.html">
MinIO Disk Cache Guide </a>
</li>
<li>
<a href="/docs/minio-monitoring-guide.html">
MinIO Monitoring Guide </a>
</li>
<li>
<a href="/docs/how-to-monitor-minio-using-prometheus.html">
How to monitor MinIO using Prometheus </a>
</li>
<li>
<a href="/docs/minio-federation-quickstart-guide.html">
MinIO Federation Quickstart Guide </a>
</li>
<li>
<a href="/docs/minio-kms-quickstart-guide.html">
MinIO KMS Quickstart Guide </a>
</li>
<li>
<a href="/docs/minio-select-api-quickstart-guide.html">
MinIO Select API Quickstart Guide </a>
</li>
<li>
<a href="/docs/minio-compression-guide.html">
MinIO Compression Guide </a>
</li>
<li>
<a href="/docs/minio-multi-user-quickstart-guide.html">
MinIO Multi-User Quickstart Guide </a>
</li>
<li>
<a href="/docs/minio-sts-quickstart-guide.html">
MinIO STS Quickstart Guide </a>
</li>
</ul>
</li>
<li class='doc-link__category '>
<a href="" class="doc-link__header">
MinIO Deployment </a>
<ul class="doc-link__items" style=''>
<li>
<a href="/docs/minio-deployment-quickstart-guide.html">
MinIO Deployment Quickstart Guide </a>
</li>
<li>
<a href="/docs/deploy-minio-on-docker-swarm.html">
Deploy MinIO on Docker Swarm </a>
</li>
<li>
<a href="/docs/deploy-minio-on-kubernetes.html">
Deploy MinIO on Kubernetes </a>
</li>
<li>
<a href="/docs/deploy-minio-on-docker-compose.html">
Deploy MinIO on Docker Compose </a>
</li>
</ul>
</li>
<li class='doc-link__category '>
<a href="" class="doc-link__header">
MinIO Client </a>
<ul class="doc-link__items" style=''>
<li>
<a href="/docs/minio-client-quickstart-guide.html">
MinIO Client Quickstart Guide </a>
</li>
<li>
<a href="/docs/minio-client-complete-guide.html">
MinIO Client Complete Guide </a>
</li>
<li>
<a href="/docs/minio-admin-complete-guide.html">
MinIO Admin Complete Guide </a>
</li>
</ul>
</li>
<li class='doc-link__category '>
<a href="" class="doc-link__header">
MinIO SDKs </a>
<ul class="doc-link__items" style=''>
<li>
<a href="/docs/javascript-client-quickstart-guide.html">
JavaScript Client Quickstart Guide </a>
</li>
<li>
<a href="/docs/javascript-client-api-reference.html">
JavaScript Client API Reference </a>
</li>
<li>
<a href="/docs/java-client-quickstart-guide.html">
Java Client Quickstart Guide </a>
</li>
<li>
<a href="/docs/java-client-api-reference.html">
Java Client API Reference </a>
</li>
<li>
<a href="/docs/python-client-quickstart-guide.html">
Python Client Quickstart Guide </a>
</li>
<li>
<a href="/docs/python-client-api-reference.html">
Python Client API Reference </a>
</li>
<li>
<a href="/docs/golang-client-quickstart-guide.html">
Golang Client Quickstart Guide </a>
</li>
<li>
<a href="/docs/golang-client-api-reference.html">
Golang Client API Reference </a>
</li>
<li>
<a href="/docs/dotnet-client-quickstart-guide.html">
.NET Client Quickstart Guide </a>
</li>
<li>
<a href="/docs/dotnet-client-api-reference.html">
.NET Client API Reference </a>
</li>
<li>
<a href="/docs/haskell-client-quickstart-guide.html">
Haskell Client Quickstart Guide </a>
</li>
<li>
<a href="/docs/haskell-client-api-reference.html">
Haskell Client API Reference </a>
</li>
</ul>
</li>
<li class='doc-link__category '>
<a href="" class="doc-link__header">
Cookbook </a>
<ul class="doc-link__items" style=''>
<li>
<a href="/docs/s3cmd-with-minio.html">
S3cmd with MinIO </a>
</li>
<li>
<a href="/docs/aws-cli-with-minio.html">
AWS CLI with MinIO </a>
</li>
<li>
<a href="/docs/restic-with-minio.html">
restic with MinIO </a>
</li>
<li>
<a href="/docs/store-mysql-backups-in-minio.html">
Store MySQL Backups in MinIO </a>
</li>
<li>
<a href="/docs/store-mongodb-backups-in-minio.html">
Store MongoDB Backups in MinIO </a>
</li>
<li>
<a href="/docs/store-postgresql-backups-in-minio.html">
Store PostgreSQL Backups in MinIO </a>
</li>
<li>
<a href="/docs/setup-caddy-proxy-with-minio.html">
Setup Caddy proxy with MinIO </a>
</li>
<li>
<a href="/docs/setup-nginx-proxy-with-minio.html">
Setup Nginx proxy with MinIO </a>
</li>
<li>
<a href="/docs/aggregate-apache-logs-with-fluentd-and-minio.html">
Store Apache Logs into MinIO </a>
</li>
<li>
<a href="/docs/rclone-with-minio-server.html">
Rclone with MinIO Server </a>
</li>
<li>
<a href="/docs/setup-apache-http-proxy-with-minio-server.html">
Setup Apache HTTP proxy with MinIO Server </a>
</li>
<li>
<a href="/docs/upload-files-from-browser-using-pre-signed-urls.html">
Upload files from browser using pre-signed URLs </a>
</li>
<li>
<a href="/docs/how-to-run-minio-in-freenas.html">
How to run MinIO in FreeNAS? </a>
</li>
<li>
<a href="/docs/how-to-use-aws-sdk-for-php-with-minio-server.html">
How to use AWS SDK for PHP with MinIO Server </a>
</li>
<li>
<a href="/docs/how-to-use-aws-sdk-for-ruby-with-minio-server.html">
How to use AWS SDK for Ruby with MinIO Server </a>
</li>
<li>
<a href="/docs/how-to-use-aws-sdk-for-python-with-minio-server.html">
How to use AWS SDK for Python with MinIO Server </a>
</li>
<li>
<a href="/docs/how-to-use-aws-sdk-for-javascript-with-minio-server.html">
How to use AWS SDK for JavaScript with MinIO Server </a>
</li>
<li>
<a href="/docs/how-to-run-multiple-minio-servers-with-traef-k.html">
How to run multiple MinIO servers with Træfɪk </a>
</li>
<li>
<a href="/docs/how-to-use-aws-sdk-for-go-with-minio-server.html">
How to use AWS SDK for Go with MinIO Server </a>
</li>
<li>
<a href="/docs/how-to-use-aws-sdk-for-java-with-minio-server.html">
How to use AWS SDK for Java with MinIO Server </a>
</li>
<li>
<a href="/docs/how-to-use-paperclip-with-minio-server.html">
How to use Paperclip with MinIO Server </a>
</li>
<li>
<a href="/docs/how-to-use-aws-sdk-for-net-with-minio-server.html">
How to use AWS SDK for .NET with MinIO Server </a>
</li>
<li>
<a href="/docs/how-to-use-minio-s-server-side-encryption-with-aws-cli.html">
How to use MinIO's server-side-encryption with aws-cli </a>
</li>
<li>
<a href="/docs/generate-let-s-encypt-certificate-using-concert-for-minio.html">
Generate Let's Encrypt certificate using Certbot for MinIO </a>
</li>
</ul>
</li>
</ul>
</div>
</aside>
<section class="content">
<header class="header">
<a href="" class="sidebar-toggle"></a>
<div class="search">
<input type="text" class="search__text" placeholder="Search documents...">
</div>
<div class="doc-actions">
<div class="dropdown doc-actions__edit">
<a href="" class="dropdown__toggle dropdown__toggle--caret doc-actions__item">
<img src="/docs/img/icons/edit.svg" alt="">
<span>Suggest an Edit</span>
</a>
<div class="dropdown__menu dropdown__menu--right">
<ol>
<li>Log into Github.</li>
<li>Edit files with your changes by clicking on 'Edit the file in your fork of this project' button in Github.</li>
<li>Commit changes via 'Create a new branch for this commit and start a pull request'.</li>
</ol>
<a href="https://github.com/minio/minio/blob/master/docs/erasure/README.md" class="button button--primary button--block">Edit in Github</a>
</div>
</div>
<div class="dropdown">
<a href="" class="dropdown__toggle dropdown__toggle--caret doc-actions__item">
<img src="/docs/img/icons/translate.svg" alt="">
<span>Translation</span>
</a>
<div class="dropdown__menu dropdown__menu--right">
<a class="dropdown__item" href="/cn/">Chinese</a>
</div>
</div>
</div>
</header>
<section class="doc-preview">
<h1>MinIO Erasure Code Quickstart Guide <a href="https://slack.min.io"><img src="https://slack.min.io/slack?type=svg" alt="Slack"></a></h1>
<p>MinIO protects data against hardware failures and silent data corruption using erasure code and checksums. With the highest level of redundancy, you may lose up to half (N/2) of the total drives and still be able to recover the data.</p>
<h2>What is Erasure Code?</h2>
<p>Erasure code is a mathematical algorithm to reconstruct missing or corrupted data. MinIO uses Reed-Solomon code to shard objects into variable data and parity blocks. For example, in a 12 drive setup, an object can be sharded to a variable number of data and parity blocks across all the drives - ranging from six data and six parity blocks to ten data and two parity blocks.</p>
<p>By default, MinIO shards the objects across N/2 data and N/2 parity drives. Though, you can use <a href="https://github.com/minio/minio/tree/master/docs/erasure/storage-class">storage classes</a> to use a custom configuration. We recommend N/2 data and parity blocks, as it ensures the best protection from drive failures.</p>
<p>In 12 drive example above, with MinIO server running in the default configuration, you can lose any of the six drives and still reconstruct the data reliably from the remaining drives.</p>
<h2>Why is Erasure Code useful?</h2>
<p>Erasure code protects data from multiple drives failure, unlike RAID or replication. For example, RAID6 can protect against two drive failure whereas in MinIO erasure code you can lose as many as half of drives and still the data remains safe. Further, MinIO's erasure code is at the object level and can heal one object at a time. For RAID, healing can be done only at the volume level which translates into high downtime. As MinIO encodes each object individually, it can heal objects incrementally. Storage servers once deployed should not require drive replacement or healing for the lifetime of the server. MinIO's erasure coded backend is designed for operational efficiency and takes full advantage of hardware acceleration whenever available.</p>
<p><img src="https://github.com/minio/minio/blob/master/docs/screenshots/erasure-code.jpg?raw=true" alt="Erasure"></p>
<h2>What is Bit Rot protection?</h2>
<p>Bit Rot, also known as data rot or silent data corruption is a data loss issue faced by disk drives today. Data on the drive may silently get corrupted without signaling an error has occurred, making bit rot more dangerous than a permanent hard drive failure.</p>
<p>MinIO's erasure coded backend uses high speed <a href="https://github.com/minio/highwayhash">HighwayHash</a> checksums to protect against Bit Rot.</p>
<h2>How are drives used for Erasure Code?</h2>
<p>MinIO divides the drives you provide into erasure-coding sets of <em>4 to 16</em> drives. Therefore, the number of drives you present must be a multiple of one of these numbers. Each object is written to a single erasure-coding set.</p>
<p>Minio uses the largest possible EC set size which divides into the number of drives given. For example, <em>18 drives</em> are configured as <em>2 sets of 9 drives</em>, and <em>24 drives</em> are configured as <em>2 sets of 12 drives</em>.</p>
<p>The drives should all be of approximately the same size.</p>
<h2>Get Started with MinIO in Erasure Code</h2>
<h3>1. Prerequisites</h3>
<p>Install MinIO - <a href="https://docs.min.io/docs/minio-quickstart-guide">MinIO Quickstart Guide</a></p>
<h3>2. Run MinIO Server with Erasure Code</h3>
<p>Example: Start MinIO server in a 12 drives setup, using MinIO binary.</p>
<pre lang="sh"><code>minio server /data{1...12}
</code></pre>
<p>Example: Start MinIO server in a 8 drives setup, using MinIO Docker image. </p>
<pre lang="sh"><code>docker run -p 9000:9000 --name minio \
-v /mnt/data1:/data1 \
-v /mnt/data2:/data2 \
-v /mnt/data3:/data3 \
-v /mnt/data4:/data4 \
-v /mnt/data5:/data5 \
-v /mnt/data6:/data6 \
-v /mnt/data7:/data7 \
-v /mnt/data8:/data8 \
minio/minio server /data{1...8}
</code></pre>
<h3>3. Test your setup</h3>
<p>You may unplug drives randomly and continue to perform I/O on the system.</p>
</section>
<footer class="footer">
<div class="footer__widget">
<a href="https://slack.min.io">Talk to the community</a>
</div>
</footer>
</section>
<script src="/docs/vendors/jquery/jquery-3.3.1.min.js"></script>
<script src="/docs/vendors/highlight.js/highlight.min.js"></script>
<script src="/docs/vendors/clipboard.js/clipboard.min.js"></script>
<!-- Algolia DocSearch -->
<script type="text/javascript" src="https://cdn.jsdelivr.net/npm/docsearch.js@2/dist/cdn/docsearch.min.js"></script>
<script type="text/javascript">
docsearch ({
apiKey: '0ba0d26da4d1483f96c03fe508304a64',
indexName: 'minio',
inputSelector: '.search__text',
algoliaOptions: { 'facetFilters': ["tags: en"] }
});
</script>
<!-- Google Analytics -->
<script>
(function (i, s, o, g, r, a, m) {
i['GoogleAnalyticsObject'] = r;
i[r] = i[r] || function () {
(i[r].q = i[r].q || []).push (arguments)
}, i[r].l = 1 * new Date ();
a = s.createElement (o),
m = s.getElementsByTagName (o)[0];
a.async = 1;
a.src = g;
m.parentNode.insertBefore (a, m)
}) (window, document, 'script', 'https://www.google-analytics.com/analytics.js', 'ga');
ga ('create', 'UA-56860620-1', 'auto');
ga ('send', 'pageview');
</script>
<!-- LeadFeeder -->
<script>
(function(){
window.ldfdr = window.ldfdr || {};
(function(d, s, ss, fs){
fs = d.getElementsByTagName(s)[0];
function ce(src){
var cs = d.createElement(s);
cs.src = src;
setTimeout(function(){fs.parentNode.insertBefore(cs,fs)}, 1);
}
ce(ss);
})(document, 'script', 'https://lftracker.leadfeeder.com/lftracker_v1_DzLR5a5kKJA7BoQ2.js');
})();
</script>
<!-- KickFire -->
<script>
var DID = 258964;
var MyPageName = window.location.href.replace("http://","");
</script>
<script src="https://stats.sa-as.com/live.js"></script>
<script src="/docs/js/gluegun.min.js"></script>
</body>
</html>