-
Notifications
You must be signed in to change notification settings - Fork 3
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Added installation instructions. - Fixed typo. - Added Go report badge.
- Loading branch information
1 parent
96974c1
commit 78719b2
Showing
1 changed file
with
56 additions
and
9 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,20 +1,67 @@ | ||
bitrot | ||
====== | ||
[![Go Report Card](https://goreportcard.com/badge/github.com/marcopaganini/bitrot)](https://goreportcard.com/report/github.com/marcopaganini/bitrot) | ||
|
||
Bitrot Scrubber - Scrubs your disks (or array) looking for bitrot (silent disk data corruption.) | ||
# bitrot | ||
|
||
The size of hard drives has grown steadily over the years, but their bit error rate has remained relatively unchanged. As an example, the Seagate "Barracuda" line has a "Non Recoverable Bit Error per bit Read" rate of 1 bit per 1e14 bits read ([Barracuda datasheet](http://www.seagate.com/staticfiles/docs/pdf/datasheet/disc/barracuda-ds1737-1-1111us.pdf)). This means that 1 bit may be incorrectly reported every 1e14 bits read from the drive, or 11 TB (bytes) of data. This may seem a lot, but with 4-6+ TB drives becoming commonplace, the problem becomes very obvious. | ||
# Description | ||
|
||
Some forms of read errors are detectable by the drive firmware. These errors are usually very visible to the system operator and generate ominous error messages in the system log. If a disk array is used, such conditions will normally cause the offending drive to be marked as defective and removed from the array. Our main concern are the "invisible" bit read errors, popularly known as bitrot, which are not detected by the drive firmware and could cause silent data corruption. | ||
Bitrot Scrubber - Scrubs your disks (or array) looking for "bit rot" (silent | ||
disk data corruption.) | ||
|
||
The "proper" solution for bitrot protection is to use an advanced file system such as ZFS, which was specifically created to address data integrity issues. Unfortunately, not everybody is willing to move to ZFS and deal with the resulting technical complications or licensing structure. | ||
The size of hard drives has grown steadily over the years, but their bit error | ||
rate has remained relatively unchanged. As an example, the Seagate "Barracuda" | ||
line has a "Non Recoverable Bit Error per bit Read" rate of 1 bit per 1e14 bits | ||
read ([Barracuda datasheet](http://www.seagate.com/staticfiles/docs/pdf/datasheet/disc/barracuda-ds1737-1-1111us.pdf)). | ||
This means that 1 bit may be incorrectly reported every 1e14 bits read from the | ||
drive, or 11 TB (bytes) of data. This may seem a lot, but with 4-6+ TB drives | ||
becoming commonplace, the problem becomes very obvious. | ||
|
||
This program is a simple alternative to detect silent data corruption on disk drives. The idea is to "scrub" your collection of files periodically, so that an eventual corruption might be detected and the file restored from backups. This program will not protect your data from write errors or cache/buffer corruptions in memory (use ECC RAM!). Also, it was *not* meant to be used as a security audit tool (it is, in fact, useless in that capacity as it purposedly ignores files with metadata changes.) | ||
Another problem is the ever growing sizes of solid state media. Many flash | ||
cards fail silently and don't report any error conditions the host operating | ||
system. By the time the problem is detected, usually even backups already have | ||
a copy of corrupt data. | ||
|
||
The principle of operation is simple: Run "bitrot <directory>" periodically and the program will read all regular files under that directory recursively, calculating their MD5 hashes. The program then compares the MD5 hash of each file read from disk with a saved version from a previous run. If a file exists with the same filesystem metadata (modification date, size) and the MD5 hash doesn't match, the scrubber assumes data corruption happened and reports the file. The program assumes that modified files (metadata changes) contain valid modifications and it won't try to compare the MD5 hash on those. Instead, the current MD5 file is saved for future runs. | ||
Some forms of read errors are detectable by the drive firmware. These errors | ||
are usually very visible to the system operator and generate ominous error | ||
messages in the system log. If a disk array is used, such conditions will | ||
normally cause the offending drive to be marked as defective and removed from | ||
the array. Our main concern are the "invisible" bit read errors, popularly | ||
known as bitrot, which are not detected by the drive firmware and could cause | ||
silent data corruption. | ||
|
||
The "proper" solution for bitrot protection is to use an advanced file system | ||
such as ZFS, which was specifically created to address data integrity issues. | ||
Unfortunately, not everybody is willing to move to ZFS and deal with the | ||
resulting technical complications or licensing structure. | ||
|
||
This program is a simple alternative to detect silent data corruption on disk | ||
drives. The idea is to "scrub" your collection of files periodically, so that | ||
an eventual corruption might be detected and the file restored from backups. | ||
This program will not protect your data from write errors or cache/buffer | ||
corruptions in memory (use ECC RAM!). Also, it was *not* meant to be used as a | ||
security audit tool (it is, in fact, useless in that capacity as it purposely | ||
ignores files with metadata changes.) | ||
|
||
# Installation | ||
|
||
The easiest way to install bitrot is to download it from the releases page in | ||
github: https://github.com/marcopaganini/bitrot/releases/tag/v1.0.0. Just | ||
download the desired version and give it a go. | ||
|
||
-- Paga | ||
If you prefer to install from source: | ||
|
||
* Clone the repo locally: `git clone https://github.com/marcopaganini/bitrot` | ||
* Change to the repo directory: `cd bitrot` | ||
* Build the version: `make` | ||
* Install in your local binaries directory: `sudo make install` | ||
|
||
# Usage | ||
|
||
Run `bitrot <directory>` periodically and bitrot will read all regular files | ||
under that directory, recursively, calculating their MD5 hashes. The program | ||
then compares the MD5 hash of each file read from disk with a saved version | ||
from a previous run. If a file exists with the same filesystem metadata | ||
(modification date, size) and the MD5 hash doesn't match, the scrubber assumes | ||
data corruption happened and reports the file. The program assumes that | ||
modified files (metadata changes) contain valid modifications and it won't try | ||
to compare the MD5 hash on those. Instead, the current MD5 file is saved for | ||
future runs. |