Spreading sector errors to single byte errors on multiple sectors?

[oe1rsa]

I tried to follow the readme file closely, which is a very good introduction to long term storage!
Thank you for that.

I would like to ask if the ecc file created is also suited for the expected case that an entire sector fails, i.e. an error burst occurs. Wouldn't it be better here to spread a single sector error into multiple single byte errors across multiple sectors?

This would mean that I could no longer open a zip file on the storage medium directly, for example, but would it not increase tolerance to errors?

[lrq3000]

Thank you for your feedback :-)

About failing sectors, yes this is the primarily targeted use case, failing sectors on a hard drive, aka burst error type as you mention.

As you note, yes interleaving would be better, especially given that nowadays most sectors are at least 512 bytes long, which is longer than a RS GF(2^8) can protect at once. In practice, it's unlikely that the message protected by pyFileFixity is exactly stored on one sector, it's likely there will be a misalignment, so the failed sector will likely span over 2 pFF messages and hence we can assume that we can cover 512 bytes, a whole sector, with 2 messages and hence 2 ECCs in practice. And also remember that ECC is implemented in most hard drives nowadays (not the case for RAM, but for hard drives yes), so pFF comes in addition to the hard drive's native error correction capabilities.

But interleaving would solve this issue, as does for example Multipar. Now the issue is: how do you define the interleaving? Answer: with additional metadata. And this metadata is then not a single point of failure. Which is the case with the Multipar standard.

When I was actively developing this software, this was the end stage I reached, I unfortunately had to stop my ideation process at this point due to lack of time and (personal) funding to continue further. If you have an idea to implement interleaving in a robust way (ie, the metadata can also be protected), then I can add this to the TODO. I plan on improving this software at some point in the future, but I cannot guarantee I ever will unfortunately.

An alternative if you really want interleaved burst error type correction, I would recommend trying DVDisaster, which only works on optical mediums IIRC but is really awesome.