Single file "WFDB file"

[cx1111]

This topic has probably been raised thousands of times. Why are WFDB header and signal files separate? Why isn't there a single file that contains the record metadata and the signal samples?

Reasons for having separate files:

1. The header file is in UTF-8 and its contents are human-readable (if you know how the fields are arranged 😄 ). Having a separate file makes it easy to open and read with various text editors and browsers.
   • In a combined WFDB file, this could be partially addressed by making WFDB reading software more easy to use/install, and more ubiquitous. Sites (such as PhysioNet) could also offer a WFDB file previewer for the metadata.
2. Stream writing records with signals that are stored in different formats. eg. If a record has a 16 bit channel that is stored in a format 16 dat file, and a 24 bit channel stored in a 24 bit dat file, the only way to stream write samples from both channels, is to have separate dat files (I think?). There is no dat format that allows interleaving samples of different resolutions.
   • Note that this may be very important for records that use FLAC files for signal storage, as a FLAC file only supports 8 channels. A WFDB record can support many more, and may require using multiple FLAC files.
   • A 'workaround' is to store all channels in a single highest-resolution file. But that's less efficient for storage.
3. A niche reason geared specifically towards PhysioNet: The MIMIC waveform database is huge. The matched subset is also huge, and its signal files are actually just sym-links of the signal files from the full dataset. Helps save on storage costs and ensures consistency (though the latter is a weak reason).

However, there are clear benefits to storing all of the data in a single file:

1. Clarity and ease of use. When you think of a file, such as a jpeg, pdf, EDF, text file, or a DICOM, it's just one file. There's no ambiguity regarding its makeup. The less conventional multi-file setup hinders understanding and adoption.
2. Ease of managing the data. eg. If you write a program to move a WFDB record from A to B, it is not as straightforward as moving a single file. There's the additional complexity of figuring out all the files associated with it, and moving all of them. And if the operation fails mid-way through, with only some of the files being successfully transported, dealing with the case is also additionally complex. This is a major drawback IMO.

[cx1111]

One way to bring about a new single-file WFDB file format, in the simplest way possible, would be to have a wrapper file, such as a zero-compression zip file with a .wfdb extension, that holds all of the individual files. This would work for both single and multi-segment records. This is similar to JAR files, and others. In a similar manner, I believe that some powerpoint-like files are zipped XMLs.

New datasets could be created using the current WFDB file formats, and then zipping all of the record files into a single file after it is certain that the record will not need to be altered/appended. This offers some benefits of both the single and multi-file approaches.

I believe that this would be a very valuable feature.

[tompollard]

I love the idea of the wrapper solution. Thanks for looking into this! It sounds like it could be a small-ish technical change with big benefits in terms of user-friendliness.

[alistairewj]

As long as the decompression is streamable, I agree! It would also allow for including annotations with the WFDB record, e.g. no one analyzes MIT-BIH without the annotations, so you end up managing 3 files for each record.

Might be worth taking a look at CCDEF (published paper here), specifically the format page. Gives an idea of how other researchers have chosen to structure their data, and seems to give more justification to having a single file. It does raise two questions of (1) whether we allow for including annotations in the file (I would say yes) and (2) whether we allow for including numerics in the file (probably needs thorough discussion).

[crazydogen]

As a waveform database, I will recommend parquet for offline high compression rate and arrow for online/offline fast loading and analytics. Both formats are supported by big data tools. PyArrow comes with an abstract filesystem interface, as well as concrete implementations for various storage types. Still, time series database like Timescale and TDengine plus metadata as extra columns is a good choice for large-scale apps.