[Proposal] Multimodality

[mmcdermott]

This is a tracking issue for discussions around supporting multimodality, in general. A few starting ideas.

1. It is very easy to in principle support single-point-in-time modalities such as images or notes; these are just different value columns in the MEDS schema. In fact, a few different projects already use text_value columns for notes, for example. Other types of columns, e.g., images, would need to be typed/standardized, or we'd need to have a library of supported types related to on-disk storage, e.g.,
2. It is more challenging to support time-varying modalities, such as waveforms, videos, or audio recordings. While these can be so short that they could be functionally considered single-point-in-time, it nevertheless challenges our current paradigm.

[EthanSteinberg]

I think the best solution for images is to introduce a new path in our MEDS extracts and then have strings that refer to those paths in the event/parquet files.

The worry is that the images are so large that including them in the parquets themselves will cause problems.

I agree that time-varying modalities require a lot more though.

[mmcdermott]

I agree that images should be stored via file paths

[mmcdermott]

I propose we keep this issue open, but add sub-issues for individual modalities as they gain sufficient traction and need in the community. E.g., an issue for text_value for text features, for paths to images, paths to short-duration waveform data, etc.