Why does .raw exist?

[sjfleming]

With the adata.layers construct being so useful, and potentially being unified with the adata.X construct (#706), I am wondering why adata.raw exists at all? Perhaps there is a great reason I just don't understand.

But it seems to me that raw data can (and should) be stored as a layer, maybe adata.layers['raw'] (up to the user). But all the annotations in adata.obs and adata.var still apply to .raw, and so why is there the need for adata.raw at all? I always wondered if it was kind of a legacy thing that would be deprecated after layers came around.

[LuckyMD]

Hey! adata.raw is matched on .obs, but not .var so you can store unfiltered data with all genes in there. It is somewhat legacy code, and there have been discussions about deprecating this... but I still quite like it ;).

[ivirshup]

raw largely exists due to dense storage of X being prohibitive back when people centered, imputed, or used other densifying transforms their single cell expression data. This way you could just densify a smaller set of features, but keep the full expression around in raw.

Now we don't need to do that as much, and it was always a bit poorly supported.

But it seems to me that raw data can (and should) be stored as a layer, maybe adata.layers['raw']

Maybe something based on the "processing" fields from the FOM schema?

[sjfleming]

Okay thanks very much! I guess having some way to store things with different numbers of features could be useful… and if people use it, then yeah. (I usually end up confusing myself when I use .raw, but I guess for someone like me, layers are perfect, and I can use that.) 😄

[sjfleming]

Yes thanks for that link @ivirshup , that’s the kind of thing I like doing with layers.