Allow including null fields in filter query with addition of a new query param flag

[corneliusroemer]

In Loculus we'll soon have dates represented as ranges instead of points. Some sequences might lack dates entirely. In that case we will likely not provide a lower range, as any lower range could be too high (one could set it to something arbitrarily really small but that's somehow unpleasant). So null actually means -infinity in that case.

There's an interesting use case that seems currently impossible to implement with LAPIS as is:

One might want to filter for all sequences that could have been collected before 1980. This is simple to express: the lower range should be smaller than 1980. Null in the lower range field should be interpreted as -infinity. And -infinity is of course smaller than 1980. So any nulls should be included in the result.

However, if I understand LAPIS correctly, one can't actually perform this search right now. Nulls will always be missing on all range queries.

Would it be possible to work around this somehow with a new feature? Something to add nulls back in? This might be the most generic way to implement something that would allow the above use case to work.

You'd simply add a new special boolean fieldNameIncludeNulls that is interpreted as an OR together with other filters on the same field. So if fieldName>5 is set and fieldNameIncludeNulls=true then all entries matching one or the other would be included.

Maybe I'm missing something and this is already possible, otherwise this could be a neat addition to broaden the expressive pwoer of LAPIS.