Use variable-length strings for storing alleles in Zarr

[tomwhite]

Currently we use fixed-length strings for storing alleles, but this is inefficient since the length is the size of the longest allele in the whole dataset.

For example, in some 1000 genomes data (chr22) I noticed that the longest allele is 414 base pairs, which means the data type is "S414" - unnecessarily large for the vast majority of variants. This meant that the variant_allele data took up 15 MB (compressed), rather than something like one tenth of that if a variable-length encoding were used (like scikit-allel does).

It would be worth investigating how we could use Zarr's variable-length strings (the number of alt alleles would remain fixed though, in contrast to #634). In particular, there may be some work to get this representation to work nicely with xarray.

[timothymillar]

@tomwhite the freebayes variant caller is a good example for producing a range of allele lengths. The VCFs avialable here (see releases) were called with freebayes and have allele lengths of up to ~ 60bp. Note that these were called with --haplotype-length -1 in an attempt to minimize variant length.

[tomwhite]

It would be worth investigating how we could use Zarr's variable-length strings

In the case of writing Zarr sequentially from VCF, we already do this (it can be enforced by passing target_part_size=None to vcf_to_zarr). The parallel case concatenates a collection of Zarr files, and as a byproduct of that the variable-length strings are converted to fixed-length strings. We should change that.

[tomwhite]

I'm not sure how to fix this, so I opened pydata/xarray#5769 with a minimal example.