index_pointer dataset in example compartment reports contains an extra offset at the end

[hernando]

@kaeldai I remember we discussed this last July in person, but since then we haven't taken any decision about what we should do. The question is: Should we update the spec to reflect that there's an extra offset at the end which matches the number of values in a frame, or should we remove it?
I would like to sort this out soon because Brion can't open this reports at the moment and we are not keen on the idea of introducing a hack for an undocumented feature in the examples.

[kaeldai]

I think for now it's best to just make it explicit that the extra offset is required. It will help to validate the hdf5 files in the case the data is being written in blocks but the programmer forgets to cut off the end values. Without the extra offset the software will end up reading in junk values as valid ones. I will explicit state in the doc that the extra offset is mandatory.

Juan and Judit are looking into using padding in which case this issue will not be valid anymore.

[jplanasc]

@kaeldai , @antonarkhipov , @jdcourcol , @mgeplf , @jorblancoa , @alkino :
This issue has been open for a long time. After internal discussion with some BBP developers, we would like to suggest that we add an extra element in the index_pointer dataset.

In any case, this is exactly what Kael implemented and it's in the current implementation of pysonata (number of elements in the dataset is total number of GIDS + 1):
https://github.com/AllenInstitute/sonata/blob/master/src/pysonata/sonata/reports/compartment/compartment_writer.py#L180

With this, the only thing that's not strictly complete is the specification (which doesn't mention this detail of the extra element). @alkino will do a PR to correct it and you can review it. So, we could also close this issue (the padding option was discarded).

Thanks!