C ++ HSPY File Writer

[CSSFrancis]

Direct Electron has a (partial) c++ implementation for writing to the .hspy file format as well as for the DE5 file format (a varient of the .emd file format). Something that I would like is to improve the ability for companies to write directly into the .hspy file format as well as to the .emd file format (and eventually into the .zspy file format).

As far as performance goes, I don't know if there is much difference from C++ --> python but I think in many instances a c++ implimentation should be better able to integrate with different aquisition softwares.

I'd be curious to hear peoples thoughts on the subject?

1. Is there interest from other companies to write directly into a .hspy/ .zspy / emd file?
2. Are 2 seperate implementations worse than 1 in this case as there could potentially be some conflicts?
3. What liscense should this be released under? We don't necessarily want to put restrictions on this but it would be nice to know if someone else is using this code comercially.
4. @sk1p Should we be using Rust or something other than c++ here :).

[jlaehne]

@noemiebonnet @pieleric @Attolight-NTappy a discussion possibly relevant for your companies as you at least have reader implementations for RosettaSciIO/HyperSpy

[sk1p]

Thanks for tagging me!

Direct Electron has a (partial) c++ implementation for writing to the .hspy file format as well as for the DE5 file format (a varient of the .emd file format). Something that I would like is to improve the ability for companies to write directly into the .hspy file format as well as to the .emd file format (and eventually into the .zspy file format).

As far as performance goes, I don't know if there is much difference from C++ --> python but I think in many instances a c++ implimentation should be better able to integrate with different aquisition softwares.

Indeed, also if you need something resembling (firm) real-time assurances, this is a lot easier with C++/Rust than Python.

1. Is there interest from other companies to write directly into a `.hspy`/ `.zspy` / `emd` file?

2. Are 2 seperate implementations worse than 1 in this case as there could potentially be some conflicts?

I'm working on adding zarr/zspy support to LiberTEM-live, meaning any detector detector we support will be able to directly produce zspy soon. So that would be implementation number three.

I think it should be easy enough to support the general structure of zspy files, from my perspective. It could be helpful to have a small acceptance test that checks the structure, basically an executable version of the textual description, which might catch subtleties that are not expressed in the language here:

rosettasciio/rsciio/zspy/_api.py

Lines 42 to 67 in 3c7b9f6

	# -----------------------
	# File format description
	# -----------------------
	# The root must contain a group called Experiments
	# The experiments group can contain any number of subgroups
	# Each subgroup is an experiment or signal
	# Each subgroup must contain at least one dataset called data
	# The data is an array of arbitrary dimension
	# In addition a number equal to the number of dimensions of the data
	# dataset + 1 of empty groups called coordinates followed by a number
	# must exists with the following attributes:
	# 'name'
	# 'offset'
	# 'scale'
	# 'units'
	# 'size'
	# 'index_in_array'
	# The experiment group contains a number of attributes that will be
	# directly assigned as class attributes of the Signal instance. In
	# addition the experiment groups may contain 'original_metadata' and
	# 'metadata'subgroup that will be
	# assigned to the same name attributes of the Signal instance as a
	# Dictionary Browsers
	# The Experiments group can contain attributes that may be common to all
	# the experiments and that will be accessible as attributes of the
	# Experiments instance

I think the more challenging part would be writing the metadata tree and keeping that consistent between implementations. See also this previous discussion. I think most helpful would be to have a formal source of truth, like a schema, that documents the metadata attributes in a clear way.

it would be nice to know if someone else is using this code comercially.

I think the options are BSD/MIT where you would get a notice in the docs or somewhere in a menu of the application. (A)GPL might be a hard sell to most commercial users, from my experience.

4. @sk1p Should we be using Rust or something other than c++ here :).

If there is interest, I would be happy to collaborate on a rust library, with a C or C++ API (and Python?) on top for integration into other software. The argument for rust would be that it's IMHO easier to write correct rust code compared to C++, and with PyO3 there is a really good option for building Python bindings.

[Attolight-NTappy]

Thanks for tagging me in this discussion!

We really like the *.hspy file format at attolight and recommend using it to our customers for sharing data and saving them once processed using hyperspy.

In the future, there could be use cases where writing data to that format directly would be interesting. However, I am not certain that licensing conditions (also of hdf5 data standard) allow to do this easily.

[sk1p]

also of hdf5 data standard

AFAIK the libraries (C, Python) are BSD licensed, so compliance should be easy even for closed-source software