HOWTO: EPICS area detector and HDF5 master:external file pattern

[prjemian]

A thread on the Nikea Slack website started with a request how Bluesky should writefiles that comply with the NeXus NXsas application definition. Quickly, the discussion pointed towards how to write NeXus-compliant files in general, and more specifically using EPICS and area detector. (NXsas files are raw data using a 2-D detector. Other geometries are proposed but that's the way it stands now.)

One proposition was to use the apstools.callbacks.NXWriter() class to write a master NeXus/HDF5 file that makes external file links to the NeXus/HDF5 file create by the EPICS area detector IOC. This might better fit the original question since the questioner uses PyTango.

On Slack, I wrote:

Using the master:external style of HDF5 file writing is actually easier than waiting for the run to finish, then reading the image data and writing all to one HDF5 file. Why, you might ask? The AD HDF5 plugin is not done until it is unstaged. The bluesky run must be complete before any and all are devices are unstaged. The classic standoff of you first. To get this done, you might launch a separate thread that does this work after the run is closed. Which has some additional consequences I ran into during unit testing.

Just keep in mind that it is easiest if the master and external file are in the same directory or that you identify the external file's directory by a path that is relative to the master file rather than absolute. With an absolute directory path, the pair of files become locked to the file system, assuming that no else will have exactly the same absolute directory paths.

[prjemian]

Keep in mind:

Since the original question concerned PyTango (and not EPICS), a document showing how to write a master NeXus/HDF5 file from Bluesky that makes external file links to image data in another HDF5 file will be more useful.

Where the software that writes the image data is actually irrelevant to the HowTo, other than it provides the directory, file name, and HDF5 address to the image data.

[prjemian]

Work on #240 before this one.

[prjemian]

f["/entry/data"] = h5py.ExternalLink("image.hdf5", "/entry/data")

[prjemian]

Consider using one extension for the master files and a different one for the external data files. It helps to keep things recognizable and understandable.