Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EM refactoring / EMv2 #136

Merged
merged 69 commits into from
Mar 1, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
764093c
Fixed missing references to EM/CGMS/ICME-related base classes in the …
atomprobe-tc Dec 13, 2023
8a3ae1b
Refactoring storage of images and spectra
atomprobe-tc Dec 13, 2023
6663bec
Minor edit
atomprobe-tc Dec 13, 2023
dba0e64
Refactoring apm has three aims: i) extent NXapm appdef to take advant…
atomprobe-tc Jan 3, 2024
261f91b
Testing the use of NXapm_paraprobe_tool_config/results base classes o…
atomprobe-tc Jan 3, 2024
7abeded
Applied to i)
atomprobe-tc Jan 3, 2024
b50f160
Working on ii) mirrored concept used for NXem with NXapm_msr and NXap…
atomprobe-tc Jan 4, 2024
cdcebea
Implemented design of event_data_apm, correlated signals with keyword…
atomprobe-tc Jan 5, 2024
7e4be9b
Editing NXapm_ranging remains and a final check of the NXapm_new, the…
Jan 5, 2024
373da83
Refactored NXapm_ranging, renamed NXapm_new, a through and complete c…
Jan 5, 2024
df219e8
Reviewing, fixing typos, a minor glitches
Jan 6, 2024
6b649bc
Further reviewing and corrections
Jan 6, 2024
701f277
NXapm reviewing and typos
Jan 7, 2024
34fc49b
Fixed issues detected during make local, manual builds successfully, …
atomprobe-tc Jan 8, 2024
5c7d197
Added changes relevant to use the refactored NXapm parser of pynxtool…
atomprobe-tc Jan 10, 2024
f402b33
Fixes and additions revealed from implementing parser functionalities…
atomprobe-tc Jan 17, 2024
c1aa2e3
Placed version as child of definition
atomprobe-tc Jan 17, 2024
23d34bc
Added version as a child of definition, spotted what could be a bug t…
atomprobe-tc Jan 17, 2024
0a2bd84
Fixes and modifications revealed during implementation of imgs, apex …
atomprobe-tc Jan 17, 2024
e37edaf
Added suggestions from @lukaspie on NXpeak
atomprobe-tc Jan 17, 2024
f816790
Fixes introduced for Velox EMD file parser
atomprobe-tc Jan 19, 2024
1072a9a
Fixed definitions and version
atomprobe-tc Jan 23, 2024
8041523
Fixed definitions and version
atomprobe-tc Jan 23, 2024
3b266bf
Removed unfortunated row vector style matrix isotope_vector and nucli…
atomprobe-tc Jan 23, 2024
c043182
Updated apm-specific documentation for new base classes and refactori…
atomprobe-tc Jan 23, 2024
9745686
Fixes and modifications specifically for paraprobe-toolbox tools: par…
atomprobe-tc Jan 25, 2024
4b84b08
Added appdef for selector_results, reorganized where to place common …
atomprobe-tc Jan 25, 2024
0d40ece
Reorganized transcoder, ranger, selector
atomprobe-tc Jan 26, 2024
a172c01
Refactored surfacer to use APMv2
atomprobe-tc Jan 26, 2024
8b6ee88
Refactored appdefs for distancer and tessellator to use APMv2
atomprobe-tc Jan 27, 2024
8ac38a5
Fixes added in the process of refactoring parmsetup-distancer and par…
Jan 27, 2024
7c1f072
Fixes to complete distancer and tessellator APMv2 refactoring
atomprobe-tc Jan 28, 2024
13efc84
Refactored appdefs for paraprobe-spatstat, specified symbol to use fo…
Jan 28, 2024
a04b6fb
Minor editing of spatstat
atomprobe-tc Feb 9, 2024
5291ffe
Refactored paraprobe-clusterer definitions to use APM v2
atomprobe-tc Feb 10, 2024
6122e74
Added missing profiling and program for results appdefs of tools wher…
atomprobe-tc Feb 10, 2024
23722e2
Renaming the last two missing appdefs for tools nanochem and intersec…
atomprobe-tc Feb 13, 2024
7892cf9
Refactored config files for nanochem and intersector to use APMv2
atomprobe-tc Feb 16, 2024
2df16a3
Updated apm-specific landing page
atomprobe-tc Feb 16, 2024
0bd04c7
Minor edits and fixes apm-structure
atomprobe-tc Feb 16, 2024
d5fa011
Fixing a copy-paste error
atomprobe-tc Feb 16, 2024
38bb082
Fixed inconsistence
atomprobe-tc Feb 16, 2024
f8cb8db
Refactored NXapm_paraprobe_nanochem_results and delocalization
atomprobe-tc Feb 16, 2024
7d21589
Refactored NXapm_paraprobe_intersector_results, fixed formatting issu…
Feb 16, 2024
92afaee
Proposal to move common data nicely out, fixes on nanochem config, ne…
atomprobe-tc Feb 18, 2024
70ac95f
Moved profiling and other common information out of task specific gro…
atomprobe-tc Feb 26, 2024
a2c13f5
Resolved logical problem that status was not always present
atomprobe-tc Feb 26, 2024
a0de2c4
Moving of analysis_identifier and config
atomprobe-tc Feb 26, 2024
03b15b5
First round of fixes for appdefs of nanochem tool for delocalization …
atomprobe-tc Feb 27, 2024
2b8c352
Fixes intersector config
atomprobe-tc Feb 27, 2024
ca5b4af
Changes due to comments from colleagues
atomprobe-tc Feb 28, 2024
41b354c
Final changes prior merge
atomprobe-tc Feb 28, 2024
9b8ee6b
Merge pull request #151 from FAIRmat-NFDI/composing-v0.5
mkuehbach Feb 28, 2024
b6cf0b2
Round 1, implementation feedback from colleagues
atomprobe-tc Feb 28, 2024
4fe028d
Recompiled NXDLs from YAML using nyaml==0.0.9
atomprobe-tc Feb 28, 2024
bb7f17a
Recompiled NXDLs from yaml using nyaml==0.0.9
atomprobe-tc Feb 28, 2024
d91898c
Manually merged em into apm
Feb 28, 2024
0b928c4
Deactivated use of NXactuator for now because em_refactoring branch d…
Feb 28, 2024
b37f8ec
Constrained name of ions in NXapm
Feb 28, 2024
ebd722c
Merge pull request #140 from FAIRmat-NFDI/apm_refactoring
mkuehbach Feb 29, 2024
6697f96
Manual merge of fairmat into em_refactoring
atomprobe-tc Feb 29, 2024
95b6505
Recompiled NXdata and NXsensor with nyaml==0.0.9 as they showed issue…
atomprobe-tc Feb 29, 2024
5740f4f
Clarifying situation with NXem_base
atomprobe-tc Feb 29, 2024
5fc31d0
Dropped NXem_base to not immediately be a competitor to NXem, all sug…
atomprobe-tc Feb 29, 2024
6961f34
Cleanly removed NXem_base class
atomprobe-tc Feb 29, 2024
d0358cd
Merge pull request #187 from FAIRmat-NFDI/em_refactoring_em_base_removal
mkuehbach Feb 29, 2024
3337196
Changes suggested by @domna
atomprobe-tc Mar 1, 2024
092580d
Reverted unintentional change NXsensor.nxdl.xml
mkuehbach Mar 1, 2024
7ff42ce
Revert that was unintentionally changed NXdata.nxdl.xml
mkuehbach Mar 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
274 changes: 231 additions & 43 deletions contributed_definitions/NXem.nxdl.xml

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion contributed_definitions/NXem_adf.nxdl.xml
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@
For now the base class provides for scans for which the settings,
binning, and energy resolution is the same for each scan point.
</doc>
<group name="IMAGE_R_SET" type="NXimage_r_set">
<group type="NXimage_r_set">
<field name="half_angle_interval" type="NX_NUMBER" units="NX_ANGLE">
<doc>
Annulus inner (first value) and outer (second value) half angle.
Expand Down
85 changes: 43 additions & 42 deletions contributed_definitions/NXem_base.nxdl.xml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ description on the NeXus front-page is overwhelming
considering what we learned from the diataxis workshop we write here a
specification neither a how to nor a tutorial which explains all the context
because we address here developers of software-->
<definition xmlns="http://definition.nexusformat.org/nxdl/3.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" category="base" type="group" name="NXem_base" extends="NXroot" xsi:schemaLocation="http://definition.nexusformat.org/nxdl/3.1 ../nxdl.xsd">
<definition xmlns="http://definition.nexusformat.org/nxdl/3.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" category="base" type="group" name="NXem_base" extends="NXobject" xsi:schemaLocation="http://definition.nexusformat.org/nxdl/3.1 ../nxdl.xsd">
<!--
template to be used for an application definition
-->
Expand All @@ -48,34 +48,10 @@ template to be used for an application definition
to explore physical mechanisms and phenomena, or the desire to characterize materials
using electron microscopy.
</doc>
<group type="NXprogram">
<doc>
A collection of all programs and libraries which are considered relevant
to understand with which software tools this NeXus file instance was
generated. Ideally, to enable a binary recreation from the input data.

Examples include the name and version of the libraries used to write the
instance. Ideally, the software that writes these :ref:`NXprogram` instances
also includes the version of the set of NeXus classes i.e. the specific
set of base classes, application definitions, and contributed definitions
with which the here described concepts can be resolved.

For the `pynxtools library &lt;https://github.com/FAIRmat-NFDI/pynxtools&gt;`_
which is used by the `NOMAD &lt;https://nomad-lab.eu/nomad-lab&gt;`_
research data management system, it makes sense to store e.g. the GitHub
repository commit and respective submodule references used.
</doc>
</group>
<!--each NeXus file instance should have a default plot
however as there are cases when this cannot be assured we cannot
make the default required, one example is e.g. a NeXus instance
where scientists just store conventions without a default plot-->
<group name="cs_profiling" type="NXcs_profiling">
<doc>
The configuration of the I/O writer software (e.g. `pynxtools &lt;https://github.com/FAIRmat-NFDI/pynxtools&gt;`_)
which was used to generate this NeXus file instance.
</doc>
</group>
<group type="NXentry">
<attribute name="version" type="NX_CHAR">
<doc>
Expand All @@ -91,6 +67,30 @@ where scientists just store conventions without a default plot-->
<item value="NXem"/>
sanbrock marked this conversation as resolved.
Show resolved Hide resolved
</enumeration>
</field>
<group name="cs_profiling" type="NXcs_profiling">
<doc>
The configuration of the I/O writer software (e.g. `pynxtools &lt;https://github.com/FAIRmat-NFDI/pynxtools&gt;`_)
which was used to generate this NeXus file instance.
</doc>
<group type="NXprogram">
mkuehbach marked this conversation as resolved.
Show resolved Hide resolved
<doc>
A collection of all programs and libraries which are considered relevant
to understand with which software tools this NeXus file instance was
generated. Ideally, to enable a binary recreation from the input data.

Examples include the name and version of the libraries used to write the
instance. Ideally, the software which writes these NXprogram instances
also includes the version of the set of NeXus classes i.e. the specific
set of base classes, application definitions, and contributed definitions
with which the here described concepts can be resolved.

For the `pynxtools library &lt;https://github.com/FAIRmat-NFDI/pynxtools&gt;`_
which is used by the `NOMAD &lt;https://nomad-lab.eu/nomad-lab&gt;`_
research data management system, it makes sense to store e.g. the GitHub
repository commit and respective submodule references used.
</doc>
</group>
</group>
<group name="experiment_identifier" type="NXidentifier">
<doc>
Ideally, a (globally) unique persistent identifier
Expand Down Expand Up @@ -150,25 +150,26 @@ where scientists just store conventions without a default plot-->
</doc>
</field>
<group type="NXcite"/>
<group type="NXprogram">
<doc>
The program and eventual software libraries used with which the
NeXus instance was created. For the NOMAD OASIS research data
management system e.g. pynxtools and eventually all modules
if desired.
</doc>
</group>
<!--the above-description overwrites the default description of the NXprogram base class
this is composed from the NXprogram base class
program:
\@version:
\@url:
NXnote and thumbnail dropped for the reason that these are
arbitrary binary containers without any clear provenance.-->
<group type="NXserialized">
<doc>
Possibility to store a collection of data artifacts
associated with the experiment.
Possibility to store a collection of serialized resources
that are associated with the experiment.

An example how to use this set could be to document
from which files generated by software of technology partners
the information in an instance of NXem was filled.

If resources from technology partners would also be documented
semantically annotated there would not even be a necessity to
copy over specific information in NeXus files as one could write
a inference and information/fact/knowledge retrieval algorithm whereby
a specific piece of information that is related to an experiment or simulation
is just read directly from the respective file of the technology partner.

The reason why currently this works convincingly in hardly any research
data management system is the strong heterogeneity of the information contained
in such files and the fact that often context and documentation specifically
rich semantic documentation and contextualization is missing.
</doc>
</group>
<!--using NXserialized here instead of NXnote as the former is more specific-->
Expand Down
83 changes: 66 additions & 17 deletions contributed_definitions/NXem_eds.nxdl.xml
Original file line number Diff line number Diff line change
Expand Up @@ -25,17 +25,10 @@
<!--NEW ISSUE: use computational geometry to offer arbitrary scan pattern
NEW ISSUE: make the binning flexible per scan point-->
<symbols>
<!--n_p: Number of scan points-->
<symbol name="n_y">
<doc>
Number of pixel along the y direction, the slow direction
</doc>
</symbol>
<symbol name="n_x">
<doc>
Number of pixel along the x direction, the fast direction
</doc>
</symbol>
<!--n_p: Number of scan points
n_z: Number of pixel along the z direction, the slowest direction
n_y: Number of pixel along the y direction, the slow direction
n_x: Number of pixel along the x direction, the fast direction-->
<symbol name="n_photon_energy">
<doc>
Number of X-ray photon energy (bins), the fastest direction.
Expand All @@ -53,12 +46,19 @@ NEW ISSUE: make the binning flexible per scan point-->
</symbol>
</symbols>
<doc>
Base class method-specific for energy-dispersive X-ray spectroscopy (EDS/EDX).
Base class method-specific for energy-dispersive X-ray spectroscopy (EDS/EDXS).

`IUPAC instead of Siegbahn notation &lt;https://doi.org/10.1002/xrs.1300200308&gt;`_ should be used.

X-ray spectroscopy is a surface-sensitive technique. Therefore, three-dimensional elemental
characterzation requires typically a sequence of characterization and preparation of the
surface to expose a new surface layer that can be characterized in the next acquisition.
In effect, the resulting three-dimensional elemental information mappings are truely the
result of a correlation and post-processing of several measurements which is the field
of correlative tomographic usage of electron microscopy.
</doc>
<!--NXprocess is composed from NXem_method base class
SPECTRUM_SET(NXspectrum_set) is composed from NXem_method base class
<!--NXprocess is composed from NXem_method base class instances where the spectra
are stored as instances of (NXspectrum_set) is composed from NXem_method base class
for post-processing of/with the above-defined data entries
including feedback from Christoph Pauly (from MX Uni Saarland, NFDI-MatWerk),
Sabine Bergmann and Sebastian Brückner (both from FAIRmat, IKZ),
Expand All @@ -72,7 +72,38 @@ and Adrien Teutrie, Cecile Hebert (EPFL)-->
The program with which the indexing was performed.
</doc>
</group>
<group name="PEAK" type="NXpeak">
<group name="summary" type="NXdata">
<doc>
Accumulated intensity over all pixels of the region-of-interest.
</doc>
<field name="intensity" type="NX_NUMBER" units="NX_UNITLESS">
<doc>
Accumulated counts
</doc>
<dimensions rank="1">
<dim index="1" value="n_photon_energy"/>
</dimensions>
<attribute name="long_name" type="NX_CHAR">
<doc>
Counts
</doc>
</attribute>
</field>
<field name="axis_energy" type="NX_NUMBER" units="NX_ENERGY">
<doc>
Energy axis
</doc>
<dimensions rank="1">
<dim index="1" value="n_photon_energy"/>
</dimensions>
<attribute name="long_name" type="NX_CHAR">
<doc>
Energy
</doc>
</attribute>
</field>
</group>
<group type="NXpeak">
sanbrock marked this conversation as resolved.
Show resolved Hide resolved
<doc>
Name and location of each X-ray line which was indexed as a known ion.
For each ion, an NXion instance should be created which specifies
Expand Down Expand Up @@ -109,13 +140,18 @@ and Adrien Teutrie, Cecile Hebert (EPFL)-->
</group>
<field name="element_names" type="NX_CHAR">
<doc>
List of the names of identified elements.
Comma-separated list of names of identified elements. All members of
the list have to be valid chemical_symbols from the periodic table.

This field can be used when creating instances of NXpeak is not desired.
However, a collection of instances of NXpeak with individual NXion specified
enables also to distinguish isotopic information.
</doc>
<dimensions rank="1">
<dim index="1" value="n_elements"/>
</dimensions>
</field>
<group name="IMAGE_R_SET" type="NXimage_r_set">
<group type="NXimage_r_set">
<doc>
Individual element-specific EDS/EDX/EDXS/SXES mapping

Expand All @@ -125,6 +161,12 @@ and Adrien Teutrie, Cecile Hebert (EPFL)-->
These element-specific EDS maps are NXimage_r_set instances
and need to be named with the name of the element from the
element_names field.

We often observe that signal contributions from several peaks
are summarized and shown together, e.g. the combined signal
under the curve of carbon and oxygen.

In this case specify the processing details using peaks and weights.
</doc>
<group type="NXprocess">
<field name="peaks" type="NX_CHAR">
Expand All @@ -137,6 +179,13 @@ and Adrien Teutrie, Cecile Hebert (EPFL)-->
<dim index="1" value="n_peaks"/>
</dimensions>
</field>
<field name="weights" type="NX_NUMBER" units="NX_UNITLESS">
<doc>
A list of weights by how much the intensity of each peak
under peaks was factorized to display the joint intensity
of the image.
</doc>
</field>
</group>
</group>
</group>
Expand Down
13 changes: 4 additions & 9 deletions contributed_definitions/NXem_eels.nxdl.xml
Original file line number Diff line number Diff line change
Expand Up @@ -32,19 +32,14 @@
<doc>
Base class method-specific for Electron Energy Loss Spectroscopy (EELS).
</doc>
<!--NXem_method can offers to keep a SPECTRUM_SET
NXem_method also has an NXprocess which in this base class can be
<!--NXem_method also has an NXprocess which in this base class can be
specialized to include EELS-specific post-processing-->
<group name="SPECTRUM_SET" type="NXspectrum_set">
<group type="NXspectrum_set">
<doc>
NXspectrum_set_em specialized for EELS.
</doc>
<group name="stack" type="NXdata">
<!--\@signal: data_counts
\@axes: [axis_y, axis_x, axis_energy_loss]
\@energy_loss_indices: 2
\@axis_x_indices: 1
\@axis_y_indices: 0-->
<!--\@signal:, \@axes: \@energy_loss_indices \@axis_x_indices:-->
<group name="stack_oned" type="NXdata">
<field name="axis_energy_loss" type="NX_NUMBER" units="NX_ENERGY">
<dimensions rank="1">
<dim index="1" value="n_energy_loss"/>
Expand Down
Loading
Loading