Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EM refactoring / EMv2 #136

Merged
merged 69 commits into from
Mar 1, 2024
Merged
Show file tree
Hide file tree
Changes from 66 commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
764093c
Fixed missing references to EM/CGMS/ICME-related base classes in the …
atomprobe-tc Dec 13, 2023
8a3ae1b
Refactoring storage of images and spectra
atomprobe-tc Dec 13, 2023
6663bec
Minor edit
atomprobe-tc Dec 13, 2023
dba0e64
Refactoring apm has three aims: i) extent NXapm appdef to take advant…
atomprobe-tc Jan 3, 2024
261f91b
Testing the use of NXapm_paraprobe_tool_config/results base classes o…
atomprobe-tc Jan 3, 2024
7abeded
Applied to i)
atomprobe-tc Jan 3, 2024
b50f160
Working on ii) mirrored concept used for NXem with NXapm_msr and NXap…
atomprobe-tc Jan 4, 2024
cdcebea
Implemented design of event_data_apm, correlated signals with keyword…
atomprobe-tc Jan 5, 2024
7e4be9b
Editing NXapm_ranging remains and a final check of the NXapm_new, the…
Jan 5, 2024
373da83
Refactored NXapm_ranging, renamed NXapm_new, a through and complete c…
Jan 5, 2024
df219e8
Reviewing, fixing typos, a minor glitches
Jan 6, 2024
6b649bc
Further reviewing and corrections
Jan 6, 2024
701f277
NXapm reviewing and typos
Jan 7, 2024
34fc49b
Fixed issues detected during make local, manual builds successfully, …
atomprobe-tc Jan 8, 2024
5c7d197
Added changes relevant to use the refactored NXapm parser of pynxtool…
atomprobe-tc Jan 10, 2024
f402b33
Fixes and additions revealed from implementing parser functionalities…
atomprobe-tc Jan 17, 2024
c1aa2e3
Placed version as child of definition
atomprobe-tc Jan 17, 2024
23d34bc
Added version as a child of definition, spotted what could be a bug t…
atomprobe-tc Jan 17, 2024
0a2bd84
Fixes and modifications revealed during implementation of imgs, apex …
atomprobe-tc Jan 17, 2024
e37edaf
Added suggestions from @lukaspie on NXpeak
atomprobe-tc Jan 17, 2024
f816790
Fixes introduced for Velox EMD file parser
atomprobe-tc Jan 19, 2024
1072a9a
Fixed definitions and version
atomprobe-tc Jan 23, 2024
8041523
Fixed definitions and version
atomprobe-tc Jan 23, 2024
3b266bf
Removed unfortunated row vector style matrix isotope_vector and nucli…
atomprobe-tc Jan 23, 2024
c043182
Updated apm-specific documentation for new base classes and refactori…
atomprobe-tc Jan 23, 2024
9745686
Fixes and modifications specifically for paraprobe-toolbox tools: par…
atomprobe-tc Jan 25, 2024
4b84b08
Added appdef for selector_results, reorganized where to place common …
atomprobe-tc Jan 25, 2024
0d40ece
Reorganized transcoder, ranger, selector
atomprobe-tc Jan 26, 2024
a172c01
Refactored surfacer to use APMv2
atomprobe-tc Jan 26, 2024
8b6ee88
Refactored appdefs for distancer and tessellator to use APMv2
atomprobe-tc Jan 27, 2024
8ac38a5
Fixes added in the process of refactoring parmsetup-distancer and par…
Jan 27, 2024
7c1f072
Fixes to complete distancer and tessellator APMv2 refactoring
atomprobe-tc Jan 28, 2024
13efc84
Refactored appdefs for paraprobe-spatstat, specified symbol to use fo…
Jan 28, 2024
a04b6fb
Minor editing of spatstat
atomprobe-tc Feb 9, 2024
5291ffe
Refactored paraprobe-clusterer definitions to use APM v2
atomprobe-tc Feb 10, 2024
6122e74
Added missing profiling and program for results appdefs of tools wher…
atomprobe-tc Feb 10, 2024
23722e2
Renaming the last two missing appdefs for tools nanochem and intersec…
atomprobe-tc Feb 13, 2024
7892cf9
Refactored config files for nanochem and intersector to use APMv2
atomprobe-tc Feb 16, 2024
2df16a3
Updated apm-specific landing page
atomprobe-tc Feb 16, 2024
0bd04c7
Minor edits and fixes apm-structure
atomprobe-tc Feb 16, 2024
d5fa011
Fixing a copy-paste error
atomprobe-tc Feb 16, 2024
38bb082
Fixed inconsistence
atomprobe-tc Feb 16, 2024
f8cb8db
Refactored NXapm_paraprobe_nanochem_results and delocalization
atomprobe-tc Feb 16, 2024
7d21589
Refactored NXapm_paraprobe_intersector_results, fixed formatting issu…
Feb 16, 2024
92afaee
Proposal to move common data nicely out, fixes on nanochem config, ne…
atomprobe-tc Feb 18, 2024
70ac95f
Moved profiling and other common information out of task specific gro…
atomprobe-tc Feb 26, 2024
a2c13f5
Resolved logical problem that status was not always present
atomprobe-tc Feb 26, 2024
a0de2c4
Moving of analysis_identifier and config
atomprobe-tc Feb 26, 2024
03b15b5
First round of fixes for appdefs of nanochem tool for delocalization …
atomprobe-tc Feb 27, 2024
2b8c352
Fixes intersector config
atomprobe-tc Feb 27, 2024
ca5b4af
Changes due to comments from colleagues
atomprobe-tc Feb 28, 2024
41b354c
Final changes prior merge
atomprobe-tc Feb 28, 2024
9b8ee6b
Merge pull request #151 from FAIRmat-NFDI/composing-v0.5
mkuehbach Feb 28, 2024
b6cf0b2
Round 1, implementation feedback from colleagues
atomprobe-tc Feb 28, 2024
4fe028d
Recompiled NXDLs from YAML using nyaml==0.0.9
atomprobe-tc Feb 28, 2024
bb7f17a
Recompiled NXDLs from yaml using nyaml==0.0.9
atomprobe-tc Feb 28, 2024
d91898c
Manually merged em into apm
Feb 28, 2024
0b928c4
Deactivated use of NXactuator for now because em_refactoring branch d…
Feb 28, 2024
b37f8ec
Constrained name of ions in NXapm
Feb 28, 2024
ebd722c
Merge pull request #140 from FAIRmat-NFDI/apm_refactoring
mkuehbach Feb 29, 2024
6697f96
Manual merge of fairmat into em_refactoring
atomprobe-tc Feb 29, 2024
95b6505
Recompiled NXdata and NXsensor with nyaml==0.0.9 as they showed issue…
atomprobe-tc Feb 29, 2024
5740f4f
Clarifying situation with NXem_base
atomprobe-tc Feb 29, 2024
5fc31d0
Dropped NXem_base to not immediately be a competitor to NXem, all sug…
atomprobe-tc Feb 29, 2024
6961f34
Cleanly removed NXem_base class
atomprobe-tc Feb 29, 2024
d0358cd
Merge pull request #187 from FAIRmat-NFDI/em_refactoring_em_base_removal
mkuehbach Feb 29, 2024
3337196
Changes suggested by @domna
atomprobe-tc Mar 1, 2024
092580d
Reverted unintentional change NXsensor.nxdl.xml
mkuehbach Mar 1, 2024
7ff42ce
Revert that was unintentionally changed NXdata.nxdl.xml
mkuehbach Mar 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,8 @@ pdf ::
$(SPHINX) -M latexpdf $(BUILD_DIR)/manual/source/ $(BUILD_DIR)/manual/build
cp $(BUILD_DIR)/manual/build/latex/nexus-fairmat.pdf $(BUILD_DIR)/manual/source/_static/NeXusManual.pdf

# for development purposes switch off the -W flag in the following line to make testing the local
# building of the manual more efficient
domna marked this conversation as resolved.
Show resolved Hide resolved
html ::
$(SPHINX) -b html -W $(BUILD_DIR)/manual/source/ $(BUILD_DIR)/manual/build/html

Expand Down
10 changes: 3 additions & 7 deletions base_classes/NXdata.nxdl.xml
Original file line number Diff line number Diff line change
Expand Up @@ -236,13 +236,9 @@ without this attribute being set to "true".-->
</doc>
</attribute>
<attribute name="AXISNAME_indices" type="NX_INT">
<!--
nxdl.xsd rules do not allow us to show this as a variable name
- we'll use ALL CAPS (see #562)
-->
<!--
AXISNAME_indices documentation copied from datarules.rst
-->
<!--nxdl.xsd rules do not allow us to show this as a variable name
- we'll use ALL CAPS (see #562)-->
<!--AXISNAME_indices documentation copied from datarules.rst-->
<doc>
Each ``AXISNAME_indices`` attribute indicates the dependency
relationship of the ``AXISNAME`` field (where ``AXISNAME``
Expand Down
4 changes: 2 additions & 2 deletions base_classes/NXsensor.nxdl.xml
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@
- same dimensions as "value" (may be a vector)
</doc>
<dimensions>
<dim index="1" ref="value"/>
<dim index="1" value="[1]" ref="value"/>
</dimensions>
</field>
<field name="value_deriv2" type="NX_FLOAT" units="NX_ANY">
Expand All @@ -132,7 +132,7 @@
- same dimensions as "value" (may be a vector)
</doc>
<dimensions>
<dim index="1" ref="value"/>
<dim index="1" value="[1]" ref="value"/>
</dimensions>
</field>
<group name="value_log" type="NXlog">
Expand Down
2,043 changes: 708 additions & 1,335 deletions contributed_definitions/NXapm.nxdl.xml

Large diffs are not rendered by default.

157 changes: 157 additions & 0 deletions contributed_definitions/NXapm_charge_state_analysis.nxdl.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,157 @@
<?xml version='1.0' encoding='UTF-8'?>
<?xml-stylesheet type="text/xsl" href="nxdlformat.xsl"?>
<!--
# NeXus - Neutron and X-ray Common Data Format
#
# Copyright (C) 2014-2024 NeXus International Advisory Committee (NIAC)
#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
# License as published by the Free Software Foundation; either
# version 3 of the License, or (at your option) any later version.
#
# This library is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with this library; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
#
# For further information, see http://www.nexusformat.org
-->
<definition xmlns="http://definition.nexusformat.org/nxdl/3.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" category="base" type="group" name="NXapm_charge_state_analysis" extends="NXprocess" xsi:schemaLocation="http://definition.nexusformat.org/nxdl/3.1 ../nxdl.xsd">
<symbols>
<doc>
The symbols used in the schema to specify e.g. dimensions of arrays.
</doc>
<symbol name="n_cand">
<doc>
The number of also possible but different (molecular) ions.
</doc>
</symbol>
<symbol name="n_ivec_max">
<doc>
Maximum number of allowed atoms per (molecular) ion (fragment).
</doc>
</symbol>
</symbols>
<doc>
Base class to document an algorithm for recovering charge state and nuclide composition of a (molecular) ion.

Currently ranging definitions in the research field of atom probe face have limitations:

1. A ranging definition maps all signal within a mass-to-charge-state-ratio value interval
on one iontype. Facing limited mass-resolving-power, there are mass-to-charge-state-ratio
values, though, for which not only multiple (molecular) ions are indistinguishable but
also for which the current practice of documenting classical ranging definitions is incomplete.
2. Indeed, ranging definitions often report only (for each interval) the
mass-to-charge-state-ratio intervals surplus the composition of elements
that build the (molecular) ion.
3. Therefore, classical ranging definitions demand a post-processing with an algorithm
which can identify nuclides from which the (molecular) ion is constructed
and a charge state possibly recovered. Combinatorial algorithms are used for this purpose.

This base class documents the configuration and results of such an algorithm.
</doc>
<!--Details and results of the combinatorial analyses of a ranging definition
to clarify (if possible) the charge_state of an ion and its (not necessarily)
unique combination of nuclides contained including their multiplicity.
input/config-->
<field name="element" type="NX_UINT" units="NX_UNITLESS">
<doc>
Input constraint, list of proton numbers for each element of the ranging
definition. The list contains each element as many times as its multiplicity.
As an example a ranging definition H:2 O:1 demands element vector
to be 1, 1, 8. An empty list does not release the constraint.
Instead, a list with all elements in the periodic table should be
used. Keep in mind though with such a weakly constrained parameter space
the combinatorial analysis may become very time consuming!
</doc>
<dimensions>
<dim index="1" value="i"/>
</dimensions>
</field>
<field name="mass_to_charge_range" type="NX_FLOAT" units="NX_ANY">
<doc>
Input constraint, interval within which (molecular) ions need to have the
mass-to-charge-state-ratio such that an ion qualifies as a candidate.
</doc>
<dimensions>
<dim index="1" value="1"/>
<dim index="2" value="2"/>
</dimensions>
</field>
<field name="min_half_life" type="NX_FLOAT" units="NX_TIME">
<doc>
Input constraint, minimum half life for how long each nuclide of each
(molecular) ion needs to be stable such that the ion qualifies as a candidate.
</doc>
</field>
<field name="min_abundance" type="NX_FLOAT" units="NX_DIMENSIONLESS">
<doc>
Input constraint, minimum natural abundance of each nuclide of each
(molecular) ion such that the ion qualifies as a candidate.
</doc>
</field>
<field name="sacrifice_isotopic_uniqueness" type="NX_BOOLEAN">
<doc>
If the value is false, it means that non-unique solutions are accepted.
These are solutions where multiple candidates have been built from
different nuclide instances but the charge_state of all the ions is the same.
</doc>
</field>
<!--min_abundance_product(NX_FLOAT):
doc: |
For each candidate TO BE DEFINED.
unit: NX_DIMENSIONLESS
dim: (n_cand,)-->
<!--output/results
the n_cand can be 1 in which case all quantities below are scalar-->
<field name="charge_state" type="NX_INT" units="NX_UNITLESS">
<doc>
Signed charge, i.e. integer multiple of the elementary
charge of each candidate.
</doc>
<dimensions>
<dim index="1" value="n_cand"/>
</dimensions>
</field>
<field name="nuclide_hash" type="NX_UINT" units="NX_UNITLESS">
<doc>
Table of nuclide instances of which each candidate is composed.
Each row vector is sorted in descending order. Unused values are nullified.
</doc>
<dimensions>
<dim index="1" value="n_cand"/>
<dim index="2" value="n_ivec_max"/>
</dimensions>
</field>
<field name="mass" type="NX_FLOAT" units="NX_MASS">
<doc>
Accumulated mass of the nuclides in each candidate.
Not corrected for quantum effects.
</doc>
<dimensions>
<dim index="1" value="n_cand"/>
</dimensions>
</field>
<field name="natural_abundance_product" type="NX_FLOAT" units="NX_DIMENSIONLESS">
<doc>
The product of the natural abundances of the nuclides for each candidate.
</doc>
<dimensions>
<dim index="1" value="n_cand"/>
</dimensions>
</field>
<field name="shortest_half_life" type="NX_FLOAT" units="NX_TIME">
<doc>
For each candidate the half life of that nuclide which has the shortest half
life.
</doc>
<dimensions>
<dim index="1" value="n_cand"/>
</dimensions>
</field>
</definition>
147 changes: 147 additions & 0 deletions contributed_definitions/NXapm_hit_finding.nxdl.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
<?xml version='1.0' encoding='UTF-8'?>
<?xml-stylesheet type="text/xsl" href="nxdlformat.xsl"?>
<!--
# NeXus - Neutron and X-ray Common Data Format
#
# Copyright (C) 2014-2024 NeXus International Advisory Committee (NIAC)
#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
# License as published by the Free Software Foundation; either
# version 3 of the License, or (at your option) any later version.
#
# This library is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with this library; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
#
# For further information, see http://www.nexusformat.org
-->
<definition xmlns="http://definition.nexusformat.org/nxdl/3.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" category="base" type="group" name="NXapm_hit_finding" extends="NXprocess" xsi:schemaLocation="http://definition.nexusformat.org/nxdl/3.1 ../nxdl.xsd">
<symbols>
<doc>
The symbols used in the schema to specify e.g. dimensions of arrays.
</doc>
<symbol name="n_dld">
<doc>
Number of delay-line wires of the detector.
</doc>
</symbol>
<symbol name="n_ht">
<doc>
Number of hit qualities (hit types) distinguished.
</doc>
</symbol>
<symbol name="p">
<doc>
Number of pulses collected in between start_time and end_time
resolved by an instance of :ref:`NXevent_data_apm`. If this is not defined,
p is the number of ions included in the reconstructed volume if an application
definition is used to store results of already reconstructed datasets.
</doc>
</symbol>
</symbols>
<doc>
Base class for the configuration and results from a hit finding algorithm.
</doc>
<group type="NXprogram"/>
<group type="NXserialized"/>
<!--config/input-->
<field name="number_of_dld_wires" type="NX_UINT" units="NX_UNITLESS">
<doc>
The number of wires in the detector.
</doc>
<enumeration>
<item value="1"/>
<item value="2"/>
<item value="3"/>
</enumeration>
</field>
<field name="dld_wire_names" type="NX_CHAR">
<doc>
Alias tuple (begin, end) of each DLD wire of the detector.
Order follows arrival_time_pairs.
</doc>
<dimensions>
<dim index="1" value="n_dld"/>
<dim index="2" value="2"/>
</dimensions>
</field>
<field name="arrival_time_pairs" type="NX_NUMBER" units="NX_TIME">
<doc>
Raw readings from the analog-to-digital-converter
timing circuits of the detector wires.
</doc>
<dimensions>
<dim index="1" value="p"/>
<dim index="2" value="n_dld"/>
<dim index="3" value="2"/>
</dimensions>
</field>
<!--results of the hit_finding algorithm-->
<field name="hit_positions" type="NX_NUMBER" units="NX_LENGTH">
<doc>
Evaluated ion impact coordinates on the detector.
Use the depends_on field to spec
</doc>
<dimensions>
<dim index="1" value="p"/>
<dim index="2" value="2"/>
</dimensions>
<attribute name="depends_on" type="NX_CHAR">
<doc>
The instance of :ref:`NXcoordinate_system` in which the positions are defined.
</doc>
</attribute>
</field>
<field name="hit_quality_types" type="NX_CHAR">
<doc>
Name of the hit_qualities distinguished.
AMETEK/Cameca uses e.g. golden, multiple, partial,
irrecoverable, and multi-first and multi-late.
</doc>
<dimensions>
<dim index="1" value="n_ht"/>
</dimensions>
</field>
<field name="hit_quality_identifier" type="NX_UINT">
<doc>
Identifier used for each hit_quality type.
Following the order of hit_quality_types.
</doc>
</field>
<field name="hit_quality" type="NX_UINT" units="NX_UNITLESS">
<doc>
Hit quality identifier for each pulse.
Identifier have to be within hit_quality_identifier.
</doc>
<dimensions>
<dim index="1" value="p"/>
</dimensions>
</field>
<field name="hit_multiplicity" type="NX_UINT" units="NX_UNITLESS">
<doc>
This processing yields for each ion with how many others it evaporated
if these were collected on the same pulse. Extraction of multiple ions
on one pulse on different or even the same pixel of the detector are possible.

Multiplicity must not be confused with how many atoms of the same element
a molecular ion contains (which is instead encoded with the
isotope_vector field of each :ref:`NXion` instance).
</doc>
<dimensions>
<dim index="1" value="p"/>
</dimensions>
</field>
<!--the following two quantities are relicts from ePOS files used to give some
insight into the results of the hit_finding algorithm of IVAS/APSuite but typically
used only in the context to learn about the multiplicity of an ion.
pulses_since_last_ion(NX_UINT):
dim: (n,)
pulse_identifier(NX_INT):
dim: (n,)-->
</definition>
Loading
Loading