-
Notifications
You must be signed in to change notification settings - Fork 9
Reading data from Tiled server
This is of no interest to Irena and Nika users.
Tiled server (https://blueskyproject.io/tiled/, https://github.com/bluesky/tiled) is used at APS for Bluesky (https://github.com/bluesky/bluesky) data storage and access. Igor Pro (IP) USAXS code (Indra) communicates with the Tiled server and extracts the scan data to plot them for user. Current Indra version under development (1/2024) contains GUI and is limited to USAXS specific types of scans. Extending the code is possible, even though difficult to imagine for very generic case...
The communication is done using custom URL commands to web server which returns json package. The json is downloaded by IP and parsed in memory to extract useful data. Creating these custom URL is non trivial and it is Tiled version dependent. Documentation for Tiled is available, but large and challenge to read for non expert. Here are notes needed to understand how IP code in Indra (USAXS data reduction tool) builds the URLs.
To figure out what is happening, one needs to build the URLs and test using web browser. Error messages echoed on the web page are actually useful. Once URL works on web browser (Firefox seems to work well), time to move to Igor, which can download json and print ASCII representation of it, FORMATTED, which can be copied and placed in editor (BBedit, web browser) and studied for the structure.
Some details on the URL structure. There is standard part and catalog-related part. Nice description from Dan:
URLs are structured like
/api/v1/VERB/NOUN, where VERB is something like/search,/metadata/,/array/full. NOUN is fully up to the catalog/application, it would be/{uuid}or/{beamline}/{uuid}or/raw/{beamline}/{uuid}or/whatever/you/want.
JIL note: This limits ability to support multiple catalogs/applications as we do not know the structure for unknown source. One could design code which traverses/discovers the structure, but that seems really complicated. Other option is to create string Prefix which would be catalog specific, potentially even user configurable. This may be TODO in the future, for suggestion how to do this, see Traversing to catalog at the end.
Currently, Tiled server is running on computer called usaxscontrol and is limited to use unencrypted protocol (http:). Therefore, it is limited to access inside APS or using vpn, this server cannot be accessed from outside APS. Tiled is currently running on port 8020, this can be changed in Tiled configuration.
Code: all commands start with http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/
Tiled server can provide access to multiple catalogs. We need to figure out which ones are available.
Code: this command http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/search/?fields=&sort= returns json with data which ids are names of catalogs. This is short and simple json to deal with.
Tiled provides access through few "endpoints" from which one can get different data.
- endpoint
/searchprovides access to metadata, optionally filtered. This way we can get basically everything else, than data themselves. - endpoint
/node/fullprovides access to data, optionally filtered. - endpoint
/array/fullprovides single array of data.
Code: append to command search or node/full or array/full so by now we have for example http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/search/
http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/node/full/
http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/array/full/
for example "idb_usaxs_retired_2023-12-05" which is currently the one available on the usaxscontrol.
Code: append idb_usaxs_retired_2023-12-05/ so now we have for example http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/search/idb_usaxs_retired_2023-12-05/
Now we need to select which data we want. We have at least two choices:
- We want specific data set, which is identified by its UUID, e.g.
f8c83910-4adb-4207-a465-9ff0ff0e9cd2- these are scan identifiers used by Bluesky. - We want multiple data sets, which can identified by metadata - date/times/plan_name/any other metadata matches. Note: Bluesky is using UTC time - number in seconds since 1/1/1970 - while Igor provides time in seconds from 1/1/1904, so there is conversion described in IP help.
Code:
Case 1 - to get specific data set, append the UUID/primary/data : http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/search/f8c83910-4adb-4207-a465-9ff0ff0e9cd2/primary/data
Case 2 - to get range of data sets - syntax example : ?page[offset]=0&page[limit]=250&filter[time_range][condition][since]=FROM_START_TIME&filter[time_range][condition][until]=BEFORE_END_TIME&sort=time&filter[eq][condition][key]=plan_name&filter[eq][condition][value]="tune_a2rp"
for example:
http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/search/idb_usaxs_retired_2023-12-05?page[offset]=0&page[limit]=250&filter[time_range][condition][since]=1678341600&filter[time_range][condition][until]=1678413600&filter[eq][condition][key]=plan_name&filter[eq][condition][value]="tune_a2rp"&filter[time_range][condition][timezone]=US/Central&sort=time
Explanation of syntax :
-
?page[offset]=0&&page[limit]=250- start from first data set and include values for up to first 250 data sets. Tiled has some limit on number of data which can be included in a single document, so downloading in multiple chunks of known length makes it easier to download all. My code keep downloading in 250 sets until returned json contains less than 250 data sets, then it ends. -
&filter[time_range][condition][since]=1678341600&filter[time_range][condition][until]=1678413600- this is time range. 24 hours in this case. -
&filter[eq][condition][key]=plan_name&filter[eq][condition][value]="tune_a2rp"- include ONLY data sets for which metadata field"plan_name"is equal to"tune_a2rp". Not sure how this would work if the key name would not be unique in metadata of the data set. -
&filter[time_range][condition][timezone]=US/Central- use US Central time zone (default is UTC which is bit confusing for users...) -
&sort=time- return sorted by time.
To get json data format, we need to add ?format=json
Code: append ?format=json :
http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/search/f8c83910-4adb-4207-a465-9ff0ff0e9cd2/primary/?format=json - returns json which contains all metadata for this scan but no data.
http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/array/full/f8c83910-4adb-4207-a465-9ff0ff0e9cd2/primary/data/a_stage_r?format=json - returns json which contains 1D data for a_stage_r axis during the scan. If we want to use this, we need to download each 1D array separately. The structure of the json is extremely simple one one array with no metadata at all, should be useful for large data sets.
http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/node/full/f8c83910-4adb-4207-a465-9ff0ff0e9cd2/primary/?format=json
- returns (in our USAXS specific case) json which contains 4 arrays of 1D data, first is detector, second is scanned axis position, third is time per point. Structure of the json is only bit more complicated with 4 arrays and no metadata.
To reduce the amount of data we are downloading, we can tell Tiled which fields to keep:
&fields=metadata will null all other fields but metadata. It seems to be possible to select only very specific "enumeration members", options are 'metadata', 'structure_family', 'structure', 'count', 'sorting', 'specs', 'data_sources'. Bit confusing what these mean for me...
&omit_links=true will null web links to data.
NOTE : these commands do NOT change the structure of json file, other fields than metadata are simply set to null. Depending on amount of data in the fields, this may or may not reduce the size of json too much.
Code: http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/search/idb_usaxs_retired_2023-12-05?page[offset]=0&page[limit]=250&filter[time_range][condition][since]=1678341600&filter[time_range][condition][until]=1678413600&filter[eq][condition][key]=plan_name&filter[eq][condition][value]="tune_a2rp"&filter[time_range][condition][timezone]=US/Central&sort=time&fields=metadata&omit_links=true
This is VERY efficient if one wants ONLY few items from the database, in this case we will itemize what is it we want. Note, that syntax is critical... Proper { , and : are critical. In our case I need only UUID, names of detector, motor, plan name, and start time when collected to build user GUI tables. NOTE: this changes structure of json.
Code: append &select_metadata={detectors:start.detectors,motors:start.motors,plan_name:start.plan_name,time:start.time}
http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/search/idb_usaxs_retired_2023-12-05?page[offset]=0&page[limit]=250&filter[time_range][condition][since]=1678341600&filter[time_range][condition][until]=1678413600&filter[eq][condition][key]=plan_name&filter[eq][condition][value]="tune_a2rp"&filter[time_range][condition][timezone]=US/Central&sort=time&fields=metadata&omit_links=true&select_metadata={detectors:start.detectors,motors:start.motors,plan_name:start.plan_name,time:start.time}
Disclaimer: there must be lots of errors here and misunderstandings. Manuals and web pages are written mostly for experts so this is the best I was able to reverse engineer and get from helpful suggestions of experts. This is sufficient to provide functionality in IP now- identify which scans are available in catalog, sort them by time, select time range and download to IP user selected scans. These scan can then be plotted and analyzed as user needs it.
http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/search/idb_usaxs_retired_2023-12-05?page[offset]=0&page[limit]=1&filter[time_range][condition][since]=1678341600&filter[time_range][condition][until]=1678413600&filter[eq][condition][key]=plan_name&filter[eq][condition][value]=%22tune_ar%22&filter[time_range][condition][timezone]=US/Central&sort=time
{
"data": [{
"id": "d190fb2f-7935-4a32-ad7d-9bf4c53ec042",
"attributes": {
"ancestors": ["idb_usaxs_retired_2023-12-05"],
"structure_family": "container",
"specs": [{
"name": "BlueskyRun",
"version": "1"
}],
"metadata": {
"start": {
"uid": "d190fb2f-7935-4a32-ad7d-9bf4c53ec042",
"time": 1678341890.5996535,
"pid": 1521928,
"versions": {
"apstools": "1.6.10",
"area_detector_handlers": "0.0.10",
"bluesky": "1.10.0",
"databroker": "1.2.5",
"epics_ca": "3.5.0",
"epics": "3.5.0",
"h5py": "3.8.0",
"matplotlib": "3.6.3",
"numpy": "1.23.5",
"ophyd": "1.7.0",
"pymongo": "4.3.3",
"pyRestTable": "2020.0.7",
"spec2nexus": "2021.2.5"
},
"EPICS_HOST_ARCH": "linux-x86_64",
"login_id": "[email protected]",
"beamline_id": "APS 9-ID-C USAXS",
"proposal_id": "testing",
"scan_id": 441,
"EPICS_CA_MAX_ARRAY_BYTES": "1280000",
"epics_libca": "/home/beams11/USAXS/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/epics/clibs/linux64/libca.so",
"plan_type": "generator",
"plan_name": "tune_ar",
"detectors": ["PD_USAXS"],
"motors": ["a_stage_r"],
"num_points": 35,
"num_intervals": 34,
"plan_args": {
"detectors": ["myScalerCH(prefix='9idcLAX:vsc:c0', name='scaler0', read_attrs=['channels', 'channels.chan04', 'channels.chan04.s', 'time'], configuration_attrs=['channels', 'channels.chan01', 'channels.chan01.chname', 'channels.chan01.preset', 'channels.chan01.gate', 'channels.chan04', 'channels.chan04.chname', 'channels.chan04.preset', 'channels.chan04.gate', 'count_mode', 'delay', 'auto_count_delay', 'freq', 'preset_time', 'auto_count_time', 'egu'])", "UsaxsArMotorTunable(prefix='9idcAERO:m6', name='a_stage_r', parent='a_stage', settle_time=0.0, timeout=None, read_attrs=['user_readback', 'user_setpoint'], configuration_attrs=['user_offset', 'user_offset_dir', 'velocity', 'acceleration', 'motor_egu', 'width', 'tolerance'])"],
"num": 35,
"args": ["UsaxsArMotorTunable(prefix='9idcAERO:m6', name='a_stage_r', parent='a_stage', settle_time=0.0, timeout=None, read_attrs=['user_readback', 'user_setpoint'], configuration_attrs=['user_offset', 'user_offset_dir', 'velocity', 'acceleration', 'motor_egu', 'width', 'tolerance'])", 8.843317482847699, 8.845117482847698],
"per_step": "None"
},
"hints": {
"dimensions": [[["a_stage_r"], "primary"]]
},
"plan_pattern": "inner_product",
"plan_pattern_module": "bluesky.plan_patterns",
"plan_pattern_args": {
"num": 35,
"args": ["UsaxsArMotorTunable(prefix='9idcAERO:m6', name='a_stage_r', parent='a_stage', settle_time=0.0, timeout=None, read_attrs=['user_readback', 'user_setpoint'], configuration_attrs=['user_offset', 'user_offset_dir', 'velocity', 'acceleration', 'motor_egu', 'width', 'tolerance'])", 8.843317482847699, 8.845117482847698]
},
"tune_md": {
"width": 0.0018,
"initial_position": 8.844217482847698,
"time_iso8601": "2023-03-09 00:04:50.577816"
},
"tune_parameters": {
"num": 35,
"width": 0.0018,
"initial_position": 8.844217482847698,
"peak_choice": "com",
"x_axis": "a_stage_r",
"y_axis": "PD_USAXS"
},
"purpose": "tuner",
"datetime": "2023-03-09 00:04:50.559486"
},
"stop": {
"run_start": "d190fb2f-7935-4a32-ad7d-9bf4c53ec042",
"time": 1678341898.455549,
"uid": "385fb5c4-ec76-409b-afb2-1dba9607d837",
"exit_status": "success",
"reason": "",
"num_events": {
"baseline": 2,
"primary": 35
}
},
"summary": {
"uid": "d190fb2f-7935-4a32-ad7d-9bf4c53ec042",
"scan_id": 441,
"timestamp": 1678341890.5996535,
"datetime": "2023-03-09T00:04:50.599653",
"plan_name": "tune_ar",
"stream_names": ["aps_current_monitor", "baseline", "primary"],
"duration": 7.855895519256592
}
},
"structure": {
"contents": null,
"count": 3
},
"sorting": [{
"key": "_",
"direction": 1
}],
"data_sources": null
},
"links": {
"self": "http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/metadata/idb_usaxs_retired_2023-12-05/d190fb2f-7935-4a32-ad7d-9bf4c53ec042",
"search": "http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/search/idb_usaxs_retired_2023-12-05/d190fb2f-7935-4a32-ad7d-9bf4c53ec042",
"full": "http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/node/full/idb_usaxs_retired_2023-12-05/d190fb2f-7935-4a32-ad7d-9bf4c53ec042"
},
"meta": null
}],
"error": null,
"links": {
"self": "http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/search/idb_usaxs_retired_2023-12-05?page[offset]=0&page[limit]=1",
"first": "http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/search/idb_usaxs_retired_2023-12-05?page[offset]=0&page[limit]=1",
"last": "http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/search/idb_usaxs_retired_2023-12-05?page[offset]=136&page[limit]=1",
"next": "http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/search/idb_usaxs_retired_2023-12-05?page[offset]=1&page[limit]=1",
"prev": null
},
"meta": {
"count": 136
}
}
http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/search/idb_usaxs_retired_2023-12-05?page[offset]=0&page[limit]=1&filter[time_range][condition][since]=1678341600&filter[time_range][condition][until]=1678413600&filter[eq][condition][key]=plan_name&filter[eq][condition][value]="tune_ar"&filter[time_range][condition][timezone]=US/Central&sort=time&fields=metadata&omit_links=true
{
"data": [
{
"attributes": {
"ancestors": [
"idb_usaxs_retired_2023-12-05"
],
"data_sources": null,
"metadata": {
"start": {
"EPICS_CA_MAX_ARRAY_BYTES": "1280000",
"EPICS_HOST_ARCH": "linux-x86_64",
"beamline_id": "APS 9-ID-C USAXS",
"datetime": "2023-03-09 00:04:36.101637",
"detectors": [
"PD_USAXS"
],
"epics_libca": "/home/beams11/USAXS/micromamba/envs/bluesky_2023_1/lib/python3.10/site-packages/epics/clibs/linux64/libca.so",
"hints": {
"dimensions": [
[
[
"a_stage_r2p"
],
"primary"
]
]
},
"login_id": "[email protected]",
"motors": [
"a_stage_r2p"
],
"num_intervals": 30,
"num_points": 31,
"pid": 1521928,
"plan_args": {
"args": [
"UsaxsMotorTunable(prefix='9idcLAX:pi:c0:m1', name='a_stage_r2p', parent='a_stage', settle_time=0.0, timeout=None, read_attrs=['user_readback', 'user_setpoint'], configuration_attrs=['user_offset', 'user_offset_dir', 'velocity', 'acceleration', 'motor_egu', 'width'])",
43.48869216238651,
55.48869216238651
],
"detectors": [
"myScalerCH(prefix='9idcLAX:vsc:c0', name='scaler0', read_attrs=['channels', 'channels.chan04', 'channels.chan04.s', 'time'], configuration_attrs=['channels', 'channels.chan01', 'channels.chan01.chname', 'channels.chan01.preset', 'channels.chan01.gate', 'channels.chan04', 'channels.chan04.chname', 'channels.chan04.preset', 'channels.chan04.gate', 'count_mode', 'delay', 'auto_count_delay', 'freq', 'preset_time', 'auto_count_time', 'egu'])",
"UsaxsMotorTunable(prefix='9idcLAX:pi:c0:m1', name='a_stage_r2p', parent='a_stage', settle_time=0.0, timeout=None, read_attrs=['user_readback', 'user_setpoint'], configuration_attrs=['user_offset', 'user_offset_dir', 'velocity', 'acceleration', 'motor_egu', 'width'])"
],
"num": 31,
"per_step": "None"
},
"plan_name": "tune_a2rp",
"plan_pattern": "inner_product",
"plan_pattern_args": {
"args": [
"UsaxsMotorTunable(prefix='9idcLAX:pi:c0:m1', name='a_stage_r2p', parent='a_stage', settle_time=0.0, timeout=None, read_attrs=['user_readback', 'user_setpoint'], configuration_attrs=['user_offset', 'user_offset_dir', 'velocity', 'acceleration', 'motor_egu', 'width'])",
43.48869216238651,
55.48869216238651
],
"num": 31
},
"plan_pattern_module": "bluesky.plan_patterns",
"plan_type": "generator",
"proposal_id": "testing",
"purpose": "tuner",
"scan_id": 440,
"time": 1678341876.142742,
"tune_md": {
"initial_position": 49.48869216238651,
"time_iso8601": "2023-03-09 00:04:36.118210",
"width": 12.0
},
"tune_parameters": {
"initial_position": 49.48869216238651,
"num": 31,
"peak_choice": "com",
"width": 12.0,
"x_axis": "a_stage_r2p",
"y_axis": "PD_USAXS"
},
"uid": "27a3c3d2-b768-40fc-bcba-bfa013196448",
"versions": {
"apstools": "1.6.10",
"area_detector_handlers": "0.0.10",
"bluesky": "1.10.0",
"databroker": "1.2.5",
"epics": "3.5.0",
"epics_ca": "3.5.0",
"h5py": "3.8.0",
"matplotlib": "3.6.3",
"numpy": "1.23.5",
"ophyd": "1.7.0",
"pyRestTable": "2020.0.7",
"pymongo": "4.3.3",
"spec2nexus": "2021.2.5"
}
},
"stop": {
"exit_status": "success",
"num_events": {
"baseline": 2,
"primary": 31
},
"reason": "",
"run_start": "27a3c3d2-b768-40fc-bcba-bfa013196448",
"time": 1678341888.2595327,
"uid": "00e9f178-2191-4ac5-a92d-d971fabd1d3b"
},
"summary": {
"datetime": "2023-03-09T00:04:36.142742",
"duration": 12.116790771484375,
"plan_name": "tune_a2rp",
"scan_id": 440,
"stream_names": [
"aps_current_monitor",
"baseline",
"primary"
],
"timestamp": 1678341876.142742,
"uid": "27a3c3d2-b768-40fc-bcba-bfa013196448"
}
},
"sorting": null,
"specs": null,
"structure": null,
"structure_family": "container"
},
"id": "27a3c3d2-b768-40fc-bcba-bfa013196448",
"links": null,
"meta": null
}
],
"error": null,
"links": {
"first": "http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/search/idb_usaxs_retired_2023-12-05?page[offset]=0&page[limit]=1",
"last": "http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/search/idb_usaxs_retired_2023-12-05?page[offset]=267&page[limit]=1",
"next": "http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/search/idb_usaxs_retired_2023-12-05?page[offset]=1&page[limit]=1",
"prev": null,
"self": "http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/search/idb_usaxs_retired_2023-12-05?page[offset]=0&page[limit]=1"
},
"meta": {
"count": 267
}
}http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/search/idb_usaxs_retired_2023-12-05?page[offset]=0&page[limit]=250&filter[time_range][condition][since]=1678341600&filter[time_range][condition][until]=1678413600&filter[eq][condition][key]=plan_name&filter[eq][condition][value]=%22tune_a2rp%22&filter[time_range][condition][timezone]=US/Central&sort=time&fields=metadata&omit_links=true&select_metadata={detectors:start.detectors,motors:start.motors}
{
"data": [{
"attributes": {
"ancestors": [
"idb_usaxs_retired_2023-12-05"
],
"data_sources": null,
"metadata": {
"detectors": [
"PD_USAXS"
],
"motors": [
"a_stage_r2p"
],
"plan_name": "tune_a2rp",
"time": 1678341876.142742
},
"sorting": null,
"specs": null,
"structure": null,
"structure_family": "container"
},
"id": "27a3c3d2-b768-40fc-bcba-bfa013196448",
"links": null,
"meta": null
}],
"error": null,
"links": {
"first": "http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/search/idb_usaxs_retired_2023-12-05?page[offset]=0&page[limit]=1",
"last": "http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/search/idb_usaxs_retired_2023-12-05?page[offset]=267&page[limit]=1",
"next": "http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/search/idb_usaxs_retired_2023-12-05?page[offset]=1&page[limit]=1",
"prev": null,
"self": "http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/search/idb_usaxs_retired_2023-12-05?page[offset]=0&page[limit]=1"
},
"meta": {
"count": 267
}
}To find path to the catalog, one needs to identify the URL structure (see at the beginning discussion on /api/v1/VERB/NOUN). Here is how it should be possible:
https://tiled-demo.blueskyproject.io/api/v1/search/?fields=&sort= This returns the list of catalogs on the server. This is json which contains as data catalog names and the only additional data are URLs to themselves.
Let's follow the bmm URL:
https://tiled-demo.blueskyproject.io/api/v1/metadata/bmm This returns as one data set info identifying structure family as container - and has URLs to self (same URL), search URL (https://tiled-demo.blueskyproject.io/api/v1/search/bmm) and full URL. Let's try to follow the search URL:
https://tiled-demo.blueskyproject.io/api/v1/search/bmm which returns information about ONE or just few branches here if this is part of path OR this returns lot more data sets, if we are already on data set level. For bmm it shows just one data set (container) which is /raw/
URL for search: https://tiled-demo.blueskyproject.io/api/v1/search/bmm/raw
URL to full : https://tiled-demo.blueskyproject.io/api/v1/node/full/bmm/raw
etc.
Now, let's try same for our idb_usaxs_retired_2023-12-05 catalog:
http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/search/?fields=&sort=
which provides URL to self: http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/metadata/idb_usaxs_retired_2023-12-05
which provides URL to search: http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/search/idb_usaxs_retired_2023-12-05
URL to full : http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/node/full/idb_usaxs_retired_2023-12-05, etc.
Following the search http://usaxscontrol.xray.aps.anl.gov:8020/api/v1/search/idb_usaxs_retired_2023-12-05 URL returns, however, many data sets since we are already on the data level. This is UGLY...
We can find when we get to data level and extract from that URL the "NOUN" which can be used by the code to access the data. This would make the URL to be dynamically configured for each catalog at the initialization time and need to be stored global string, instead of string constant as they are now.