store successful uploaded program tracks in housekeeping database

[RonanHiggins]

It would be useful for program track testing to store the requested track in the housekeeping database. This should be tagged with a program track name say "_". This feature would help with debugging uploaded program track and variations between the actual and commanded.

[RonanHiggins]

Format for program track hk packet, then a list of those or?

dict_structure = {
    "measurement": "program_track",
    "tags": {"program_track_tag": <timestamp>_<instrument>},
    "fields": {"azimuth": 1.0,"elevation": 1.0, "azimuth_velocity":0.1,"elevations_velocity":0.1,"flag":0},
    "time": <timestamp>
}

[rschaaf-aifa]

I can provide an HTTP endpoint in the HK system to take the data from the tcs. One option would be the monitor-telegraf service. The format which this service accepts is the following snipplet from e.g. https://github.com/ccatobs/housekeeping/blob/main/data/telegraf-host/etc/telegraf/telegraf.conf:

[[outputs.http]]
url = “http://monitor-telegraf:8070/”
timeout = “5s”
method = “POST”
data_format = “json”
json_timestamp_units = “1ms”
[outputs.http.headers]
Content-Type = “application/json”

The url should be configurable via environment variable.

[nolta]

I've checked in a hacky first draft of this. It'll send the following JSON to the XXX_PROGRAM_TRACK_UPLOAD_URL env var:

{
  "points": [
    {
      "t": "2024-07-17T17:44:03.879056215Z",
      "az": 110,
      "el": 60,
      "vaz": 0.8,
      "vel": 0,
      "azFlag": 1,
      "elFlag": 0
    }
  ]
}

[rschaaf-aifa]

Thanks, I will try to make this work tomorrow in Xanten.

[rschaaf-aifa]

Format must be

{'metrics': [{'fields': 'az': 30, 'el': 30, 'vaz': 0, 'vel': 1, 'azFlag': 0, 'elFlag': 0}, 'name': 'programtrack', 'tags': {}, , 'timestamp': 1721730110001}, ...]}

[nolta]

You can do the conversion in telegraf, e.g.:

[[inputs.file]]
    precision = "1ns"
    data_format = "xpath_json"
    xpath_native_types = true
    [[inputs.file.xpath]]
        metric_name = "'programtrack'"
        metric_selection = "/points/*"
        timestamp = "t"
        timestamp_format = "rfc3339"
        tag_selection = "../../tags/*"
        field_selection = "*[not(name() = 't')]"

[rschaaf-aifa]

Telegraf is not really involved here, I am just using the http endpoint which the telegraf components in the HK system are using. The monitor-telegraf service which is defined in XXX_PROGRAM_TRACK_UPLOAD_URL actually knows nothing about telegraf, but expects the format given above. So, please do me the favour and change the format of the data you send.
Thanks!

[rschaaf-aifa]

Instead of sending all requested points of the requested track at or before the start of the track, it would be better to send them either during the scan or at the end of the scan. This would give us the chance to handle the case of aborted scans properly. The tcs would either stop sending points to the HK system (if points are sent during the ongoing scan) or filter out requested points with timestamps after the scan is aborted (if points are sent at the end of the scan).