Aug '22 ACU Agent LAT Updates (#403)

Make updates based on LAT + SAT testing

This commit is a squash of a large PR, see #403 on GitHub for details. The full commit list is included here below.

* Update _run_specified_scan to abort when stop_and_clear is called

_run_specified_scan now checks whether the number of free upload positions
is greater than 9997, in which case it posts a log warning and ends the Task.
This update is intended to catch cases where the stop_and_clear Task is used
to abort a scan, and prevents _run_specified_scan from continuing to upload
points after stop_and_clear is called.

* fix @ocs_agent.param decorators and uncomment

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* New parameter for scan start position in generator scans

Include new string parameter az_start to indicate the beginning point
of a generator scan. Defaults to starting scans from the midpoint between
the two az endpoint inputs.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add parameters to generate_scan process

Add parameters for all newer generate_scan variables to agent.

* Add defaults to params in generate_scan

No longer require user to provide all inputs for generate_scan. All
new options still exist, but now have default values.

* Check modes instead of free positions, limit hanging at end of _run_specified_scan

Remove checks for free positions to determine if stop_and_clear was called and
replace with checks for mode.

Check for 1s to determine settling at the end of the scan (needs future update
for velocity dependence)

* Keyword arguments update, fix typos

Create dictionary of optional params in generate_scan and ** method
of passing them.

Fix indentation in docstring, remove ptstack_fmt from docstring,
fix docstring typo

* Remove hanging "attempt number", bug fix for number of free positions at end of scan

Attempt to stop scan when rounding takes awhile removed because it caused
hanging when the scan actually reaches the end position.

Free upload positions at the end of the scan seems to be 9999 instead of 10000,
so updated the wait to look for this value.

* Only check for Boresight commanded position when using satp platform config

Because there is no boresight axis in the CCAT dataset

* generate_scan checks azimuth mode

generate_scan now checks whether the azimuth mode is ProgramTrack.
If not, it cleanly exits, which allows the user to override
generate_scan with stop_and_clear

* Add mode catch in go_to

go_to now looks for mode changes from Preset and exits the task if
the mode changes

* New process restart_idle

Sends a Datasets.CmdModeTransfer RestartIdleTime command to the ACU
every 4 min 55 s. This is a response to a LAT FAT safety feature.

* Make restart_idle startup=False

* new task preset_stop_clear

New task preset_stop_clear sets motion mode to Preset a few degrees
from the current position (velocity-dependent), waits for the platform
to settle, then changes to Stop and clears the stack (twice). This task
provides a smoother motion end than stop_and_clear, as requested during LAT
tests

* add upload points publishing to generate_scan

* ACUAgent: update generate_scan for LAT testing

Defaults for wait_to_start and step_time are changed.  New parameter
"num_scans" to produce finite scan.

* ACUAgent: a bunch of stuff from LAT testing

- Optionally perform additional read of special 3rd axis dataset.
- To support 3rd axis, more enum data is added to convert string -> int.
- RestartIdleTime default period is now 60s
- Slight change in how monitor frequency is measured.
- ProgramTrack point saving is commented out, probably temporarily

* ACUAgent: generate_scan handles num_scans parameter

* ACUAgent: generate_scan handles finite scans without crashing

* ACUAgent - improve generate_scan

It has an azonly mode now, and that is the default.  Bugs fixed in the
monitoring of point stack at the end of the process.

Also, a more complete "Mode" enum for influxdb integerification.

* ACUAgent: generate_scan has ramp_up param to bias the first leg

* ACUAgent: add more axis modes and renumber to match ICD

* ACUAgent: 3 more modes...

* ACU Agent updates: go_to, generate_scan, clear_faults

- In go_to, add some logging and an azonly option
- In generate_scan, clear the stack before and after scanning
- In generate_scan, fill the stack less so canceling scans takes less time
- Add new task clear_faults to clear out motion faults/errors

* Updates to monitor process session.data information

- On 'monitor' startup, record platform name in session.data
- After calculating the response frequency, record in session.data
- Create subdictionaries 'StatusDetailed' and 'Status3rdAxis' in session.data for recording values

* Update to monitor to log mode and remote changes and time problems

- monitor checks the previous value of ctime and determines if it has changed
- monitor checks the previous values of axis mode and determines if they have changed
- monitor checks whether the ACU is in remote mode and warns if it is not

* Motion tasks/processes check whether ACU is in remote mode and warn if not

* Motion tasks/processes check whether broadcast and monitor are running and warn if data points might be stale.

* Bug fix for set_boresight

Third axis is only named boresight for satp, so differentiate between
platforms in set_boresight.

* ACUAgent: convergence - comment out invalid feed post

* ACUAgent: convergence - flake8

* ACUAgent: convergence - imports and main func

* updates to set_boresight rounding for site testing

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* ACU: wait a beat after last program track point

* Remove extraneous print statements from acu_agent.py
Rebase onto mhasself/acu-convergence

* Change stop_and_clear call to acu_control.stop() instead of acu_control.mode()

This change sets all axes to Stop mode, instead of just Azimuth and Elevation.

* partial fix to logging issue in generate_scan

* Remove _try_set_job from agent.py

* add sh.plan_scan to agent function generate_scan

* add new function plan_scan to drivers

* allow for ramp-up to scans.

add plan_scan function to drivers to plan how far in advance the scan starts
add plan_scan usage to agent

* bug fix for scans with negative velocity

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* set_boresight checks for valid position range and exits the task if invalid.

go_to position range changed to < and > instead of <= and >=

* set_boresight checks modes and exits if the mode changes from Preset.

* go_to and set_boresight only set the axes moved by the task to stop if end_stop is True

* stop_and_clear tries 5 times to stop, then 5 times to clear

* stop_and_clear returns failure if stop or clear fails

* bug fix for set_boresight

Add dsleep(0.2) between sending the go_3rd_axis command and checking mode; task otherwise quits too early.

* correct typo in set_boresight that causes crash

* add session.set_status('running') at the top of tasks

* reduce number of points uploaded in generate_scan to make stops/aborts more responsive

* After creating the session.data dictionary in monitor, use session.data.update to update the values instead of remaking the dictionary

* go_to checks motion modes more frequently during motion to determine whether they have changed from Preset.

go_to also distinguishes between azonly and not azonly for mode checking

* - Add session.set_status('stopping') at the end of all operations
- Add abort function, unclear how to implement

* Add abort implementation for go_to and set_boresight

* Update locking to prevent multiple commands from being sent to the ACU before task completion

* Partial bug fix for broadcast influx feed

* Bug fix for broadcast

ensure that process_data always has len 200

* fix ordering in go_to that causes it to loop and never stay in stop mode

* bug fix for registering a task that can be aborted

* change aborter_blocking to False in task registration because _abort_motion_op uses @inlineCallbacks

* remove _set_job_done from all operations

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove print statement from set_boresight

* change _abort_agent_ops to perform preset_stop_clear and fix bugs in preset_stop_clear

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update broadcast session.data with the same data as influx data (1 Hz)

* change how strings are cut in broadcast session.data and remove print statement

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* create separate abort features and preset_stop_clear operations for azel and boresight

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add session.set_status and status check for fromfile_scan

* acu: fix whitespace and other flakes

* acu: fix vel~0 detection

* ACU: increase post-progtrack wait time a lot

* ACU: Suppress local mode log spam

* ACU: log when UDP stream is up/down

* ACU: check_daq_streams returns false on any trouble

* ACU: remove wait arg from go_to (removed in soaculib)

* ACU: disable seek/ramp_up processing temporarily

* ACU: work-around stalling scans

* ACU: go_to uses a state machine loop

Now that we understand the real behaviors of ACU better, it is
possible to write a simpler state-machine to handle the motions.

The rounding and azonly arguments have been removed, as they probably
aren't needed anymore.  The "wait" argument will be removed once
ocs-web stops passing it.

This implementation fixes a couple of bugs: abort works reliably; and
Preset mode isn't set without first setting the target position (which
is a potential problem in some cases, such as when the ACU has just
powered up).

* ACU: refine go_to states; add error checking

This is tested at site in a variety of modes, including cold start
(warning siren delay).

* ACU: move some motion readiness checks into a function

* ACU: generalize go_to state machine by axis

* ACU: set_boresight uses the generalized state machine

* ACU: smarter time-out on "motion expected", continue paring

* ACU: remove _set_job_stop in favor of _simple_process_stop

* ACU: fix process stop on monitor, broadcast, restart_idle

* ACU: separate locks for boresight and azel

* ACU: use session.status, not self.jobs, to check processes.

* ACU: remove self.jobs

* ACU: reduce data rate logging

* ACU: generate_scan is stoppable

Some stack maintenance stuff is hard-coded in number of points instead
of time... to be fixed next.

* ACU: tune point stack maintenance thresholds in generate_scan

* ACU: generate_scan can seek to starting point

Also sets wait_time and step_time optimally.

* ACU: in plan_scan rename some vars, add docs

* ACU: also accept az_start='mid','end'

* ACU: rename acc to az_accel

* ACU: param-decorate generate_scan

* ACU: review param descriptions in generate_scan

* ACU: rename set_boresight param to "target"

* ACU: docs include session.data.

Also add PlatformType to status session.data

* ACU: bugfix last commit

* ACUAgent: simplify from_file

* ACU: remove some commented upload feed stuff

* ACU: refactor generate_scan to call a generic _run_track

* ACU: fromfile_scan uses the _run_track helper.

* ACU: remove unused functions

* ACU: misc changes for PR

Mostly removing unused functions; reformatting of log message

* ACU: more cleanup for PR review

* ACU: fix timecode function end-of-year bugs

* ACU: move timecode function into drivers.py

* ACU: replace computation with timecode call

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Matthew Hasselfield <[email protected]>