Aerotech motors do not work reliably enough for seamless bluesky integration

[APRashedAhmed]

Expected Behavior

Well-formed bluesky plans work consistently

Current Behavior

The problem appears as the motor(s) going into a faulted state of some kind at some point during a plan. It was observed when running the global homing and parking plans. Most of the issues can be remedied by doing the following:

1. Adding waits between PV puts or gets

   • .ready_motor() performs a series of gets to find out the state of the motor, and then some puts to enable, clear, and set the motor to go. This seemed like it was being done too quickly for the motor so adding a sleep each call seemed to help
   • Adding sleeps between .ready_motor() calls also seemed to help

2. Only moving 2-4 motors at a time

   • Sending commands to more than 2-4 motors at a time causes the later motors to fault or for the command to be dropped entirely
   • Mike later confirmed that there is a maximum queue length of 4 that the controllers can keep track of, and that it will simply drop commands issued when it is full

With these accomodations, the parking plan will complete execution ~80% of the time. However, for the successful plans, it is not clear if commands were dropped without directly checking the positions of the motors, and certain motors (diagnostics and L) randomly fail every now and then.

Possible Solution

I found out about the maximum queue length after I no longer had the ability to test the scripts. Batching the move commands according to which controllers they are on should fix the dropped command problem but it may also solve the seemingly randomly faulting motors.

Steps to Reproduce (for bugs)

1. Create a plan loops through the snd motors and moves them
2. Watch all the motor positions to see if they reach their destination
3. Watch for motor faults during the plan

Context

This has made higher-level plans such as the global homing and parking very problematic.

[ecating]

I'm looking at using Ophyd/Bluesky with Aerotech stages as well. I looked briefly through your code base and I've got a question for you, if you don't mind. When you call your initialization function:

def init(self, prefix, name=None, *args, **kwargs):

what do you provide as the prefix? I've been searching for this everywhere and cannot figure it out. Do you give it a Epics PV, and if so, how do you get that value?

Thanks!

[ZLLentz]

Hi @ecating, the prefix arg for ophyd devices is a string that is a partial EPICS PV, and exactly what string you need to pass is up to whoever wrote the device class. It's usually the largest "common" chunk among all PVs relevant to the device. In the case of EpicsMotor, you need to pass the motor record base PV (the one that is data type "motor").

It's possible that this isn't documented well on many classes, in this case you need to check what suffixes are being used in the class definition to figure it out. See the class definition of EpicsMotor:
https://github.com/bluesky/ophyd/blob/3cbf73b8f71b9c2494ffb097ac1a5e9b5c0f1e23/ophyd/epics_motor.py#L44

We see the suffix of the readback PV is just .RBV, so that means the prefix is everything before the .RBV. So if we get our readback PV using MOTORNAME.RBV, then our prefix is just MOTORNAME