Speeding up ophyd device load times

[tangkong]

[A part of the 2022 lightpath fix campaign]

Context

A large portion of lightpath load time is used to load every relevant ophyd device. This is a problem in not only lightpath, but also other lcls apps like hutch-python

Ideas / Approaches

• look at which devices load slowest and see if they can be made lighter. Some devices take more than their fair share of time
• See if threading / multithreading can be used more efficiently

[klauer]

First pass at this:
https://github.com/bluesky/ophyd/pull/1052/files

Last I had done the "full happi database load test" this shaved off a few seconds from the 30+ seconds it took. (If I recall correctly; this was a while back!)

[ZLLentz]

I'll start whittling away at specific instances of slowness- devices that are defined in such a way that they take ages to load. I'm sure there are other instances of systemic slowness but I suspect there are a lot of things in the specifics category.

[ZLLentz]

proposing:

happi benchmark <optional search terms>:
    checks the startup time of all devices in the database or a specific subset of them

happi load --profile <search terms>:
    extend happi load for line profiler output

[klauer]

I like the thought of a separate happi benchmark/profile since it seems repurposing load would change its meaning a bit much in my opinion:

Usage: happi load [OPTIONS] [DEVICE_NAMES]...

  Open IPython terminal with DEVICE_NAMES loaded

[ZLLentz]

I'm happy to make it separate. I was thinking that maybe someone would want to load and then "do something" to the device and catch that in the profiler, but maybe that's not very common.

[klauer]

I was thinking that maybe someone would want to load and then "do something" to the device and catch that in the profiler, but maybe that's not very common.

That's a very good point - and enough to change my mind. To restate: --profile wouldn't just be profiling the load/startup time, it could be profiling any arbitrary thing you're doing with the device.

Going the happi load --profile route, were you thinking of a flag of sorts to allow just profiling the load time without an interactive prompt? With some provided script?

Edit: or perhaps I misinterpreted your original suggestion. If you were suggesting both - and not just one/the other - that would make more sense now.

[ZLLentz]

I was suggesting one or the other but clearly we want both. I think I'll start with separate commands and leave an extension to happi load for a future update- I'll almost certainly drop myself right into the rabbit hole of fixing devices once the benchmark/profile is set up.

For happi profile, I notice that happi does not depend on pcdsutils- it's tempting to add this dependency here.