Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow for a more granular selection of poll period for certain signals #116

Open
jwlodek opened this issue May 29, 2024 · 10 comments
Open

Comments

@jwlodek
Copy link
Collaborator

jwlodek commented May 29, 2024

Currently, all the signals from the panda are polled every 0.1 seconds. Certain signals, however, can be more useful if they are polled faster. For example - if you want to use the PandA's logic to "trigger" a device that doesn't offer a hardware trigger/TTL option, you can do this right now with a CA link to one of the TTLOUT:OUT signals. However, in my experience this limits you to sub ~5 Hz trigger rates, since the PandA IOC polls everything at the same rate.

@AlexanderWells-diamond
Copy link
Contributor

I'm not sure we have the ability to do what is being asked. The issue is that, currently, we poll the PandA for all changes since the last time we asked, using the GetChanges function. There is an enum to control what values the PandA reports back, but it isn't particularly fine-grained. See the enum defined here. If the field(s) you are interested in rapidly polling are reported by one of those options, then we should be able to do faster polling for specific groups. Otherwise I don't think we can offer anything better than simply increasing the global polling rate.

@jwlodek
Copy link
Collaborator Author

jwlodek commented May 30, 2024

I think for my use case the most useful group to poll faster would be the BITS values: https://github.com/PandABlocks/PandABlocks-client/blob/c2c8c0b1d78f8317b18602e85984cab365b946d2/src/pandablocks/commands.py#L780, because I believe this includes all the block outputs that are presumably tied to your hardware via TTLOUT blocks.

@evalott100
Copy link
Contributor

Maybe we could add new command line args to specify different poll periods for specific change groups. @coretl thoughts?

@jwlodek
Copy link
Collaborator Author

jwlodek commented May 31, 2024

Just to see if I was hitting any kind of limitation in terms of the max scanning rate of the CA link mechanism from EPICS, I created a simple record that should produce a 50 hz clock signal (switches its value to the opposite every 0.01 seconds, meaning a rising edge every 0.02 seconds, 50 Hz):

record(calc, "$(P)$(R)Clock"){      
    field(CALC, "VAL = 1 ? 0 : 1")  
    field(SCAN, ".01 second")       
}                                   

And made a CA link to this instead of the PandA, and as expected I got things processing at 50 Hz. One thing to note is that the default regular scan rates supported by EPICS base is every 0.1 seconds. I image in your case the records are actually set to scan at I/O Interrupt level though, and the polling rate in the IOC is actually not related to the EPICS record scan rates, correct?

I don't think I'd need to go that fast, but it would be nice if we could get this to ~20 Hz let's say.

I think some kind of setup to specify poll periods for the different change groups would work for my case.

@coretl
Copy link
Contributor

coretl commented Jun 5, 2024

Some care is needed here:

# Its possible a bit_out field will be on for pulses shorter
# than our polling time. To represent this we will invert the
# value now, then set it back on the next update
if converted_value == record_info.record.get():
fields_to_reset.append(
(record_info.record, converted_value)
)
converted_value = not converted_value

We have a requirement to see pulse widths shorter than the poll period, so if the value has changes within the poll period then we toggle it and toggle it back again on the next poll. This means that a 100Hz will toggle the TTLOUT PV as fast as it can, which means 5Hz given the 0.1s *CHANGES period.

I am also not keen to go much faster than 10Hz because this is Python after all, and if it works most of the time but occasionally stutters then that will make something fail in the middle of an experiment.

Could we increase the update rate for *CHANGES to 20Hz? That would give 10Hz updates of bit_out PVs, and I don't think we should promise any faster than that.

SOLEIL do this by sending UDP triggers directly out of the FPGA, so maybe this is an idea worth exploring further. What kind of things are you looking to process on the EPICS side?

@jwlodek
Copy link
Collaborator Author

jwlodek commented Jun 5, 2024

For me, the use case is as follows:

I've written an areaDetector IOC that can read previously collected datasets either from an HDF file directly or via tiled endpoint. Once the data read back, it behaves like any other areaDetector, just with the catch that when it reaches the end of the dataset it either stops, or loops back to the front.

The main use case for this is to be able to run existing scans (or something very close to them) w/ real data, so that we can develop real-time analysis/processing software without requiring beamtime. Recently, I've upgraded this model to be able to also simulate hardware trigger support. This works by creating a CA link to a binary signal that is responsible for simulating the TTL signal. The playback IOC is smart enough to be able to be programmed to use either rising/falling edge, gated exposure, or gated acquisition. I have tested this with a simple CLOCK PV that just switches between 1 and 0 every 0.01 seconds, which produces a 50 Hz signal, and the playback runs at 50 Hz without issue.

What I am trying to do here, is basically have a setup so that we can run full panda based flyscan tests in the lab without access to the actual detector. There are plenty of PandA devices available, but it's a bit harder to have a Pilatus 2M available (and even harder to get it real x-ray data we can use for analysis). We do a lot of workflow/processing steps that happen on stop documents for example, and it'd be really nice to re-create the whole setup in the lab. I actually have this working fine now, with the catch that I get ~5 Hz as you said.

It's possible I can work around this by finding a device that offers a TTL input with a much faster polling rate available to an IOC, but I figured I'd see if this was possible at the PandA level.

@coretl
Copy link
Contributor

coretl commented Jun 6, 2024

As this is a single use case that is entirely under your control, could you open an asyn IP connection directly to the PandA TCP socket, and fire "*CHANGES.BITS?" at it as fast as you can from a thread in the playback IOC? That would skip the slow python level, and shouldn't put much extra load on the PandA TCP server...

@coretl
Copy link
Contributor

coretl commented Jun 6, 2024

Alternatively, you could do the same by connecting the playback IOC to the data connection in ASCII mode, and each time you get a new line then treat that as trigger:

$ nc 172.23.252.201 8889
NO_HEADER NO_STATUS
 0 0 0 0 0 0 6.4e-08 0.500000064 0 0 0
 0 0 0 0 0 0 1.000000064 1.500000064 0 0 0
 0 0 0 0 0 0 2.000000064 2.500000064 0 0 0
 0 0 0 0 0 0 3.000000064 3.500000064 0 0 0
 0 0 0 0 0 0 6.4e-08 0.500000064 0 0 0
 0 0 0 0 0 0 1.000000064 1.500000064 0 0 0
 0 0 0 0 0 0 2.000000064 2.500000064 0 0 0

@coretl
Copy link
Contributor

coretl commented Jun 6, 2024

That has the other benefit of letting you ensure that you have the right number of triggers, as you are guaranteed to get a new line for every PandA trigger, whereas *CHANGES could drop one (intentionally)

@jwlodek
Copy link
Collaborator Author

jwlodek commented Jun 6, 2024

That should work, will do some testing, and update back here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants