second run of acquire(events) ends in freeze of MM with no images taken; specific to using "DA Z Stage" as Core-Focus #194
Comments
|
I was not able to reproduce this on Windows or OSX using the demo configuration. My thought is that maybe this is a consequence of the specific Z-drive you are using? Do you see the same behavior when using the micro-manager demo configuration? |
|
Thanks for your quick response! Using the demo config it seems to work fine. My Z-drive is a simple DA Z Stage (voltage-controlled piezo) and it has no trouble using MM's Multi-D Acq. Plus, it works fine in the first run. Here are two more symptoms that may be related and may help the diagnosis:
|
|
I have some more info. My system happens to have another Thorlabs motor stage that can function as a focus stage. So I set that as the Core-Focus device and then the same script has no trouble repeatedly running. So it certainly looks like a DA Z Stage-specific issue. To be more specific, my DA Z Stage is using a NI-DAQ device's analog-output line that connects to the piezo's input. Again, this hasn't had any issue running with MM's Multi-D. |
linshaova
changed the title
linshaova
changed the title
|
Okay, good testing work! It seems like its either a problem with that specific device adapter or the acquisition engine of pycro-manager failing to reset something. I want to figure out a bit more specifically what your hardware is doing. Can you run the following with the code below and let me know what it prints? def hook_fn(event):
print(event)
return event
with Acquisition(directory=save_dir, name=save_name, pre_hardware_hook_fn=hook_fn) as acq:
events = multi_d_acquisition_events(num_time_points=4, z_start=0, z_end=20, z_step=1)
acq.acquire(events) |
|
Seems like not much useful info: [{'axes': {'z': 0, 'time': 0}, 'z': 0, 'properties': []}, {'axes': {'z': 1, 'time': 0}, 'z': 1, 'properties': []}, {'axes': {'z': 2, 'time': 0}, 'z': 2, 'properties': []}, {'axes': {'z': 3, 'time': 0}, 'z': 3, 'properties': []}, {'axes': {'z': 4, 'time': 0}, 'z': 4, 'properties': []}, {'axes': {'z': 5, 'time': 0}, 'z': 5, 'properties': []}, {'axes': {'z': 6, 'time': 0}, 'z': 6, 'properties': []}, {'axes': {'z': 7, 'time': 0}, 'z': 7, 'properties': []}, {'axes': {'z': 8, 'time': 0}, 'z': 8, 'properties': []}, {'axes': {'z': 9, 'time': 0}, 'z': 9, 'properties': []}, {'axes': {'z': 10, 'time': 0}, 'z': 10, 'properties': []}, {'axes': {'z': 11, 'time': 0}, 'z': 11, 'properties': []}, {'axes': {'z': 12, 'time': 0}, 'z': 12, 'properties': []}, {'axes': {'z': 13, 'time': 0}, 'z': 13, 'properties': []}, {'axes': {'z': 14, 'time': 0}, 'z': 14, 'properties': []}, {'axes': {'z': 15, 'time': 0}, 'z': 15, 'properties': []}, {'axes': {'z': 16, 'time': 0}, 'z': 16, 'properties': []}, {'axes': {'z': 17, 'time': 0}, 'z': 17, 'properties': []}, {'axes': {'z': 18, 'time': 0}, 'z': 18, 'properties': []}, {'axes': {'z': 19, 'time': 0}, 'z': 19, 'properties': []}, {'axes': {'z': 20, 'time': 0}, 'z': 20, 'properties': []}, {'axes': {'z': 0, 'time': 1}, 'z': 0, 'properties': []}, {'axes': {'z': 1, 'time': 1}, 'z': 1, 'properties': []}, {'axes': {'z': 2, 'time': 1}, 'z': 2, 'properties': []}, {'axes': {'z': 3, 'time': 1}, 'z': 3, 'properties': []}, {'axes': {'z': 4, 'time': 1}, 'z': 4, 'properties': []}, {'axes': {'z': 5, 'time': 1}, 'z': 5, 'properties': []}, {'axes': {'z': 6, 'time': 1}, 'z': 6, 'properties': []}, {'axes': {'z': 7, 'time': 1}, 'z': 7, 'properties': []}, {'axes': {'z': 8, 'time': 1}, 'z': 8, 'properties': []}, {'axes': {'z': 9, 'time': 1}, 'z': 9, 'properties': []}, {'axes': {'z': 10, 'time': 1}, 'z': 10, 'properties': []}, {'axes': {'z': 11, 'time': 1}, 'z': 11, 'properties': []}, {'axes': {'z': 12, 'time': 1}, 'z': 12, 'properties': []}, {'axes': {'z': 13, 'time': 1}, 'z': 13, 'properties': []}, {'axes': {'z': 14, 'time': 1}, 'z': 14, 'properties': []}, {'axes': {'z': 15, 'time': 1}, 'z': 15, 'properties': []}, {'axes': {'z': 16, 'time': 1}, 'z': 16, 'properties': []}, {'axes': {'z': 17, 'time': 1}, 'z': 17, 'properties': []}, {'axes': {'z': 18, 'time': 1}, 'z': 18, 'properties': []}, {'axes': {'z': 19, 'time': 1}, 'z': 19, 'properties': []}, {'axes': {'z': 20, 'time': 1}, 'z': 20, 'properties': []}, {'axes': {'z': 0, 'time': 2}, 'z': 0, 'properties': []}, {'axes': {'z': 1, 'time': 2}, 'z': 1, 'properties': []}, {'axes': {'z': 2, 'time': 2}, 'z': 2, 'properties': []}, {'axes': {'z': 3, 'time': 2}, 'z': 3, 'properties': []}, {'axes': {'z': 4, 'time': 2}, 'z': 4, 'properties': []}, {'axes': {'z': 5, 'time': 2}, 'z': 5, 'properties': []}, {'axes': {'z': 6, 'time': 2}, 'z': 6, 'properties': []}, {'axes': {'z': 7, 'time': 2}, 'z': 7, 'properties': []}, {'axes': {'z': 8, 'time': 2}, 'z': 8, 'properties': []}, {'axes': {'z': 9, 'time': 2}, 'z': 9, 'properties': []}, {'axes': {'z': 10, 'time': 2}, 'z': 10, 'properties': []}, {'axes': {'z': 11, 'time': 2}, 'z': 11, 'properties': []}, {'axes': {'z': 12, 'time': 2}, 'z': 12, 'properties': []}, {'axes': {'z': 13, 'time': 2}, 'z': 13, 'properties': []}, {'axes': {'z': 14, 'time': 2}, 'z': 14, 'properties': []}, {'axes': {'z': 15, 'time': 2}, 'z': 15, 'properties': []}, {'axes': {'z': 16, 'time': 2}, 'z': 16, 'properties': []}, {'axes': {'z': 17, 'time': 2}, 'z': 17, 'properties': []}, {'axes': {'z': 18, 'time': 2}, 'z': 18, 'properties': []}, {'axes': {'z': 19, 'time': 2}, 'z': 19, 'properties': []}, {'axes': {'z': 20, 'time': 2}, 'z': 20, 'properties': []}, {'axes': {'z': 0, 'time': 3}, 'z': 0, 'properties': []}, {'axes': {'z': 1, 'time': 3}, 'z': 1, 'properties': []}, {'axes': {'z': 2, 'time': 3}, 'z': 2, 'properties': []}, {'axes': {'z': 3, 'time': 3}, 'z': 3, 'properties': []}, {'axes': {'z': 4, 'time': 3}, 'z': 4, 'properties': []}, {'axes': {'z': 5, 'time': 3}, 'z': 5, 'properties': []}, {'axes': {'z': 6, 'time': 3}, 'z': 6, 'properties': []}, {'axes': {'z': 7, 'time': 3}, 'z': 7, 'properties': []}, {'axes': {'z': 8, 'time': 3}, 'z': 8, 'properties': []}, {'axes': {'z': 9, 'time': 3}, 'z': 9, 'properties': []}, {'axes': {'z': 10, 'time': 3}, 'z': 10, 'properties': []}, {'axes': {'z': 11, 'time': 3}, 'z': 11, 'properties': []}, {'axes': {'z': 12, 'time': 3}, 'z': 12, 'properties': []}, {'axes': {'z': 13, 'time': 3}, 'z': 13, 'properties': []}, {'axes': {'z': 14, 'time': 3}, 'z': 14, 'properties': []}, {'axes': {'z': 15, 'time': 3}, 'z': 15, 'properties': []}, {'axes': {'z': 16, 'time': 3}, 'z': 16, 'properties': []}, {'axes': {'z': 17, 'time': 3}, 'z': 17, 'properties': []}, {'axes': {'z': 18, 'time': 3}, 'z': 18, 'properties': []}, {'axes': {'z': 19, 'time': 3}, 'z': 19, 'properties': []}, {'axes': {'z': 20, 'time': 3}, 'z': 20, 'properties': []}] |
|
Everything printed in one big list, instead of one per line, which means the acquisition engine is trying to sequence over Z and time. I wonder if this is the behavior you want? In your setup, is your camera sending out TTL triggers that tell the Z stage to move to the next position? |
|
Yes I'm using TTL triggers from a Andor Zyla to tell the Z stage to move at each exposure, and it works with MM's Multi D. Acq. I'm not sure what "sequence over Z and time" means. Maybe the engine is doing that, but it's not like I knowingly did something for that to happen. All I wanted was to have pycro-manager reproduce what Multi. D could do. Thanks! |
|
"sequence over Z and time" just means that its using TTL triggers, and it is trying to do both a time series of z-stacks through TTLs without communicating with software in between successive time points, as opposed to running a TTL triggered Z stack, talking to software, and repeating. I'm not sure if this is any different from MM, but one thing to try would be explicitly breaking this up by calling Looking at the code just now, I notice is that there's no I compiled a new version. If you go here, you can download AcqEngJ-0.7.9.jar. Could you drop this into the plugins/micro-manager folder in your installation of MM, and get rid of the previous version to test? |
|
Thanks for the new compile! It behaves similarly as before: after the first successful run, subsequent runs of the same script would pop up the acquired-image window but nothing else happens and everything stops immediately, with "Finished" in the window title and a black image. An empty folder is created in the save folder. The only improvement is that nothing hangs and I can properly close MM and quit the iPython session. If I add a non-zero |
|
Ok so its definitely an internal thing in the acquisition engine. Can you check the core logs (in core_logs folder of your MM install directory) to see if it points to a specific error? If not, then 2 options are: 1) Try to reproduce with a minimal example making the calls to the core directly. 2) Run the acquisition engine form source and debug it (so I would need to remote control your system) |
|
Here are the relevant bits of the acquisition engine stripped out in Java code, but this can easily be translated to Python. This was before I added DoubleVector zSequence = new DoubleVector() ;
// Loop and add all z positions
zSequence.add(zPos);
core_.loadStageSequence(zStage, zSequence);
core_.prepareSequenceAcquisition(core_.getCameraDevice());
core_.startStageSequence(zStage);
while (core_.isSequenceRunning()) {
//wait
}
core_.stopSequenceAcquisition();
//N is number of images
core_.startSequenceAcquisition(N, 0, true);
//loop and call N times to throw away the images
core_.popNextTaggedImage();
|
|
Or maybe @nicost or @marktsuchida have ideas on what core calls are missing to properly clean up everything |
|
Not sure, but it would be useful to generate a trouble report (Help > Report a Problem). You can either post it here (very long!) or send it using the in-build functions, and I can have a look at it. Also @henrypinkard, in the code above, I would start the StageSequence after the camera Sequence has stopped. Otherwise, it is possible that triggers will be send to the stage that will move it to undesired positions. Does this acqEngine also incorporate Property Sequences? These are very useful for things like fast laser channel switching. |
|
Whoops I wrote that out of order. Whats actually going on is below: DoubleVector zSequence = new DoubleVector() ;
// Loop and add all z positions
zSequence.add(zPos);
core_.loadStageSequence(zStage, zSequence);
core_.prepareSequenceAcquisition(core_.getCameraDevice());
core_.startStageSequence(zStage);
//N is number of images
core_.startSequenceAcquisition(N, 0, true);
//loop and call N times to throw away the images
core_.popNextTaggedImage();
while (core_.isSequenceRunning()) {
//wait
}
core_.stopSequenceAcquisition();Is there anything obviously wrong here? Yes, its built to do property sequencing, but as far as I know no ones tested it so it might need more work |
Don't they need to be running at the same time? |
|
Yes, I thought you had code there to stop a previous camera sequence acquisition, which would have to happen before starting the stage sequence. What you have now looks correct (indeed, stage and property sequences need to start before the camera sequence starts). |
|
Hi @nicost , thanks for offering help. I sent you a bug report from MM; let me know if you receive it from lin.shao_at_yale.edu. In the bug report, it might be a bit confusing because the focus device, the DA Z Stage in my case, is referred to as Galvo. @henrypinkard I can understand what you want me to do with that script. There's some details I don't know how to handle in Python. For example, how do I create a |
|
Thanks for the pointers. I managed to write a Python script; see below. The same Java exception "Circular buffer is empty" occurred (error messages following the script). I still think some initialization step is missing. from pycromanager import Bridge
bridge = Bridge()
core_ = bridge.get_core()
nZ = 40
zStart = -20
zStep = 1
zSequence = bridge.construct_java_object('mmcorej.DoubleVector', args=[nZ])
for i in range(nZ):
zSequence.set(i, zStart + i*zStep)
zStage = core_.get_focus_device()
core_.load_stage_sequence(zStage, zSequence)
core_.prepare_sequence_acquisition(core_.get_camera_device())
core_.start_stage_sequence(zStage)
nT = 3
tot = nZ * nT
core_.start_sequence_acquisition(tot, 0, True)
for i in range(tot):
core_.pop_next_tagged_image() #exception occurs here
while core_.is_sequence_running():
pass
core_.stop_sequence_acquisition()Error messages: |
|
You just need the code to swallow the exceptions until images are available. Corrected version below. It works for me with demo configuration (after turning on sequencing in the demo z). from pycromanager import Bridge
bridge = Bridge()
core_ = bridge.get_core()
nZ = 40
zStart = -20
zStep = 1
zSequence = bridge.construct_java_object('mmcorej.DoubleVector', args=[nZ])
for i in range(nZ):
zSequence.set(i, zStart + i*zStep)
zStage = core_.get_focus_device()
core_.load_stage_sequence(zStage, zSequence)
core_.prepare_sequence_acquisition(core_.get_camera_device())
core_.start_stage_sequence(zStage)
nT = 3
tot = nZ * nT
core_.start_sequence_acquisition(tot, 0, True)
i = 0
while i < tot:
try:
core_.pop_next_tagged_image() #exception occurs here
i += 1
except:
pass
while core_.is_sequence_running():
pass
core_.stop_sequence_acquisition() |
|
Okay I changed the script as suggested. And what I got is the same as before: the first run goes through and the second run makes everything hang (both MM and iPython have to be forced quit). Thanks! |
|
Can you figure out on which line it hangs? |
|
Good reminder... It seems to hang at the line of |
|
Based on this I think this is a issue with your device adapter (i.e. not specific to pycromanager). Though I have no idea why this works with the MM MDA. There is perhaps something in there that's masking the problem with the device adapter. I think you should probably open an issue on the main micro-manager repository since thats where the device adapters live. Maybe you can also experiment with calling |
|
Thanks! I tried calling which is still not right but allows proper MM shutdown at least. More importantly, I can reproduce the same symptoms using a ~directly translated Beanshell script (attached here I'm also curious why MDA works. While I have no clue so far, I did notice a key difference between running MDA- and the pycroM-based acquisition: in the former, there's a noticeable pause (based on laser shutting) between time points even if the In any case, thanks for your patience with me and all the help with uncovering an issue unrelated to pycro-manager at all! I'll open this issue in MM's repository. |
|
You're welcome, glad you're able to get a little closer to figuring it out
I think this is almost certainly because I've yet to implement property sequencing in the acquisition engine. Should be straitforward. I'll let you know when I've got it in there. And if you feel inspired to delve into the confusing mess that is the MDA acquisition engine, its here |
|
Closing this for now. I made another issue for property sequencing: #207 |
Bug report
Bug summary
Code for reproduction
Expected outcome
This should run as many times as I ask it to
Actual outcome
Only the first run works; after that when I run it again, the image window pops up but everything freezes. Micro-manager can't even be closed normally; has to be force-closed from TaskManager.
Version Info
The text was updated successfully, but these errors were encountered: