Possibility to switch from visual studio build tools to CMake

[jacopoabramo]

I would like to compile locally the core in order to experiment a bit with it. Since the repository is quite big, I started doing the newbie thing and checked on google if there's an automation tool to build CMakeLists.txt from existing vcxproj. Turns out, there is.

So I ran it on my fork and got an output for almost all vcxproj files, except for mmCoreAndDevices\SecretDeviceAdapters\Usb3CamHS\Usb3CamHS.vcxproj, which apparently doesn't exists and it produced the following output:

1.233151 155> ERR : C:\git\mmCoreAndDevices\SecretDeviceAdapters\Usb3CamHS\Usb3CamHS.vcxproj project file doesn't exist ... skipping

... but since it's labeled as "secret" I guess I shouldn't worry too much.

The flags where converted to $ENV{MM_<FLAG>}, for example:

1.252149 187> WARN L1 : file:/C:/git/mmCoreAndDevices/DeviceAdapters/JAI/JAI.vcxproj(87): ('Release', 'x64') : Unknown variable: $(MM_BOOST_LIBDIR), trying $ENV{MM_BOOST_LIBDIR}
1.253150 187> WARN L1 : file:/C:/git/mmCoreAndDevices/DeviceAdapters/JAI/JAI.vcxproj(87): ('Release', 'x64') : Unknown variable: $(MM_3RDPARTYPRIVATE), trying $ENV{MM_3RDPARTYPRIVATE}

Now I want to try and compile at least the demo camera somehow to see if this works. Honestly, the final goal would be to use nanobind and build an experimental binding which would not use swig.

Would that be a feasible replacement to the current build toolchain? Should there be something I need to take into consideration?

[tlambert03]

not that this answers the question here, but I was chatting with @jacopoabramo anyway today so briefly talked about this. I told him that @marktsuchida is the build guru and will ultimately have the best advice. Told him about meson and #418, and
pointed him to https://github.com/pymmcore-plus/pymmdevice which was an experiment in using pybind11 (predecessor to nanobind) to do something very similar.

[marktsuchida]

Thanks, @tlambert03.

@jacopoabramo For MMCore, you can use the Meson build added in #416. It is not official yet but is maintained (see the GitHub CI), and could probably be a starting point for experimenting with new bindings. See the pymmcore meson branch for how the SWIG bindings can be built.

Are you aiming to create bindings for MMCore, or directly the device adapters?

One thing I'd point out is that I would not rely on the interface between MMCore and MMDevice (that is, the functions that device adapter DLLs expose for use by MMCore) as a stable API (in contrast to the MMCore API called from pymmcore/MMCoreJ and the MMDevice API against which device adapters are written). We really need this to be the case in order to be able to evolve things given the other legacy constraints we are working under.

It might also be useful to know that device adapter binaries built under different compilers and build systems are usually compatible with MMCore as long as the device interface version matches (MMDevice is designed that way).

BTW I'm familiar with pybind11 but had not encountered nanobind -- this sounds nice. Curious to hear more about what you are aiming at.

[tlambert03]

I can weigh in a bit on what I think is the broader goal here. Similar to that little pymmdevice experiment i was doing before, I think we're all kinda curious to explore the space of what it could look like to use (the amazing) existing catalog of MMDevices without fully buying in to MMCore being the one and only manager of devices. i.e. using MM devices as a "library" in a broader context (I know @marktsuchida and I have also talked about that being a nice potential goal, in parallel with providing the full on MM user experience).

One thing I'd point out is that I would not rely on the interface between MMCore and MMDevice (that is, the functions that device adapter DLLs expose for use by MMCore) as a stable API (in contrast to the MMCore API called from pymmcore/MMCoreJ and the MMDevice API against which device adapters are written).

I mentioned something like this to @jacopoabramo from my own experiments. That CoreCallback object is problematic, and really it does seem much safer to just let MMCore be the only thing that talks to MMDevices. I think I personally tend to forget how easy it is to just use a real MMCore object to control MMDevices, even if you intend for it to only be "one of multiple" device managing objects running in the process.

We also talked about SWIG being a relatively important binding when we're thinking about maintaining a single interface across both python and Java, but pybind11/nanobind being attractive experimental side wrappers for python-only targets. I have to say i really liked my brief experiments with pybind11 (and nanobind is the same API), and since the Core API doesn't really change all that often, what you lose by not having the SWIG autoconversion isn't that big of a deal.

just some random thoughts that can be ignored :)

[marktsuchida]

existing catalog of MMDevices without fully buying in to MMCore being the one and only manager of devices

As I hopefully mentioned in the past to you, I really do sympathize with this. Perhaps in the long term it can be achieved by gradually shifting MMCore to be more layered, with lower-level wrappers that can then be reused elsewhere. There may be other possible ways to reach the same goal. All I wanted to point out is that MMDevice as it currently stands does not feel like the right interface to use as a a stable API.

I fully support experimenting with alternatives to SWIG, for Python. Not sure what the compatibility story would look like (though perhaps easier to deal with in Python than it would be for Java), but in terms of maintainability and evolvability, pybind11 (and I assume nanobind) would be miles ahead. There could be a tradeoff where compatibility is less critical. And auto-generation can be scripted as needed.

[jacopoabramo]

The main target of my train of thought is what @tlambert03 already suggested. An initial approach in order to not modify the codebase at all would be divided in the following steps:

1. wrap MMCore API with nanobind; @marktsuchida if you're experienced with pybind11 you'll be happy to know that nanobind is shipped directly via pypi and apparently it's also pretty fast in compilation and also in performance tests (although the toy examples are far from this repo's complexity); it comes with a couple of caveats such as it forces you to use the scikit-build backend;
2. if point 1 applies, create a C++ container of cores (a core of cores... ugh sounds awful to the hear) which provides some sort of magic multithreading support in case multiple cores are created and having some safety measures in general.

Point one would allow to have multi-camera support without relying on the MultiCamera device adapter; point 2 is just to make sure that nothing weird happens when for example a new core tries to import an already loaded DLL.

Honestly my goal is to get rid of the SWIG interface, since - as already stated by @tlambert03 - I just want the pure python/C++ interface.

To be clear, this is not a PR proposal: my fork would be only keeping track of the main fork and synchronize whenever new changes are triggered; it would probably remove the MMCoreJ because of the point I stated before but it will never apply factual changes to the code base. Factually, it's just a way to expand pymmcore (and pymmcore-plus by reflection) to provide more fine-grained control over the existing hardware.

[jacopoabramo]

Thinking about it, using pymmcore-plus makes point 2 totally moot since it's already well enstablished. And with the existence of python free multithreaded, all multithreading checks can be done in python directly.

[jacopoabramo]

At any rate I will answer to my own question after the discussion I had with @tlambert03 : SWIG can't be replaced in the main fork since it provides compatibility with MMCoreJ so there's that.

[marktsuchida]

Point one would allow to have multi-camera support without relying on the MultiCamera device adapter; point 2 is just to make sure that nothing weird happens when for example a new core tries to import an already loaded DLL.

I don't seem to have good intuition for how this can be done from outside of MMCore code, but intriegued! Or do you mean more like having a modified MMCore that you then wrap with nanobind?

forces you to use the scikit-build

Oh, slight bummer. But it sounds like they are not hostile to the use of build systems other than CMake. Also it's probably not that bad to consume (the headers and binaries from) a Meson project in a CMake one.

To be clear, this is not a PR proposal: my fork would be only keeping track of the main fork and synchronize whenever new changes are triggered; it would probably remove the MMCoreJ because of the point I stated before but it will never apply factual changes to the code base. Factually, it's just a way to expand pymmcore (and pymmcore-plus by reflection) to provide more fine-grained control over the existing hardware.

Sure, but let us know how it goes. Would be nice to improve MMCore itself where possible, even if it takes some work on our side.

[jacopoabramo]

Alright, this is all current speculation in relation to what I learned over the past few years about MMCore and it's api, so I apologize in advance in my speculations are somewhat faulty.

Normally, one initializes the MMCore via setting the search path for where to find the device adapters, then calling loadSystemConfiguration. This is the standard procedure if I'm correct, right?

Now, I attempted and succeeded to use the core by just loading a specific device adapter via loadDevice and initDevice, so skipping the full system configuration. I did this in a separate fork of ImSwitch which is currently used to control the setup I worked on in my lab. It works after making some more robust error management on the side of ImSwitch.

So... why not chopping off existing configuration files, let them be handled by separate cores, and synchronize multiple cores? It could also be arranged that a core is only dedicated to a specific range of devices but I would not force it.

I might be missing some assumptions over which I don't have knowledge, but it seemed to me like a feasible approach to provide basically guaranteed backward compatibility with existing device adapters. The extra synchronization class on top of it might be necessary to ensure some threading mechanisms over which I'm not familiar with, or any other things that I might not be taking into account.

[jacopoabramo]

To be frank the one thing I need is a crash course in CMake but for that I'll have to call upon the God Emperor of Mankind (aka google). And some time investment in it.

EDIT: Come to think of it, I need a crash course in toolchains because other than downloading Visual Studio and setting up some small projects I don't have much experience with it... I linger in my own shame

[tlambert03]

Normally, one initializes the MMCore via setting the search path for where to find the device adapters, then calling loadSystemConfiguration... Now, I attempted and succeeded to use the core by just loading a specific device adapter via loadDevice and initDevice, so skipping the full system configuration

correct, you can absolutely use an MMCore object without ever worrying about a config file. You an actually think of a config file as a literal, procedural, line-by-line set of commands, each of which would have a corresponding public MMCore API call. (see here for actual logic)

[jacopoabramo]

As for the synchronization, perhaps it could mostly work if the idea is to prevent application threads from calling into more than one CMMCore at the same time. The potentially problematic cases are where something internal causes the Core to call a device: for example, when a camera finishes a sequence acquisition and autoshutter is enabled, the shutter is closed -- though I don't know if this particular case is actually going to cause problems.

Could you maybe try to elaborate a bit more on this point? Identifying possible bottlenecks over which I have no prior knowledge is my priority and I might not be considering corner cases like the one you mentioned.

Do you mean that some device adapters are independent from each other in some capacity?

[marktsuchida]

Sure. MMCore and the device adapters started out without support for multithreading (in the very early days). A lock (mutex) was then retrofitted to protect device adapters so that no 2 threads can call into the same device adapter at the same time. For whatever reason (perhaps no reason), this lock is per-device adapter but also per CMMCore instance: if two CMMCore instances talk to the same device adapter (even to different devices), there is nothing to protect them from the application calling functions for the two devices from different threads. Some devices may not care (for example, I'd expect DemoCamera to work), but in general this can lead to misbehavior and crashes in the device adapter, since device adapters are not written (much less tested) to work on multiple threads.

So if we want clean support for 2 or more CMMCore instances, talking to an overlapping set of device adapters (e.g., two cameras of the same make and model), we'd need to restructure the internals of MMCore such that this mutex is shared between all CMMCore instances. (I don't think such a change is impossible but haven't completely thought it through.)

I interpreted your idea (not sure if correctly; please correct me if not) as wrapping all access to MMCore from the application side with a mutex or equivalent (let's call this the global core lock or GCL :), which would prevent any problems when calling from multiple application threads. The edge case I mentioned is one in which the action is initiated independenly of application threads (actually it usually runs on a thread created by the camera device adapter) and therefore will not and cannot hold the GCL, though I can't think of any other such examples off the top of my head. And this particular case may not cause any issues if the shutter comes from a different device adapter that is only loaded in one CMMCore.

(Holding a GCL during every call into MMCore (from the app) could however be a severe bottleneck for at least some usage patterns. I don't think it is an architecture that I'd like to have in the long term.)