RFC: Tracking progress to integrate with SciPy #112
Comments
|
Sounds great! Do you want to take a stab at adding it or would you like me to make an initial branch with We will want to publish our own Python package in addition to providing PRIMA via SciPy, so the bulk of this work ought to be usable for both use cases. We currently have both CMake and Make build systems that were contributed by two different people (@jschueller and @zaikunzhang, respectively). I think we will want to consolidate to at most 2 or better yet even 1, but that work can go on in parallel to the work necessary for SciPy or afterwards. |
Awesome, I will probably take a stab at makes for an easy onboarding task for me to start delving into the code.
If the idea is to provide bindings within PRIMA, then I wonder if it would be preferable (HiGHS does this) to have a C++ interface and use Pybind11. For SciPy alone I would have suggested (and was planning to implement) a Python-C handwritten extension, because much of the user interface comes from SciPy. The reason to prefer a C++ binding followed by Pybind11 over a pure Python-C extension is just that it would be easier to also integrate with other C++ libraries (it would be just a minor OOP wrapper over the C code). It will of course depend on the exact scoping of what level of interface is to be provided from
Generally |
|
I've used pybind11 before and I appreciate how easy it makes everything. I haven't made a Python-C extension so I can't really compare and contrast on that front, but we do generally want to provide bindings to as many languages as possible, which obviously includes C++. If you're suggesting using pybind11 for both the SciPy work as well as for our own Python packages I'm definitely on board, I think it would make things a lot simpler. |
Yup, Pybind11 is great, writing a Python-C extension by hand is just a lot more fragile and verbose.
Awesome, then the plan will be:
|
HaoZeke
changed the title
meson as a build system
Just wondering, since we both seem to like pybind11, why did you suggest Python-C extension in the first place? Are there some advantages to it compared to pybind11 or disadvantages of pybind11 compared to Python-C extension? I will start working on your last 3 bullet points. Callback support is implemented in a branch, it's just waiting for the options PR to merge since it's built on top of it. How long do you think it will take you to get the meson build going? Just looking for a rough idea here not necessarily an exact date. |
Well, for some projects, having an intermediate call through C++ might be too much of an overhead. In theory, there are probably some more micro optimizations possible if one writes hand-written Python-C code. Even Cython is yet another attempt to not write Python-C wrappers by hand just because of the same fragility. SciPy itself has some light Python-C wrappers (e.g. for
I expect it to be done by next Monday, I'll start it on Friday (I have limited time until the weekend). |
|
Fair point. I will revisit this over the weekend with basic wrappers in both form.
…--- Rohit
On 11/27/23 1:40 PM, Julien Schueller ***@***.***> wrote:
while pybind11 is great, I think it is overkill for prima because it
will be just one function now (#114
<#114> ), the ctypes wrapper is
quite simple and requires no compilation on top of the C interface, see
#52 <#52>
—
Reply to this email directly, view it on GitHub
<#112 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABBCUT6WKQVCWPVE4BNBWZLYGSJ4TAVCNFSM6AAAAAA72BOZVGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRXHA2TGNBTGY>.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
|
Hi, @jschueller , @nbelakovski , @HaoZeke , thank you very much for your precious contributions. Sorry for missing the discussion here. Nickolai @nbelakovski is visiting me in Hong Kong, and I have told him my opinions --- nothing technical (as I know almost nothing about the technology involved), just some general comments on how things should be organized (in the technical/coding sense). Many thanks! |
|
@HaoZeke I've implemented pybind11 bindings for To use it you'd do the following, assuming you are starting in the repo root (I can add these to a README if we go this direction) We could do either pybind11 or ctypes. I would tend to prefer something compiled since I think that means the compiler is doing some more checks for us. Also with pybind11 we can compile everything into a single shared library to distribute around, whereas with ctypes we need to distribute both the compiled library and the python file on top of it. |
|
Excellent, I will check it out ASAP. We will not be using ctypes since that is not compatible with SciPy (and indeed should never be used).
…On 12/1/23 3:35 AM, nbelakovski ***@***.***> wrote:
@HaoZeke <https://github.com/HaoZeke> I've implemented pybind11 bindings
for |newuoa| in a branch:
https://github.com/nbelakovski/prima/tree/python_binding
<https://github.com/nbelakovski/prima/tree/python_binding>.
To use it you'd do the following, assuming you are starting in the repo
root (I can add these to a README if we go this direction)
|mkdir build cd build cmake .. make -j8 export PYTHONPATH=$(pwd)/python
cd ../python/examples python example_newuoa.py |
We could do either pybind11 or ctypes. I would tend to prefer something
compiled since I think that means the compiler is doing some more checks
for us. Also with pybind11 we can compile everything into a single
shared library to distribute around, whereas with ctypes we need to
distribute both the compiled library and the python file on top of it.
—
Reply to this email directly, view it on GitHub
<#112 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABBCUT7CESJLG2JNB6SLUCTYHFF7XAVCNFSM6AAAAAA72BOZVGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZVGQYDEMBTGA>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@HaoZeke Just FYI I just pushed an update to that branch that includes the unified interface that was recently merged. I have 3/5 algorithms working through the python bindings, just need to do LINCOA and COBYLA and I hope to finish those this week. Once that's done I may start working on the meson stuff myself since we'll need it for publishing our own python package anyway. |
|
@HaoZeke We just merged the callback PR and I've made a lot of progress on the python bindings. It works for all 5 algorithms, I just have two outstanding items: COBYLA needs to be able to accept a list that can include LinearConstraint and NonlinearConstraint, and I'm seeing a GIL related issue that I need to debug. However, I'm having a really hard time getting the python bindings to compile correctly on Windows. It seems that the python module itself needs to be compiled with MSVC, which does not include support for Fortran. And CMake does not like mixing MSVC and gfortran, although I'm installing the intel compiler now so we'll see if that is any better. Do you have any advice or suggestions here? Do we need to use a specific compiler? Do these problems go away upon switching to Meson? |
|
yes it seems there could be a limitation with msvc not liking mixing c and fortran sources, but that's not specific to cmake, a simple solution is to split fortran and c libs |
|
I managed to find a solution by copying the compile/link flags used by SciPy. It turns out you can compile Python modules on Windows with GCC, you just have to link ucrt statically (and a couple other flags for C). I also wasn't sure what the right path was since we have so many Fortran compilers that we support, but after talking about it with @zaikunzhang we agreed that since we don't expect users to be building the Python bindings themselves (for the most part) we can just pick one compiler to support the Python bindings. I think gfortran makes sense since that's how SciPy is doing things on Windows (and I couldn't quite get ifx and MSCV to work :D ). |
|
Using the Intel compiler is probably not a good solution since that introduces distribution complexity. You want a FOSS compiler for this so that you can guarantee you're legally able to upload prebuilt binaries (e.g. python wheels).
The build system should really be agnostic to this, why do you need the path? |
Good point
I didn't mean path as in system path, I meant it as in "path forward" |
|
Ah, sorry for the confusion then. :P |
zaikunzhang
added
python
|
@HaoZeke Good news! The Python bindings are merged 🥳 I started working on the Meson setup here: https://github.com/nbelakovski/prima/tree/meson It doesn't include the python bindings yet, I was waiting until they were merged so that I didn't have to keep track of a branch on top of a branch. Do you have bandwidth to carry them through or should I keep going on them? |
As noted over at SciPy, I'll be able to spend some time ensuring the integration of PRIMA into SciPy via Python-C extension module. :)
The first step is to have a
mesonbuild backend. This is needed also because the goal is to useprimaas asubproject, e.g. as is done for HiGHs.The text was updated successfully, but these errors were encountered: