Best practices for distributing MEX files with toolboxes

[imciner2]

Some toolboxes are wrappers around C/C++ code compiled in MEX files. Currently, there isn't really a good set of documentation/practices for how to integrate these platform-specific MEX files into a packaged toolbox that is suitable for hosting on File Exchange. It would be good to add this to the best practices document.

[rpurser47]

Absolutely! We definitely want to offer recommendations around building and distributing toolboxes that include MEX. They are a little tricky since they need to be built on all platforms that your toolbox supports.

[imciner2]

They are a little tricky since they need to be built on all platforms that your toolbox supports.

Yep, building the platforms can be tricky.

I think that is slightly compensated for now since there are the Matlab runners on GitHub actions. Those would be a good way to build GitHub-hosted toolboxes in a CI environment (and that is what I am going to look into doing).

[rpurser47]

I think that is slightly compensated for now since there are the Matlab runners on GitHub actions. Those would be a good way to build GitHub-hosted toolboxes in a CI environment (and that is what I am going to look into doing).

Yes -- I'm hoping to put together an example that does just this in the future.

[alexander-hagerman-bose]

Just wanted to check in and see if this was still on the horizon?

[DavidRConnell]

I've been working on a toolbox for wrapping a C library for use in MATLAB, following along with the guidelines here and I think I've got a good start on distributing. There are a few walls, however, that I do not think are possible to get past at the moment for uploading to the file exchange or running in MATLAB online.

My project, matlab-igraph, wraps the igraph library. It uses CMake for building all the components and github actions for building on different OSes and releasing pre-compiled toolboxes. The results should be independent of any non-standard external dependencies so I believe all that is required to run them is downloading the correct package, unzipping, and adding to MATLAB's path.

Some of the problems I have run into:

1. On github's side, arm based macOS runners are not available without paying at the moment, so as of now I cannot provide a package for that system.
2. According to the official create and share toolboxes doc, the file exchange will not accept any toolboxes with mex files (so nothing I can do there).
3. The The github release section of the file exchange doc says it either downloads a packaged toolbox (mltbx file) from the release or the raw source. We would need this to be able to accept multiple toolboxes (one for each system supported). It looks like matlab.addons.toolbox.packageToolbox can be used to automate generation of the toolboxes. Should be possible to use this with ToolboxOptions to add a buildtool target to create for each OS in a github action.
4. On linux, MATLAB ships with an older version of stdlibc++.so.6. This leads to runtime symbol resolution issues. It can be fixed by using an older version of gcc, linking against MATLAB's version instead of the system version, or removing the version that is shipped with MATLAB so MATLAB finds the system version at runtime. For me, I couldn't get igraph to compile against the older version so I'm stuck with the last solution. In the github runner, however, it's not possible to delete that lib from the MATLAB install, as a result a bunch of my linux tests fail.

Overall it's close to working. On MathWorks side, we would need them to allow mex files and be able to upload OS specific toolboxes to get it to work on the file exchange.

[DavidRConnell]

I have modified my actions workflow to distribute mltbx files instead of tarballs using the package toolbox command*. I can confirm the actions built toolbox installs and runs fine on my linux computer. It also does run fine in MATLAB online (with exception to the above mentioned issue with stdlibc). File Exchange has accepted the toolbox. File Exchange only grabs the first toolbox it finds though, so it should not be possible to install correctly on another OS from the toolbox add-ons interface. At the moment the Windows build is not working but that's an issue on my side related to needing to package DLLs.

If MATLAB were to provide a method for allowing File Exchange packages to be associated with multiple toolboxes (and a naming convention for distinguishing between architectures), I think everything else is ready for distributing toolboxes dependent on MEX files. We could also just dump all the compiled files into a single toolbox and let MATLAB select the architect specific file at runtime but that increases the size of the toolbox.

*For some reason on macOS the package command failed with a permissions error related to not having a license for the MATLAB_Distrib_Comp_Engine, which is for running MATLAB on clusters, I can't understand why that license would have been needed. But that's the reason everything gets packaged at the end in a linux runner.

[SpieringsAE]

@DavidRConnell
I'm trying to get something like this working aswell, my approach is distributing the the c code and then compiling it when the package is installed, however I am encountering an issue with the postInstall.m file. It doesn't want to run.
I can see that it is included in the .mltbx, but when I install it, it is instant and it didn't run my compile script.

Do you or anyone else have an example of a post install script?

[SpieringsAE]

I managed to get something working, because my toolbox is a simulink blockset I was able to set a callback function when the library is loaded, in this callback function I can compile my mex files, this is my compilation script.

[DavidRConnell]

@SpieringsAE I don't remember the exact details but I came to the conclusion there wasn't a way to automate running a postinstall script when the user install the toolbox (although I may be wrong). This post-install hook answer thread looked promising but there didn't end up being a working solution. This may be intentional as it would be hard to prevent someone from adding malicious code to a postinstall script so you don't want to let someone run their code when a user clicks install.

My not-so-great solution was to provide a postinstall function in the toolbox and try to get users to run it manually. The function then ends with deleting itself. I gave up on finding a better solution as it's not that imporntant for my toolbox, it just cleans up some files that don't bother anyone if they remain.

In the past, when I have rewritten MATLAB code into C, I usually keep the mex function private then have a public MATLAB function that calls the mex function if available. Then the MATLAB function can check if the mex function is compiled and if not can handle that in someway such as warning the user they can compile it or running the mex build command itself. So an alternative to postinstall would be having users run a make all function, but if they don't, build it for them the first time they try to use a mex function.

[SpieringsAE]

@DavidRConnell Yeah I found that one too, I did consider providing a function like that, but because mine are level 2 s functions ment for use with simulink code generation it was easy to include in a library startup callback.

It also makes it impossible to provide a matlab script shim inbetween.

The only annoying part of my method is that it will run every time simulink is opened, so I built a little mechanism that checks if it has been compiled already so it exits pretty much instantly if it doesn't need to do anything.

[rpurser47]

The only thing I'll add here is that buildtool now supports incremental builds, and you could use a singleton to track if the build has been done, and check at the entry point of your various MATLAB functions.

[rpurser47]

We are actively working on MEX advice for toolboxes. Please stay tuned!