A better build system?

[erikd]

We are using this library at my work place and would like to provide you with a better, more standard build/install system.

I have previous experience with:

• autoconf/automake/libtool
• Cmake

I am also willing to look into other well known standard build systems like Meson if you would prefer.

Using a more common/standard build system improves the chances of your library being packaged by all the various Linux distributions as well as package management systems for Mac (like brew) and Windows (like chocolatey).

[dot-asm]

packaged by all the various Linux distributions

Well, it makes little sense, because blst is not really a general-purpose library. Besides, you're obviously talking about the C part, while the library value is rather in its bindings and those are handled within the corresponding build frameworks. Well, not all of them really, and that's where contributions would be most appreciated;-) For example, Python bindings are not "wired" into Python infrastructure. But I'm not suggesting that you jump into it! As we all have own things to do:-) So maybe instead of speaking in hypotheticals, just tell what is it that you're actually missing at work. If your project relies on the "naked" C (why?), and it resides in git, what prevents you from creating a git reference to blst and simply call blst/build.sh as part of your build procedure? It doesn't have any system dependencies and doesn't require any autoconfiguration...

[erikd]

We use blst from Haskell. Haskell has a whole bunch of machinery to handle C libraries via the pkg-config mechanism (even for Mac and Windows). This is for all the C libraries we currently use.

If the existing blst build system provided a pkg-config file I would probably not be asking this question.

[dot-asm]

If the existing blst build system provided a pkg-config file I would probably not be asking this question.

But it's not something we can just add to the repository, right? I mean the file is supposed to be in a specific place on the target system and has to have an absolute path. Unless you're accustomed to set and extend PKG_CONFIG_PATH to your liking. But even then it doesn't tell you anything about how to build. What are the discovery mechanisms there?

[dot-asm]

We use blst from Haskell.

It's getting weirder and weirder :-) But on a serious note. Out of curiosity (and for future reference), do you have to mirror structure declarations or are they "imported" straight from blst.h? Or maybe you ended up querying sizes and using correspondingly sized arrays of 64-bit integers as an opaque storage... Like in Java and C#...

[dot-asm]

What are the discovery mechanisms there?

Even if there are none on the Haskell side, you (not you personally, but the organization) ought to have some procedure to accommodate custom packages into your environment. Again, what prevents you from using git references? Even if git references is not an option, you can create a Makefile (or an equivalent of your choice) that does git clone blst, executes its build.sh and generate pkg-config...

[erikd]

But it's not something we can just add to the repository, right?

No, it requires dynamic system dependent content.

Out of curiosity (and for future reference), do you have to mirror structure declarations or are they "imported" straight from blst.h?

Basically the later. Haskell has very good interoperability with C.

Even if there are none on the Haskell side

Our preference would be for all our C library dependencies use the same mechanism, ie pkg-config. We would like to minimize the number of special snowflakes that have their own special method. At the moment we do git clone blst but then we have special snowflake build code to hook it into the rest of our system, but blst is the only C library we use that has special requirements.

[dot-asm]

But it's not something we can just add to the repository, right?

No, it requires dynamic system dependent content.

"No" as in "not right" or "no" as in "not something we can add"? If former, then elaborate.

snowflake build

It doesn't really answer the question about what are the discovery mechanism. So let me rephrase, if given an opportunity to add one artefact to the blst repository, what would it be? On the other hand, even if there an artefact you would still have to git clone, and you would still classify it as a "snowflake build." :-) :-) :-) In which case no amount of artefacts would help...

[erikd]

"No" as in "not right" or "no" as in "not something we can add"? If former, then elaborate.

No, you can't just add a static file to the repo. The file changes depending on installation location.

So let me rephrase, if given an opportunity to add one artefact to the blst repository, what would it be?

I would have blst provide a pkg-config file.

Having to git clone is less than idea, but not as big a deal as having to MacGyver up a pkg-config file.

[dot-asm]

I would have blst provide a pkg-config file.

Now I'm utterly confused. How do you reconcile "no, you can't" and "I would do it"? As for MacGyver-ing one. Is it really that much of a challenge to have the script that does git clone and calls build.sh generate a few lines?

[eli-schwartz]

So let me rephrase, if given an opportunity to add one artefact to the blst repository, what would it be?

If that artefact was a file called meson.build then it could internally generate a spec-compliant pkg-config file containing paths correct for the installation location, using the single line import('pkgconfig').generate(the_library) after building a library.

(Neither cmake nor autotools have pkgconfig generators, by the way.)

BTW: meson could also handle the python bindings natively, including handling the correct versioned filename as well as installing it to the system site-packages, which would greatly simplify the current extremely ad-hoc approach. Meson also can generate emscripten projects as well as compile jars / JNI and compile rust projects if you manually wrap Cargo.toml dependencies using meson (that's probably more work than you're interested in, though it's on the roadmap to generate this on the fly).

You can do a lot better than build.sh, basically. :)

[dot-asm]

You can do a lot better than build.sh, basically. :)

Problem is that there are too many people that would define "do" differently, let alone "better":-) Anyway, the intention behind build.sh is to provide a baseline if you wish. Something people can use as a starting point in a build system of their choice. I'm suggesting to use build.sh, because if it works, then it's done. But if there is an aversion to using it, it's not really a secret that it takes just two files to compile, no configuration needed.

As for Python. The preference is to have something that works with minimum extra requirements. I mean if Python has an "official" way to build modules that requires only Python itself, then it would be preferred over anything else. As already mentioned, I didn't actually dig into it, expecting that users would fill in:-) The run.me-s are really about testing.

But instead of just talking, care to concoct an example?

[dot-asm]

Please see #196.

[SamWilsn]

I am also interested in providing a meson build configuration for blst. If I (or someone else) were to put up a pull request switching over, is there a chance of getting it merged?

I'm packaging sigp/lighthouse for Gentoo, and blst is one of the last bits I need. Using one of the supported build systems would make that much easier. Can handle cross compilation, CPU flag detection, generating pkg-config files, and all that fun stuff. It's Gentoo's policy to de-bundle/un-vendor as much as possible.

This isn't a problem unique to Gentoo either. NixOS complained, and FreeBSD works around it.

[dot-asm]

I'm packaging sigp/lighthouse for Gentoo, and blst is one of the last bits I need.

I don't quite follow. It's a Rust project and blst is handled transparently by cargo... So what would "switching over" actually mean in the context? Or maybe more relevant is what does meson bring to the table cargo doesn't? You ought to talk about non-Rust cases, right? But even then what would "switching over" mean? The critique appears to be rather "lack of," so the goal is to add something, not to replace... Or are you suggesting to literally remove build.sh? Is it actually an impediment to anything and meson in particular? But either way...

If I (or someone else) were to put up a pull request switching over, is there a chance of getting it merged?

Here is the thing. Something being merged effectively makes us responsible for maintenance. Now, if the amount of effort is such that you require prior guarantees, then it kind of raises a flag if we would have capacity to keep it up-to-date further down the line... In other words I'm reluctant to answer the question... Is it actually that much of an effort? 10x #196(*)? If so, is it 20x the value?

(*) Hmm, it seems to be up-voted by this request's originator with an emoji. Please don't, leave a message instead :-) BTW, the suggested curl command works even without PR being merged. @erikd , can confirm that it brings value and requires no further adjustments?

[SamWilsn]

Thanks for getting back to me!

I don't quite follow. It's a Rust project and blst is handled transparently by cargo... So what would "switching over" actually mean in the context?

"transparently" is a bit of a fuzzy term to define. In most distributions (Gentoo included), dependencies are expected to be installed system-wide, and discovered by dependants, often with pkg-config. C language dependencies of Rust projects specifically are supposed to be unbundled and installed separately (see here.) The idea is that libraries can update separately from the applications that use them, and that the system administrator can choose all the little configuration details of each package.

The merits of separating dependencies have been discussed elsewhere in depth (for example in Debian.) I'm only trying to follow the packaging guidelines for Gentoo, and have no control over their policies.

So what would "switching over" actually mean in the context? Or maybe more relevant is what does meson bring to the table cargo doesn't? You ought to talk about non-Rust cases, right? But even then what would "switching over" mean?

I haven't delved deeply into your build system yet (wanted to see how you felt before burning too much time), so some of this may be wildly off base, but with that in mind...

I'm guessing that there are two "steps" in your build: one to compile the C code into a shared/static library, and then the normal Rust build. I think I saw some perl scripts in there to generate assembly, but I don't know where that fits in yet.

Switching over would entail:

• Adding a meson.build that compiles/installs the non-Rust code.
• Modifying the Rust bindings to link against the globally installed blst library, if specified by an environment variable. This is fairly common practice.
• Remove build.sh and build.bat, and optionally replace with stubs that forward to meson.

Here is the thing. Something being merged effectively makes us responsible for maintenance.

Yes, I am familiar with open source software development 😉

Now, if the amount of effort is such that you require prior guarantees, then it kind of raises a flag if we would have capacity to keep it up-to-date further down the line... In other words I'm reluctant to answer the question... Is it actually that much of an effort? 10x #196(*)? If so, is it 20x the value?

You have to maintain a build system regardless of whether it's a shell script or a meson script.

I asked ahead of time because you look like you have quite a complicated build, and there's no point in me investing any time in learning it if my pull request would be rejected outright. I don't expect the maintenance load to get any worse with meson after the initial migration.

In other words I'm reluctant to answer the question...

All I need to know before starting is whether you'll seriously consider merging my pull request and not reject it off hand. If you're married to shell/batch files, I'd rather not make the attempt.

[dot-asm]

I'm guessing that there are two "steps" in your build: one to compile the C code into a shared/static library, and then the normal Rust build.

No. blst instructs cargo what to do, no additional steps are required on the user side. Same with Go. Corresponding users simply do what they are accustomed to do, no hassle.

I think I saw some perl scripts in there to generate assembly, but I don't know where that fits in yet.

Nowhere. The scripts' outputs are pre-generated and committed to repository, they end up on crates.io (or pkg.go.dev), no additional steps are required on the user side.

The fact that I need to explain this suggests that there is misunderstanding about the scope of the problem. Note that the original problem was to facilitate in-house Haskel application development. That, i.e. non-Rust-or-Go application developers, is the right scope. What value a meson.build would bring for them.

Just in case. Drawing the line between what would be a "system" component or a piece of code linked into application is a trade-off. It's argued that blst is of the latter kind. Because it's small and is pretty useless outside specific language bindings. It's more appropriate to leave the freedom of how it's linked into any particular application to its developer. To give a concrete example, it would be unreasonable to demand that a Java module would require dependency on a "system" component, self-sufficient [multi-platform] .jar is the way to go...

[SamWilsn]

No. blst instructs cargo what to do, no additional steps are required on the user side. Same with Go. Corresponding users simply do what they are accustomed to do, no hassle.

Sorry, perhaps I wasn't clear. Conceptually there are two steps: C compilation followed by Rust compilation. Whether there is user interaction required or build.rs does it automatically is immaterial, at least to me.

blst is very similar to other *-sys crates in this regard.

Nowhere. The scripts' outputs are pre-generated and committed to repository, they end up on crates.io (or pkg.go.dev), no additional steps are required on the user side.

Great, so nothing for meson to do here either!

The fact that I need to explain this suggests that there is misunderstanding about the scope of the problem. Note that the original problem was to facilitate in-house Haskel application development. That, i.e. non-Rust-or-Go application developers, is the right scope. What value a meson.build would bring for them.

Changing the build system would bring little to no benefit to developers. Instead, it brings benefits for packagers, which is the hat I'm wearing for this conversation.

Aside: Theoretically it should also ease your lives a bit (like cross-platform SIMD detection, among other things), but I'm not qualified to make any stronger assurances there.

Drawing the line between what would be a "system" component or a piece of code linked into application is a trade-off. It's argued that blst is of the latter kind. Because it's small and is pretty useless outside specific language bindings. It's more appropriate to leave the freedom of how it's linked into any particular application to its developer. To give a concrete example, it would be unreasonable to demand that a Java module would require dependency on a "system" component, self-sufficient [multi-platform] .jar is the way to go...

Unfortunately this really isn't my fight. Linux distributions (excluding containerized ones...) generally agree that dependencies should be installed separately from dependants. Different distributions go to different lengths, for sure, but the tendency is there. Gentoo's policy is to unbundle as much as possible. They even have a page to share with upstream maintainers like yourself.

I don't mean to repeat myself too much, but in short: would you consider merging a pull request switching to meson if I put one up?

[dot-asm]

Sorry, perhaps I wasn't clear.

Since we're trying to be clear. Forget about Rust and Go.

I don't mean to repeat myself too much, but in short: would you consider merging a pull request switching to meson if I put one up?

Now I [for one] am more reluctant to answer the question. The problematic keyword is "switching". The scope is specifically to add something useful to people, not to dictate everybody what to do.

[SamWilsn]

I can certainly do meson in addition to what you already have, but that's just adding yet another build system to an already complicated build. If that's what you want, I'll go ahead and give it a try!