Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cross-compiling or statically linking sqlx fails because of OpenSSL #670

Closed
emk opened this issue Sep 2, 2020 · 4 comments
Closed

Cross-compiling or statically linking sqlx fails because of OpenSSL #670

emk opened this issue Sep 2, 2020 · 4 comments

Comments

@emk
Copy link

emk commented Sep 2, 2020

Hello! Thank you for such an awesome Rust SQL client. We've been trying it out at work, and it looks super-promising. We'd love to use it to replace diesel and tokio-postgres in some of our open source tools.

However, many of our open source tools like falconeri and dbcrossbar are shipped as statically-linked binaries using my rust-musl-builder. (rust-musl-builder seems to be popular for other Rust projects, with something like 740,000 downloads. So if I can figure out how to make sqlx work for me, that would hopefully help out some other people, too.)

The problem: Compile-time OpenSSL connections cause linker headaches when cross-compiling

In order to produce a statically-linked binary using sqlx, we need to compile it twice, for two different environments:

  1. The host environment. Here, sqlx-macros is compiled against GNU Libc, and Rust requires it to be a shared library that can be loaded as a "plugin" for the compiler.
  2. The target environment. Here, sqlx is compiled against musl-libc, which is a 100% static C library that will allow a Rust binary to work on any Linux distribution with a modern kernel. This is super-useful for distributing one Linux binary that works everywhere. In this case, sqlx must be compiled as a static library, not a shared one.

The problem occurs because:

  • sqlx-rt always links against OpenSSL, in both the host and target environments.
  • The Rust openssl-sys library can be configured to link either statically or dynamically using OPENSSL_STATIC, but we're forced to choose one.

Here are three proposed solutions. I'd be happy to submit a PR for any of them.

Proposed solution 1: Fix openssl-sys to support both static and dynamic versions of OpenSSL at the same time

This is possible, but it might be complicated. The only reason to do this would be to link against OpenSSL in a macro while cross-compiling. But OpenSSL is native C code, and it makes everything related to cross-compiling excessively difficult.

Proposed solution 2: Modify sqlx-macros to have an offline-only feature

I could submit a PR that modified sqlx-macros to support an offline-only feature that required an up-to-date sqlx-data.json file, and that had no ability to connect to the database. This would only be useful when cross-compiling. This might be a relatively simple solution for people who can pass --features to cargo build when cross compiling, and who can set up appropriate features.

Proposed solution 3: Add support for rustls to sqlx-rt

If solution (2) seems unappealing, another option would be to allow using sqlx-rt without any C dependencies at all. To do this, we could use a native Rust library like rustls. I've used this in other Rust projects, and it seems to be well behaved and easy to set up.

If we had support for rustls, cross-compiling would magically work, even for people who just have a stock Rust install. There would be no need for tools like rust-musl-builder.

I'd also be happy to submit a PR for this approach.

Next steps

  1. Does this seem like a problem worth solving?
  2. If I send you a PR solving this problem, would you be interested in merging it?
  3. Which of the proposed solutions above (1, 2 or 3) seems like the best approach?

I'd be delighted to send you a PR + documentation. And I could add an sqlx-specific test to rust-musl-builder to make sure that every release of rust-musl-builder still supports cross-compiling with sqlx. This would make it very easy for Rust developers to build static Rust binaries using sqlx.

Once again, thank you very much for such an awesome Rust SQL interface! I really love the way it allows us to use natural SQL while still preserving the benefits of Rust's type checking.

@mehcode
Copy link
Member

mehcode commented Sep 2, 2020

For an immediate workaround, look to:

#473 (comment)

@mehcode
Copy link
Member

mehcode commented Sep 2, 2020

I don't mind adding support for rustls. It's just non-trivial because of a limitation with Cargo.

See: #575 (comment)

@emk
Copy link
Author

emk commented Sep 2, 2020

There's a very promising fix proposed in sfackler/rust-openssl#1337 (comment).

I'm going to try adding this to rust-musl-builder and see what happens.

emk added a commit to emk/rust-musl-builder that referenced this issue Sep 3, 2020
`sqlx` requires linking against OpenSSL twice:

- `sqlx-macros` needs to link against OpenSSL as a shared libary for use
  at compile time.
- `sqlx` needs to link against OpenSSL a static library for use at
  runtime.

You can find the details here.

launchbadge/sqlx#670
sfackler/rust-openssl#1337

We go with the fix proposed by sfackler, and add a test program that we
can use to verify everything works correctly.
emk added a commit to emk/rust-musl-builder that referenced this issue Sep 3, 2020
- `sqlx-macros` needs to link against OpenSSL as a shared libary for use
  at compile time.
- `sqlx` needs to link against OpenSSL a static library for use at
  runtime.

You can find the details here:

launchbadge/sqlx#670
sfackler/rust-openssl#1337

We go with the fix proposed by @sfackler, and add a test program that we
can use to verify everything works correctly.

We remove a few `OPENSSL`-related variables that were added 4 years ago,
and that no longer appear to be needed.
emk added a commit to emk/rust-musl-builder that referenced this issue Sep 3, 2020
`sqlx` requires linking against OpenSSL twice:

- `sqlx-macros` needs to link against OpenSSL as a shared libary for use
  at compile time.
- `sqlx` needs to link against OpenSSL a static library for use at
  runtime.

You can find the details here:

launchbadge/sqlx#670
sfackler/rust-openssl#1337

We go with the fix proposed by @sfackler, and add a test program that we
can use to verify everything works correctly.

We remove a few `OPENSSL`-related variables that were added 4 years ago,
and that no longer appear to be needed.
@emk
Copy link
Author

emk commented Sep 4, 2020

I have updated rust-musl-builder to set:

X86_64_UNKNOWN_LINUX_MUSL_OPENSSL_DIR=/usr/local/musl/
X86_64_UNKNOWN_LINUX_MUSL_OPENSSL_STATIC=1

...as proposed at sfackler/rust-openssl#1337 (comment). This makes sqlx work very nicely, out of the box. I've also added an sqlx example program to rust-musl-builder, and it's compiled as part of the test suite. It's still necessary to use cargo sqlx prepare to avoid connecting to a database at build time.

Thank you for your help figuring this out! I'm delighted to be able to build entirely static Linux binaries using sqlx, and I hope this helps somebody else.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants