Calling MQF from Rust #3

luizirber · 2019-11-03T21:24:33Z

This uses bindgen to generate Rust bindings from the gqf.h header. Slowly adding more functions, but this initial PoC is based on the "simple counting test" in tests/CountingTests.cpp

TODO

Make the API more like Rust, based on the lower level parts (we don't want to use unsafe in the public API!)
Resolve some quirks with linking the C++ stdlib (I had to use Troubleshooting: linking with cc failed rust-fuzz/afl.rs#141 (comment) and https://flames-of-code.netlify.com/blog/rust-and-cmake-cplusplus/#another-static-library in an Ubuntu system, but there are probably better ways) Punted to Linking with C++ stdlib #6
stxll static lib is generated with different names depending on being release or debug. Add logic to build.rs
Set it up on CI
~~Figure out how to catch C++ exceptions (right now everything just blows up). Some pointers: Support C++ exceptions without undefined behaviour rust-lang/rust-bindgen#1208~~ Punted to Figure out how to catch C++ exceptions (right now everything just blows up). #7

luizirber · 2019-11-04T01:17:43Z

Question for @shokrof and @mr-eyes: is there a way to tell CMake to build only the static lib, and skip binaries and tests?

luizirber · 2019-11-04T03:04:30Z

include/gqf.h

@@ -109,7 +109,7 @@ extern "C" {

 	void qf_destroy(QF *qf);

-	void qf_copy(QF *dest, QF *src);
+	void qf_copy(QF *dest, const QF *src);


@shokrof how do you feel about making src into a const QF *? This makes my life easier (because of mutability and ownership rules in Rust), and should still be valid wherever qf_copy was called before, right?

shokrof · 2019-11-04T17:25:02Z

I think we need to implement a c wrapper for rust to catch exceptions and communicate the messages in a "Rusty" way. I use exceptions to signal when mqf is full and If you are trying to access out of the boundaries items.

luizirber · 2019-11-04T18:04:09Z

I think we need to implement a c wrapper for rust to catch exceptions and communicate the messages in a "Rusty" way. I use exceptions to signal when mqf is full and If you are trying to access out of the boundaries items.

A mix of https://github.com/luizirber/ukhs/blob/master/bbhash-sys/ffi.cpp and rust-lang/rust-bindgen#1208 should be enough. Ideally it would be better to have exceptions never cross thru a C call (wrap in a try/catch block, and return an error code?), but I can use this ideas from the links to wrap all functions that do throw exceptions and expose them to Rust.

mr-eyes · 2019-11-04T19:23:35Z

Question for @shokrof and @mr-eyes: is there a way to tell CMake to build only the static lib, and skip binaries and tests?

@luizirber Here you are #4

Build with cmake $DIR -DSUPRESS_BIN=ON -DSUPRESS_TESTS=ON

CMake options to suppress building binaries & tests

luizirber · 2019-11-07T06:04:13Z

build.rs

+fn main() {
+    let dst = Config::new(".")
+        .define("BUILD_STATIC_LIBS", "ON")
+        .build_target("MQF")


@mr-eyes I ended up reverting the CMake changes, this was enough to avoid all the other targets (including binaries and tests)

luizirber · 2019-11-07T06:17:28Z

build.rs

+        "debug" => "_debug",
+        _ => "",
+    };
+    println!("cargo:rustc-link-lib=static=stxxl{}", mode);


@shokrof I can't skip linking stxxl here, since libMQF.a brings the on-disk MQF and if it is not included there are many symbols missing when I try to run the tests. Not sure how to (or even if it is desirable) to modularize what gets added to the static/dynamic library,

Do you mean the c++ test?

No, the Rust tests. The code builds, but when I try to link the static library to the Rust code (to run the tests) there are missing symbols.

I can create a new target for you that only has MQF if it will make your life easier

luizirber · 2019-11-07T21:17:02Z

Currently there is enough functionality exposed to have a working SBT with internal MQF nodes in sourmash-bio/sourmash#772, so how do people feel about:

merging this PR
punting remaining TODOs to issues
add more API/methods over time (in new PRs)

Oh, and also releasing it to crates.io (the Rust package repository)?

drtamermansour · 2019-11-07T23:22:57Z

@luizirber @shokrof This is really going above my head :)
Does this affect the compilation of the MQF or increase the dependencies? Is it better to make the merge to the master or keep it in another branch?

shokrof · 2019-11-07T23:42:37Z

@luizirber Do we need to add a special target on the cmake to do the rust bindings?

luizirber · 2019-11-07T23:44:48Z

@luizirber @shokrof This is really going above my head :)

Oops, sorry! I suggested merging because it is already a minimum-viable wrapper, and it can be improved over time (instead of trying to make everything perfect now).

Punting to issues and making new PRs is something we do in sourmash, to avoid having huge PRs to review and merge.

Publishing to crates.io is preferred to installing from git in the Rust world, and it is actually a blocker for me updating the sourmash crate, because crates published in crates.io can't have dependencies external to crates.io.

Does this help a bit? =]

Does this affect the compilation of the MQF or increase the dependencies?
There are no changes for the C/C++ code (what is already in this repo).

Is it better to make the merge to the master or keep it in another branch?
I would rather have this merged in master, because keeping feature branches up to date is a bit annoying. Because the Rust wrapper is being checked by CI it also helps catch changes at the C/C++ level that break the wrapper, which is something that will not happen (automatically) if this stays in a feature branch.

luizirber · 2019-11-07T23:46:27Z

@luizirber Do we need to add a special target on the cmake to do the rust bindings?

No, it's all taken care by Cargo (the Rust package manager). You would need a cmake target if you want to create a Rust static library, but that doesn't make much sense for this project. In sourmash I generate a Rust static lib because that's what the Python side sees (including generating a C header to link against the right functions in the Rust static lib, which is something not being used in the project).

shokrof · 2019-11-07T23:48:10Z

if we have a new version, will Cargo update itself automatically or you have to run some commands?

luizirber · 2019-11-07T23:49:43Z

if we have a new version, will Cargo update itself automatically or you have to run some commands?

You need to bump the version in Cargo.toml, and run cargo publish to update the package registry. I can set up a github action or travis job to do that (if you want)

shokrof · 2019-11-07T23:51:32Z

it is ok. I got it

luizirber · 2019-11-08T00:12:59Z

Yay! But it was merged before I added the cargo metadata, please see #5 =]

luizirber added 4 commits November 3, 2019 20:32

Can initialize, yay

e05a9b3

more test

e82888b

build cleanup and comments

1883d92

add rust ffi to travis

96d51f0

luizirber force-pushed the rust_ffi branch from e213de2 to 96d51f0 Compare November 4, 2019 00:44

luizirber added 3 commits November 3, 2019 17:29

start rust API, move raw FFI into another module

5417584

avoid using lib

9182606

start rust API, move raw FFI into another module

4f08dff

luizirber force-pushed the rust_ffi branch from 27b29a8 to 4f08dff Compare November 4, 2019 02:14

implement drop and clone

5d33b5a

luizirber commented Nov 4, 2019

View reviewed changes

first pass at iteration

49a49ee

cmake options to suppress building binaries & tests

bc49f2b

mr-eyes and others added 6 commits November 4, 2019 21:25

changing variable name

8a1db65

serialization

8053c8d

Merge pull request #4 from mr-eyes/master

3b2a53b

CMake options to suppress building binaries & tests

avoid building binaries and tests

ab2e7ef

add merge and remove stxxl from build.rs

d6f0e5b

implement Sync, but need more testing

b557436

luizirber changed the title ~~[WIP] Calling MQF from Rust~~ Calling MQF from Rust Nov 7, 2019

luizirber added 2 commits November 7, 2019 03:01

bring linking to stxxl back

c469559

revert cmake changes

dee6e9d

luizirber force-pushed the rust_ffi branch from 40702d1 to dee6e9d Compare November 7, 2019 06:00

luizirber commented Nov 7, 2019

View reviewed changes

luizirber mentioned this pull request Nov 7, 2019

Playing with MQF sourmash-bio/sourmash#772

Merged

5 tasks

shokrof merged commit fc5f9d1 into master Nov 7, 2019

luizirber deleted the rust_ffi branch November 7, 2019 23:52

luizirber added the rust Rust-related issues and PRs label Nov 8, 2019

This was referenced Nov 8, 2019

Linking with C++ stdlib #6

Open

Figure out how to catch C++ exceptions (right now everything just blows up). #7

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Calling MQF from Rust #3

Calling MQF from Rust #3

luizirber commented Nov 3, 2019 •

edited

Loading

luizirber commented Nov 4, 2019

luizirber Nov 4, 2019

shokrof commented Nov 4, 2019

luizirber commented Nov 4, 2019

mr-eyes commented Nov 4, 2019 •

edited

Loading

luizirber Nov 7, 2019

luizirber Nov 7, 2019

shokrof Nov 7, 2019

luizirber Nov 7, 2019

shokrof Nov 7, 2019

luizirber commented Nov 7, 2019

drtamermansour commented Nov 7, 2019

shokrof commented Nov 7, 2019

luizirber commented Nov 7, 2019

luizirber commented Nov 7, 2019

shokrof commented Nov 7, 2019

luizirber commented Nov 7, 2019

shokrof commented Nov 7, 2019

luizirber commented Nov 8, 2019

Calling MQF from Rust #3

Calling MQF from Rust #3

Conversation

luizirber commented Nov 3, 2019 • edited Loading

TODO

luizirber commented Nov 4, 2019

luizirber Nov 4, 2019

Choose a reason for hiding this comment

shokrof commented Nov 4, 2019

luizirber commented Nov 4, 2019

mr-eyes commented Nov 4, 2019 • edited Loading

luizirber Nov 7, 2019

Choose a reason for hiding this comment

luizirber Nov 7, 2019

Choose a reason for hiding this comment

shokrof Nov 7, 2019

Choose a reason for hiding this comment

luizirber Nov 7, 2019

Choose a reason for hiding this comment

shokrof Nov 7, 2019

Choose a reason for hiding this comment

luizirber commented Nov 7, 2019

drtamermansour commented Nov 7, 2019

shokrof commented Nov 7, 2019

luizirber commented Nov 7, 2019

luizirber commented Nov 7, 2019

shokrof commented Nov 7, 2019

luizirber commented Nov 7, 2019

shokrof commented Nov 7, 2019

luizirber commented Nov 8, 2019

luizirber commented Nov 3, 2019 •

edited

Loading

mr-eyes commented Nov 4, 2019 •

edited

Loading