Rework library interface

[bakaq]

This completely reworks the library interface, moving in the direction of #2490. This already has #![deny(missing_docs)], but I just did very basic documentation. This isn't intended to be a final version, and we probably want to adjust it a little¹, especially in the naming. For now this is mostly about the interface and I haven't properly implemented a lot of them or even migrated the old tests yet, but I plan to do that soon. After we decide a good direction for the APIs, implement most of it and migrate the tests, we can write proper proper documentation, new tests and examples².

I have a rendered version of the cargo doc output from this PR here so that it's easier to review the interface. I will try to keep it updated with the tip of this branch.

Some things I want to bring attention to:

• PrologTerm³ (the old Value) uses OrderedFloat<f64>, Integer and Rational, which are from the ordered_float and dashu crates. This is a semver hazard, because we would need a breaking change every time one of those crates has a breaking change. It's specially hazardous because neither of them are 1.0 yet, which tends to mean "unstable" in the Rust ecosystem. I don't think this is too bad actually, as the cadence of major versions of both of those crates seem to be very slow. I also think it wouldn't be very wise to wrap lot of dashu in Scryer Prolog to try to avoid this. On the other hand, I don't think OrderedFloat<f64> is necessary in PrologTerm. Using f64 instead would mean that we lose Eq, which isn't that bad.
• StreamConfig and MachineConfig are now opaque and (kind of) use the builder pattern, which is a very common thing in the Rust ecosystem and means we can freely change the internal representation of these types (which I plan to do a lot in the future to enable some really cool stuff in Wasm).
• I don't have a CompleteAnswer type. I don't really see a benefit of using it instead of just collecting QueryState into a Vec<LeafAnswer> or something like that. Please give any examples if you think of any.

The following are some things I think I'm already going to do. If someone disagrees with any of them, please let me know!

• I think that Machine::new_lib() should just be a Default implementation. Maybe also get rid of Machine::new() in favor of something like MachineConfig::build() to get the full benefits of the builder pattern.
• Change MachineConfig::with_toplevel() to accept non-static strings, so that people can use a runtime generated toplevel without having to leak. This would need some deep changes probably, so I'm not sure if it's very simple to do.

@mthom @Skgland @triska @lucksus I will appreciate if you take a look at this.

Footnotes

1. For example, I would really like if we leave space in the API for "lazy" APIs that don't need to allocate. They would be specially useful for the C API. ↩

2. It will also unblock ISSUE-2464: exposing scryer prolog functionality in libscryer_prolog.so for client library consumption #2465, because I think the interface will be mostly stable after that so there will not be many conflicts. ↩

3. I wanted to call it Term, but because Term already exists in the parser and we have wildcard imports everywhere there are a lot of conflicts that seem kind of complicated to fix. It seems that the rebis-dev branch gets rid of that type, so that's kind of exciting. ↩

[bakaq]

• Migrated the tests (except the "integration tests", which I already did in a34996a and will reintegrate into this branch soon), and benches, so the CI passes now.
• Renamed MachineConfig to MachineBuilder and removed Machine::new() and similar to solidify the builder pattern. The builder pattern will make it very smooth to add arbitrary additional configuration in the future.
• Removed some "extra" helper methods to make the API surface for this MVP (Minimum Viable Product) as small as possible. I believe the current interface is enough for almost all use cases, and future improvements would be to add helper methods for common patterns (like just checking if a query succeeds).

As always, documentation current to the tip of this branch is available here.

[bakaq]

Actually, the integration tests depend on JSON serialization. Should I integrate #2493 into this PR? I think it would be a better idea to merge this first and then rebase that one onto master, because I think JSON serialization still needs a lot of discussion that is mostly disconnected to the discussion that this PR needs.

[bakaq]

Reviewing this, I think the only thing I still want to change here is for <QueryState as Iterator>::Item to be just LeafAnswer instead of Result<LeafAnswer, String>, because LeafAnswer already has an Exception(Term) variant, or maybe Result<LeafAnswer, Term> to special-case errors from other exceptions (I think this may be better).

[bakaq]

Ok, with this I think it is good enough to stabilize. As always, the up-to-date rendered documentation is here.

Also, reminding that this is just the API skeleton, although most of it is already implemented. This still doesn't have support for residual goals for example, but it leaves space for it to be implemented later without an API breaking change.

Stabilizing the Rust API will unblock parallel ventures such as #2465 and improving the Wasm interface because we will all have a stable foundation to build upon. There's still work to be done here, but it will be almost completely independent of these other ventures.

Pinging relevant people @mthom @jjtolton @Skgland

[Skgland]

A few notes and suggestions for things we could do while we are at it.
Otherwise this looks good to me.

[jjtolton]

@bakaq should I rebase #2465 on top of this?

[bakaq]

I think I addressed everything that was pointed out.

should I rebase #2465 on top of this?

Well, if you want. But that branch is already rebased onto #2493, so it would make sense to rebase that one onto this first. I'll get to it soon, but getting this merged first would make everything easier.

[Skgland]

Looks good 👍