Fix for Rust multi-license declaration, plus new Rust license tests

[johnbatty]

Added some Rust multi-licensing tests to help diagnose #856

• A couple of tests of the clearlydefined scanner, which show that it is correctly declaring the Rust licenses
• Another test that runs some real raw data through the multiple scanners and aggregrator. This demonstrates that Rust multi-licenses are declared incorrectly, and is useful to run under a debugger with breakpoints to view/understand the operation.

The latter test should possibly be moved into a separate "functional" test, as it is testing real data through multiple components. I plan to clean it up to run through multiple example data files to confirm that they are all declared as expected.

I'm not sure whether the raw data files should be in an evidence directory, or a fixtures directory, or somewhere else - I used evidence as there was a similar example that used this structure.

[johnbatty]

@nellshamrell I have made a proposed fix for the Rust crate licensing, following our discussion.

I have modified the non-clearlydefined summarizers to avoid declaring the license for Rust crates because:

• their current logic gets it wrong (ANDing together the license types from the license files)
• they don't read Cargo.toml to find out what the declared license is, which has got to be the best way to get a crate license
• the clearlydefined summarizer does read Cargo.toml and declares the license from that

The tests that I have added show that this works for selection of example cases I have added.

There is a quirk of the ClearlyDefined SPDX parser adding unnecessary parentheses when there are 3 licenses. e.g.

• MPL-2.0 OR MIT OR Apache-2.0 declared as MPL-2.0 OR (MIT OR Apache-2.0)

I see you've previously raised this as an issue with the SPDX library: clearlydefined/spdx#21.

It would be good to fix this, as it seems that Cargo license expression parsing does not support parentheses:

• Parens for setting precedence ehuss/license-exprs#3

Although the license declared by ClearlyDefined won't be used directly by Cargo, it seems a little wrong to declare a license string that cannot by parsed by Cargo. I think I'll take a look at whether we can just validate the SPDX expression providing in Cargo.toml, and if valid then use the string as-is, rather than using the "normalized" string.

As we discussed, there is the (hopefully rare) possibility that the crate declares the license incorrectly, and there are files with other licenses in the repo. However, I think that trying to deal with this case now would add significant complexity. I'd rather fix up the existing problem first to get the licensing declared correctly for the vast majority of crates with license choices.

[nellshamrell]

@johnbatty this looks fantastic! Thank you so much!

And I agree with you, based on previous work, the evidence directory seems like the best place to put the raw data files.

Approving this now, will likely deploy this tomorrow.