Insert MSM and FFT code and their benchmarks.

[einar-taiko]

This PR moves MSM and FFT to halo2curves as suggested in #84.

Adds benchmarks for FFT and MSM to serve as comparison standard in the future: current situation vs privacy-scaling-explorations/halo2#40 vs #29 vs post #163.

Note: This might require moving privacy-scaling-explorations/halo2#202 to halo2curves cc @jonathanpwang

Resolves taikoxyz/zkevm-circuits#150.

[mratsim]

Hi team,

Rationale for this PR is mentioned in privacy-scaling-explorations/halo2#84.

Following this I'd like to:

• Change best_multiexp name to best_msm
• Standardize an API for MSM / NTT so that accelerator providers can easily be integrated
  • or alternative backends for easy comparison or differential fuzzing
• implement the rest of Towards state-of-the-art multi-scalar-muls #163
  • Note: we're hitting the Rust orphan rule when adding the "Jacobian Extended" coordinate system. Like bn and bls12-381 curves, it might be worthwhile to consider integrating pasta instead of depending on upstream (see also discussion around upstream serialization default: fix: Improve serialization for prime fields #85 (comment) )

Depending on what is merged first either this PR or privacy-scaling-explorations/halo2#202 will need to be updated

[CPerezz]

This looks good! I'd like to run benchmars with and without this commit in our server cc: @AronisAt79 @ed255 So that we can also see if we really benefit from multithreading without exploding RAM comsumption.

Once we wee no RAM explosions, I'mm happy to merge this!
If you have any benchmark results in other machines or memory profiling @mratsim it would be nice if you could share it.

I'll try to run this locally and get it. But only have 16 threads avaliable.

[han0110]

LGTM! Checked locally the fft.rs and msm.rs are ported from halo2_proofs with only 2 lines of non-logic change.

Also got a question about MSM benchmark.

[mratsim]

The EC points for benchmark should be moved so it's not repeated and it's parallelized, running the 2^18 to 2^22 part of the benches take almost 30min.

Looking at the code for Point::random, 2 things can be slow when creating a point randomly, sqrt and clear_cofactor

halo2curves/src/derive/curve.rs

Lines 365 to 377 in 6e2ff38

	if let Some(y) = Option::<$base>::from(y2.sqrt()) {
	let sign = y.to_bytes()[0] & 1;
	let y = if ysign ^ sign == 0 { y } else { -y };
	let p = $name_affine {
	x,
	y,
	};
	use $crate::group::cofactor::CofactorGroup;
	let p = p.to_curve();
	return p.clear_cofactor().to_affine()

But BN254 has a cofactor of 1, so only sqrt is slow. On my optimized library (using addition chains not Tonelli Shanks) it takes 6877ns.

Creating 2^22 points in that case would take almost 30s

[CPerezz]

We could file an issue for that!
I've been wanting to add addition chains for some time. Maybe using https://github.com/kwantam/addchain from Riad S. Whaby.

[mratsim]

The bench came from my library, with addition chains ;), so likely in Halo2curves it might even take a minute to generate the points.

[han0110]

LGTM!

[CPerezz]

Nice work!

[mratsim]

Hold on on merging, @einar-taiko is looking into parallelizing and caching the generation of test (coeffs, points) pairs

[CPerezz]

Hold on on merging, @einar-taiko is looking into parallelizing and caching the generation of test (coeffs, points) pairs

Didn't do it as per your last comment :)

[mratsim]

The new bench with parallel point generation now completes under a minute

[han0110]

LGTM! Just a small suggestion, could be ignored.

[mratsim]

Here are my benchmarks on my laptop (i9-11980HK 8 cores). Unfortunately I have to go to zkSummit just after so benchmark on my 18-core workstation will have to wait end of week.

For 2^16 there is a factor 3-3.5x between halo2curves (90ms) and Gnark (30ms) / Constantine (22.5ms)

Halo2curves

Gnark

Constantine

[han0110]

I think we still need to add required-features = ["multicore"] for msm.rs to pass the CI build without default features.