Python Supercircuit #125

qwang98 · 2023-09-10T18:15:17Z

Will be PR'ed to main.

@leolara Ready for review and the MiMC7 example is fully functional.

My technical approach

Pass over JSON of sub-circuit AST to Rust. Obtain uuid.
Pass over JSON of sub-circuit TraceWitness to Rust using the same uuid.
halo2_mock_prover function takes vector of uuids and compiles super circuit before passing it to the mock prover.

qwang98 · 2023-09-10T22:37:13Z

src/frontend/python/chiquito/chiquito_ast.py


    def enable(self: Lookup, enable_annotation: str, enable_expr: Expr):
-        enable = ASTConstraint(enable_annotation, enable_expr)
-        if self.enable is None:
+        enabler = ASTConstraint(enable_annotation, enable_expr)
+        if self.enabler is None:
            for constraint, _ in self.exprs:
-                constraint = self.multiply_constraints(enable, constraint)
-            self.enable = enable
+                constraint = self.multiply_constraints(enabler, constraint)


Called enabler here so that it doesn't clash with the function name.

qwang98 · 2023-09-10T22:55:17Z

src/frontend/python/chiquito/dsl.py

-        self.tables.add(table)
+        if self.super_circuit is None:
+            raise SyntaxError(
+                "Circuit: new_table() is only available for Circuit with initiated super_circuit field."
+            )
+        self.super_circuit.tables[table.uuid] = table


Note that here I opted to store the tables under supercircuit and then circuit/steptype both have reference to the supercircuit, so they can reference the tables as well.

qwang98 · 2023-09-18T18:52:09Z

src/frontend/python/mod.rs

+pub fn chiquito_super_circuit_halo2_mock_prover(rust_ids: Vec<UUID>) {
+    let mut super_circuit_ctx = SuperCircuitContext::<Fr, ()>::default();
+
+    // super_circuit def
+    let config = config(SingleRowCellManager {}, SimpleStepSelectorBuilder {});
+    for rust_id in rust_ids.clone() {
+        let circuit_map_store = rust_id_to_halo2(rust_id);
+        let (circuit, _, _, _) = circuit_map_store;
+        let assignment = super_circuit_ctx.sub_circuit_with_ast(config.clone(), circuit);
+        add_assignment_generator_to_rust_id(assignment, rust_id);
+    }
+
+    let super_circuit = super_circuit_ctx.compile();
+    let compiled = chiquitoSuperCircuit2Halo2(&super_circuit);
+
+    let mut mapping_ctx = MappingContext::default();
+    for rust_id in rust_ids {
+        let circuit_map_store = rust_id_to_halo2(rust_id);
+        let (_, _, assignment_generator, witness) = circuit_map_store;
+        if let Some(witness) = witness {
+            mapping_ctx.map_with_witness(&assignment_generator.unwrap(), witness);
+        }
+    }
+
+    let super_assignments = mapping_ctx.get_super_assignments();
+
+    let circuit = ChiquitoHalo2SuperCircuit::new(compiled, super_assignments);
+
+    let prover = MockProver::<Fr>::run(10, &circuit, circuit.instance()).unwrap();
+
+    let result = prover.verify_par();
+
+    println!("result = {:#?}", result);
+
+    if let Err(failures) = &result {
+        for failure in failures.iter() {
+            println!("{}", failure);
+        }
+    }
+}


chiquito_super_circuit_halo2_mock_prover

qwang98 · 2023-09-18T18:52:51Z

src/frontend/python/chiquito/dsl.py

+    def sub_circuit(self: SuperCircuit, sub_circuit: Circuit) -> Circuit:
+        assert self.mode == SuperCircuitMode.SETUP
+        if sub_circuit.rust_id != 0:
+            raise ValueError(
+                "SuperCircuit: sub_circuit() cannot be called twice on the same circuit."
+            )
+        ast_json: str = sub_circuit.get_ast_json()
+        sub_circuit.rust_id: int = rust_chiquito.ast_to_halo2(ast_json)
+        self.ast.sub_circuits[sub_circuit.rust_id] = sub_circuit.ast
+        return sub_circuit


sub_circuit Python function.

qwang98 · 2023-09-18T19:17:30Z

For the mapping, you should serialise a SuperAssginments. You are doing something complicated with the uuid, just make sure that the AST uuid generated in the python side can be used to index the SuperAssigments.

I'd comment that historically we only serialize a TraceWitness, pass it over to Rust, and then process it with an AssignmentGenerator to obtain Assignments. We don't serialize Assignments directly, as it requires Python versions of Rust functions like assign_step, find_placement, and get_selector_assignment from assignment.rs, which are better left to Rust alone. Therefore, I wonder by saying serialize SuperAssignments, which is a mapping of UUID to Assignments, you actually meant to serialize a mapping UUID to TraceWitness?

That's actually what I did, to serialize a mapping of TraceWitness and then associate it with the corresponding AST under the same UUID. To walk you through the steps, here's the MiMC7 example:

mimc7 = Mimc7SuperCircuit()
mimc7_witnesses = mimc7.gen_witness((F(1), F(2)))
mimc7.halo2_mock_prover(mimc7_witnesses)

We first call gen_witness on the SuperCircuit mimc7, which returns a Dict[UUID, TraceWitness]. This TraceWitness dictionary is then passed to halo2_mock_prover, which serializes each TraceWitness, send it to Rust, and store it under the same UUID as its corresponding AST.

Then in Rust, the chiquito_super_circuit_halo2_mock_prover function converts each TraceWitness to an Assignments, using the AssignmentGenerator with the same UUID. These conversions are all done under a MappingContext, which eventually spits out a SuperAssignments.

Just to clarify, I created a UUID -> CircuitMapStore master mapping in Rust for the Python frontend, where CircuitMapStore is defined as:

type CircuitMapStore = (
    Circuit<Fr, ()>,
    ChiquitoHalo2<Fr>,
    Option<AssignmentGenerator<Fr, ()>>,
    Option<TraceWitness<Fr>>,
);

This is how all corresponding items are grouped under the same UUID.

Again, for your convenience, I'm marking the gen_witness Python function and halo2_mock_prover Python function each as a comment here.

qwang98 · 2023-09-18T19:18:14Z

src/frontend/python/chiquito/dsl.py

+    # called at the outermost level
+    # generates TraceWitness mapping
+    def gen_witness(self: SuperCircuit, args: Any) -> Dict[int, TraceWitness]:
+        self.mode = SuperCircuitMode.Mapping
+        self.mapping(args)
+        self.mode = SuperCircuitMode.NoMode
+        witnesses: Dict[int, TraceWitness] = self.ast.witnesses
+        del (
+            self.ast.witnesses
+        )  # so that we can generate different witness mapping in the next gen_witness() call
+        return witnesses


gen_witness Python function for SuperCircuit

qwang98 · 2023-09-18T19:18:25Z

src/frontend/python/chiquito/dsl.py

+    def halo2_mock_prover(self: SuperCircuit, witnesses: Dict[int, TraceWitness]):
+        for rust_id, witness in witnesses.items():
+            witness_json: str = witness.get_witness_json()
+            if rust_id not in self.ast.sub_circuits:
+                raise ValueError(
+                    f"SuperCircuit.halo2_mock_prover(): TraceWitness with rust_id {rust_id} not found in sub_circuits."
+                )
+            rust_chiquito.add_witness_to_rust_id(witness_json, rust_id)
+        rust_chiquito.super_circuit_halo2_mock_prover(
+            list(self.ast.sub_circuits.keys())
+        )


halo2_mock_prover Python function for SuperCircuit

qwang98 · 2023-09-18T19:18:51Z

Ready for review again. Mostly responding to comments. I also merged latest changes. @leolara

…okuptables

qwang98 · 2023-09-19T17:09:16Z

This PR now contains both fixed gen refactoring from #117 and super circuit related changes.

@leolara Ready for final review.

Changes:

Passes over super_witness which is a dict of uuid to trace witness json to halo2_mock_prover for super circuit rather than storing any trace witness on the Rust side. super_witness is parsed in Rust.
Removed dict of uuid to lookup table from super circuit. All lookup table will be accessed directly, either from the circuit itself, or from another circuit through imports.
Renamed read_only flag to finished_flag and created set_finished_flag method for encapsulation.

leolara · 2023-09-21T05:01:23Z

examples/mimc7.py

+        self.transition(eq(self.circuit.row + 1, self.circuit.row.next()))
+
+        self.add_lookup(
+            self.circuit.imports.apply(self.circuit.row).apply(self.circuit.c)


@qwang98 I think it makes sense to call it self.circuit.constants_table

Fixed. I replaced the imports parameter for Circuit as **kwargs, so that the user can pass in arbitrarily named parameters for imports.

leolara · 2023-09-21T05:01:41Z

examples/mimc7.py

+        self.add(self.mimc7_last_step, x_value, k_value, c_value, row_value)
+
+
+class Mimc7FirstStep(StepType):


@qwang98 I think it is more clear to put the steps before the circuit

leolara · 2023-09-21T05:03:20Z

src/frontend/dsl/mod.rs

@@ -172,6 +162,27 @@ impl<F, TraceArgs> CircuitContext<F, TraceArgs> {
    }
 }

+impl<F: Field + Hash, TraceArgs> CircuitContext<F, TraceArgs> {
+    /// Sets the fixed generation function for the circuit. The fixed generation function is


@qwang98 this does not "set", but "executes" and sets the assignations

leolara · 2023-09-21T05:07:55Z

src/frontend/python/mod.rs

+    Circuit<Fr, ()>,
+    ChiquitoHalo2<Fr>,
+    Option<AssignmentGenerator<Fr, ()>>,
+    // Option<TraceWitness<Fr>>,


@qwang98 remove commented out line

leolara · 2023-09-21T05:08:34Z

src/frontend/python/mod.rs

+        circuit_map_store.2 = Some(assignment_generator);
+    });
+
+    println!("Added AssignmentGenerator to rust_id: {:?}", rust_id);


@qwang98 remove println

leolara

Just a few comments, but it is approved

qwang98 added 14 commits August 29, 2023 09:18

initial commit and mimc7 example

2d7027c

all updates

545d651

put mimc7 constants to anothe file

8ede657

merge main

c784a55

addressed leo's comments

4e937b1

updated wit_gen

98eb019

address leo's comments

62c5fae

merge main

6428248

dsl

965b465

addressed all leo's comments

2a1905b

fmt

02a3c24

initial commit

d5fdb8e

ast

00aba02

fixed gen

a18e578

qwang98 changed the title ~~initial commit~~ Python Supercircuit WIP Sep 10, 2023

qwang98 added 5 commits September 10, 2023 14:25

dsl

9dd654a

partially functional mimc7 example

912b0f9

black fmt

832923f

rust id instead of ast id

9a5093f

hooked up frontend and backend

38b0a5e

qwang98 requested a review from leolara September 10, 2023 22:32

qwang98 changed the title ~~Python Supercircuit WIP~~ Python Supercircuit Sep 10, 2023

qwang98 marked this pull request as ready for review September 10, 2023 22:33

qwang98 commented Sep 10, 2023

View reviewed changes

qwang98 force-pushed the steve/python-supercircuit-v3 branch from 2305e64 to 38b0a5e Compare September 10, 2023 22:43

qwang98 added 2 commits September 10, 2023 18:46

some fmt

f027cde

more fmt

c9f0622

qwang98 commented Sep 10, 2023

View reviewed changes

qwang98 added 2 commits September 10, 2023 18:56

minor

3d178b3

fixed Fr issue due to montgomery form; now MiMC7 example fully functions

c5334fb

qwang98 commented Sep 18, 2023

View reviewed changes

qwang98 added 2 commits September 18, 2023 15:33

fixed args related errors

bfde1cc

update pypi

06851e2

qwang98 changed the base branch from steve/python-fixed-gen-lookup-mimc7 to main September 19, 2023 14:49

qwang98 added 4 commits September 19, 2023 12:27

removed dict of uuid to lookup and modified lookupbuilder to store lo…

e891154

…okuptables

pass over super witness rather than storing them

9b3186b

Merge branch 'main' into steve/python-supercircuit-v3

c312a53

update pypi

3b8f938

qwang98 mentioned this pull request Sep 19, 2023

Python: fixed gen, lookup, and mimc7 #117

Closed

qwang98 self-assigned this Sep 19, 2023

This was linked to issues Sep 19, 2023

Refactor fixed assignments #90

Closed

Port mimc7 example to python frontend #100

Closed

Implement super circuit in python front-end #99

Closed

leolara reviewed Sep 21, 2023

View reviewed changes

leolara approved these changes Sep 21, 2023

View reviewed changes

last comments fixed

4a94033

qwang98 added this pull request to the merge queue Sep 25, 2023

Merged via the queue into main with commit e0faa21 Sep 25, 2023
4 checks passed

alxkzmn deleted the steve/python-supercircuit-v3 branch August 7, 2024 11:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python Supercircuit #125

Python Supercircuit #125

qwang98 commented Sep 10, 2023 •

edited

Loading

qwang98 Sep 10, 2023

qwang98 Sep 10, 2023

qwang98 Sep 18, 2023

qwang98 Sep 18, 2023

qwang98 commented Sep 18, 2023

qwang98 Sep 18, 2023

qwang98 Sep 18, 2023

qwang98 commented Sep 18, 2023

qwang98 commented Sep 19, 2023 •

edited

Loading

leolara Sep 21, 2023

qwang98 Sep 25, 2023

leolara Sep 21, 2023

qwang98 Sep 25, 2023

leolara Sep 21, 2023

qwang98 Sep 25, 2023

leolara Sep 21, 2023

qwang98 Sep 25, 2023

leolara Sep 21, 2023

qwang98 Sep 25, 2023

leolara left a comment

		self.add(self.mimc7_last_step, x_value, k_value, c_value, row_value)


		class Mimc7FirstStep(StepType):

Python Supercircuit #125

Python Supercircuit #125

Conversation

qwang98 commented Sep 10, 2023 • edited Loading

My technical approach

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qwang98 commented Sep 18, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qwang98 commented Sep 18, 2023

qwang98 commented Sep 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leolara left a comment

Choose a reason for hiding this comment

qwang98 commented Sep 10, 2023 •

edited

Loading

qwang98 commented Sep 19, 2023 •

edited

Loading