Implement Combinatorial Betting and Futarchy #1364

maltekliemann · 2024-09-13T13:16:32Z

No description provided.

* Revert "New Asset System (#1295)" (#1338) * Revert "New Asset System (#1295)" This reverts commit a956877. * Fix formatting * Update copyright * Remove pallet-assets dependency * Fix fuzz tests * Merge `main` into `develop` (#1345) * Update dependencies to Polkadot v1.1.0 (#1331) * Update dependencies (#1319) * Update dependencies to polkadot-v1.1.0 * Format code * Remove duplicate dependencies * Update zrml-asset-router (#1321) * Update zrml-primitives * Partially update asset-router * Finalize logic adjustments in asset-router * Make asset-router tests compilable * Correct Inspect routing for market assets in Currencies * Directly invoke Inspect API for Currencies * Add tests for remaining Unbalances functions * Update remaining Zeitgeist pallets (#1322) * Update zrml-asset-router (#1321) * Upgrade zrml-market-commons * Upgrade zrml-authorized && use MockBlock instead of MockBlockU32 * Upgrade zrml-court * Upgrade zrml-global-disputes * Upgrade liquidity mining * Upgrade zrml-rikiddo * Upgrade zrml-simple-disputes * Upgrade zrml-styx * Upgrade zrml-orderbook * Upgrade zrml-parimutuel * Upgrade zrml-swaps * Upgrade zrml-prediction-markets * Upgrade zrml-neo-swaps * Upgrade zrml-hybrid-router * Update license headers * Update runtime (#1323) * Update weight files & Runtime enum * Use workspace metadata * Always use serde serialization for asset types * Make battery station standalone runtime compilable * Make benchmark and try-runtime feature compilable * Make BS build with all features * Make parachain tests compile * Partially fix xcm tests * Use safe xcm version 2 * Update Zeitgeist runtime (except xcm tests) * Format code * Remove deprecated comment * Integrate new xcm-emulator (#1324) * Integrate new xcm-emulator environment * Utilize new xcm-emulator interfaces * Spawn relay-para network using patched xcm-emulator * Use proper collator genesis config * Fix Rococo tests * Finalize Battery Station XCM tests * Finalize Zeitgeist XCM tests * Update client (#1327) * Fix rpc and work on client update * Finalize standalone client * Update parachain client * Use same try-runtime subcommand in every case * Update node/src/cli.rs Co-authored-by: Malte Kliemann <[email protected]> * Update try-runtime* Makefile targets --------- Co-authored-by: Malte Kliemann <[email protected]> * Make CI succeed and add migrations (#1329) * Fix rpc and work on client update * Finalize standalone client * Update parachain client * Use same try-runtime subcommand in every case * Satisfy Clippy * Fix benchmarks * Add migrations * Satisfy Clippy * Update moonkit depedencies * Free disk space more aggressively --------- Co-authored-by: Malte Kliemann <[email protected]> * Update spec version, try-runtime Makefile * Fix copyright notices * Fix broken chain state (#1336) * Add `StorageVersion` fix and contrats fix migrations * Don't set pallet-balances' storage version * Remove migrations from pallet-contracts config * Clear storage tries of contracts * Fix migration and info logs in try-runtime * Fix licenses and comments * Fix formatting --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> --------- Co-authored-by: Harald Heckmann <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Merge * Fix benchmark * Fix compiler error * Fix tests and imports * Fix imports (again...) * Fix orderbook benchmarks * Fix fuzz tests * Fix formatting * Fix orderbook fuzz --------- Co-authored-by: Harald Heckmann <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Update versions to v0.5.3 * Restructure math module * More scaffolding * Implement combinatorial buy math * Implement price calculation for combo * Remove `println!` * Implement equalization * Implement selling * Add tests for combinatorial buying * Add more tests for combinatorial buying * Add tests for equalization * Add more tests/corner cases * Implement full testing, fix critical bug --------- Co-authored-by: Harald Heckmann <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Implement classical buying using combinatorial buys * Calculate classical sells with combinatorial math * Implement `combo_buy` * Implement `combo_sell`

* Fix compiler and clippy issues * Fix formatting * Fix tests

* Scaffold combo pallet * Fix dependencies/features

* More scaffolding * Add `IdManager` trait * Scaffold ID manager * Use generics * WIP * Rough outline of `CryptographicIdManager` implementation * Improve `MaybeToBytes` implementation for `bool` * Implement pseudo-root * Add more tests * Implement `quadratic_residue` and add tests * Partial implementation of decompression algorithm * Loads of cleaning up * Refactor * Clean up `hash_pair` * Simplify interface * Simplify serialization * More cleanup * Clean up and tests * Better `ToBytes` implementation * Abstract `decompressor` tests * Reorganize tests * Minor clean up * More clean up * Test `get_collection_id`, fix bugs * Clean up * Expose `force_max_work` parameter and test it properly * Properly forget dummies * Add more tests * Fix some error handling, docs, and add missing tests * Prettify

* Extend `Config` * wip * Some refactors * Implement splitting tokens * Implement merging tokens

…1370) * . * Use `core::hint` * Replace `halo2curves` dependency with `ark-*` dependency * Fix clippy issues * Fix formatting

* Add mock for zrml-combinatorial-tokens * Add test framework * Add negative tests for `split_token` * Add more tests, fix some bugs, extend `Event` object * Add more tests * Add more integration tests * Add more integration tests * Add more integration tests * More tests * Add merge tests * final tests * fixes

codecov-commenter · 2024-10-13T17:23:06Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 95.31650% with 256 lines in your changes missing coverage. Please review.

Project coverage is 93.53%. Comparing base (2858166) to head (93f0663).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
zrml/neo-swaps/src/lib.rs	85.78%	61 Missing ⚠️
zrml/neo-swaps/src/weights.rs	55.33%	46 Missing ⚠️
zrml/combinatorial-tokens/src/lib.rs	86.44%	40 Missing ⚠️
...waps/src/types/decision_market_benchmark_helper.rs	0.00%	34 Missing ⚠️
...src/types/combinatorial_tokens_benchmark_helper.rs	0.00%	18 Missing ⚠️
zrml/futarchy/src/mock/types/scheduler.rs	71.42%	12 Missing ⚠️
zrml/futarchy/src/lib.rs	74.28%	9 Missing ⚠️
zrml/futarchy/src/weights.rs	50.00%	9 Missing ⚠️
zrml/neo-swaps/src/utility.rs	0.00%	9 Missing ⚠️
primitives/src/types.rs	0.00%	6 Missing ⚠️
... and 5 more

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1364      +/-   ##
==========================================
+ Coverage   93.21%   93.53%   +0.32%     
==========================================
  Files         132      178      +46     
  Lines       29358    34547    +5189     
==========================================
+ Hits        27366    32315    +4949     
- Misses       1992     2232     +240

Flag	Coverage Δ
tests	`93.53% <95.31%> (+0.32%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

* Add numerical thresholds to combinatorial betting * Add protected `exp` for normal bets * Ensure correctness of partitions * Check partitions

* Add licenses * Add remark about LGPL-3.0 licensing by Gnosis

* Fix formatting * Fix mess with copyright notices

Chralt98 · 2024-12-05T13:12:42Z

zrml/combinatorial-tokens/src/tests/merge_position.rs

+    ExtBuilder::build().execute_with(|| {
+        let alice = Account::new(0).deposit(Asset::ForeignAsset(1), _99).unwrap();
+
+        // Market has three outcomes, but there's an element in the partition of size two.


Suggested change

// Market has three outcomes, but there's an element in the partition of size two.

Chralt98 · 2024-12-05T13:31:51Z

zrml/combinatorial-tokens/src/tests/redeem_position.rs

+
+#[test_case(vec![B0, B1, B0, B1]; "incorrect_len")]
+#[test_case(vec![B0, B0, B0]; "all_zero")]
+#[test_case(vec![B0, B0, B0]; "all_one")]


Suggested change

#[test_case(vec![B0, B0, B0]; "all_one")]

#[test_case(vec![B1, B1, B1]; "all_one")]

Chralt98 · 2024-12-05T13:34:37Z

zrml/combinatorial-tokens/src/tests/redeem_position.rs

+}
+
+#[test]
+fn redeem_position_fails_if_tokens_have_to_value() {


Suggested change

fn redeem_position_fails_if_tokens_have_to_value() {

fn redeem_position_fails_if_tokens_have_no_value() {

Chralt98 · 2024-12-06T13:48:23Z

zrml/futarchy/src/dispatchable_impls.rs

+        let try_mutate_result = Proposals::<T>::try_mutate(to_be_scheduled_at, |proposals| {
+            proposals.try_push(proposal.clone()).map_err(|_| Error::<T>::CacheFull)
+        });
+
+        Self::deposit_event(Event::<T>::Submitted { duration, proposal });
+
+        Ok(try_mutate_result?)


Suggested change

let try_mutate_result = Proposals::<T>::try_mutate(to_be_scheduled_at, |proposals| {

proposals.try_push(proposal.clone()).map_err(|_| Error::<T>::CacheFull)

});

Self::deposit_event(Event::<T>::Submitted { duration, proposal });

Ok(try_mutate_result?)

Proposals::<T>::try_mutate(to_be_scheduled_at, |proposals| {

proposals.try_push(proposal.clone()).map_err(|_| Error::<T>::CacheFull)

})?;

Self::deposit_event(Event::<T>::Submitted { duration, proposal });

Ok(())

Chralt98 · 2024-12-06T14:08:17Z

zrml/futarchy/src/pallet_impls.rs

+            if result.is_ok() {
+                Self::deposit_event(Event::<T>::Scheduled { proposal });
+            } else {
+                Self::deposit_event(Event::<T>::UnexpectedSchedulerError);


This case is unfortunate. This can obviously lead to the following scenario: A proposal is submitted and stored in Proposals, but because the scheduler has too many scheduled calls at proposal.when, it fails and the whole proposal is removed from the storage, although it has been evaluated to be approved. It might be better to have another extrinsic call that allows approved proposals to be scheduled at the time of the root origins choosing. So essentially this function here could be called evaluate_proposal that either emits Approved and stores a boolean flag approved in the proposal or Rejected that removes the proposal. After the successful scheduling from the root origin, the proposal can be removed too.

Chralt98 · 2024-12-06T14:12:18Z

zrml/futarchy/src/pallet_impls.rs

+use frame_support::{
+    dispatch::RawOrigin,
+    pallet_prelude::Weight,
+    traits::schedule::{v3::Anon, DispatchTime},


Suggested change

traits::schedule::{v3::Anon, DispatchTime},

traits::schedule::{v3::Anon, DispatchTime, HARD_DEADLINE},

Chralt98 · 2024-12-06T14:12:41Z

zrml/futarchy/src/pallet_impls.rs

+            let result = T::Scheduler::schedule(
+                DispatchTime::At(proposal.when),
+                None,
+                63,


Suggested change

63,

HARD_DEADLINE,

Chralt98 · 2024-12-06T14:30:16Z

zrml/futarchy/src/benchmarking.rs

+    #[benchmark]
+    fn submit_proposal() {
+        let duration = T::MinDuration::get();
+
+        let oracle = T::BenchmarkHelper::create_oracle(true);
+        let proposal = Proposal {
+            when: Default::default(),
+            call: Bounded::Inline(vec![7u8; 128].try_into().unwrap()),
+            oracle,
+        };
+
+        #[extrinsic_call]
+        _(RawOrigin::Root, duration, proposal.clone());


At to_be_scheduled_at there could be 15 existing proposals in the worst case. This isn't accounted here in the benchmark.

Chralt98 · 2024-12-06T14:40:25Z

zrml/futarchy/src/benchmarking.rs

+    #[benchmark]
+    fn maybe_schedule_proposal() {
+        let when = u32::MAX.into();
+        let oracle = T::BenchmarkHelper::create_oracle(true);
+        let proposal =
+            Proposal { when, call: Bounded::Inline(vec![7u8; 128].try_into().unwrap()), oracle };
+
+        #[block]
+        {
+            Pallet::<T>::maybe_schedule_proposal(proposal.clone());
+        }
+
+        let expected_event = <T as Config>::RuntimeEvent::from(Event::<T>::Scheduled { proposal });
+        System::<T>::assert_last_event(expected_event.into());
+    }


At now in on_initialize there could be 15 existing proposals in the worst case. T::DbWeight::get().reads_writes(1, 1) does not account for storage decoding and encoding.

Chralt98 · 2024-12-06T14:48:35Z

zrml/futarchy/src/traits/mod.rs

+// Copyright 2024 Forecasting Technologies LTD.
+//
+// This file is part of Zeitgeist.
+//
+// Zeitgeist is free software: you can redistribute it and/or modify it
+// under the terms of the GNU General Public License as published by the
+// Free Software Foundation, either version 3 of the License, or (at
+// your option) any later version.
+//
+// Zeitgeist is distributed in the hope that it will be useful, but
+// WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+// General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License
+// along with Zeitgeist. If not, see <https://www.gnu.org/licenses/>.


This file and directory can be removed, since it has no content.

zrml/neo-swaps/src/weights.rs

Chralt98 · 2024-12-12T11:17:58Z

primitives/src/asset.rs

-    CombinatorialOutcome,
+    CombinatorialToken(CombinatorialId),
    PoolShare(PoolId),
    #[default]
    Ztg,
    ForeignAsset(u32),
    ParimutuelShare(MarketId, CategoryIndex),
 }
+// TODO Needs storage migration


Does not need a storage migration, because the enum variant CombinatorialOutcome has never been used before in the storage.

Each enum variant is encoded separately in the storage. https://docs.substrate.io/reference/scale-codec/

The two potential storage items that could contain a CombinatorialOutcome are Markets that contains the base_asset as a potential Asset. The other is Pools. But the Asset CombinatorialOutcome was never assigned.

Chralt98 · 2024-12-20T13:25:44Z

zrml/neo-swaps/src/lib.rs

@@ -444,7 +573,7 @@ mod pallet {
        ///
        /// # Parameters
        ///
-        /// - `market_id`: Identifier for the market related to the pool.
+        /// - `poold_id`: Identifier for the pool to withdraw liquidity from.


Suggested change

/// - `poold_id`: Identifier for the pool to withdraw liquidity from.

/// - `pool_id`: Identifier for the pool to withdraw liquidity from.

Chralt98 · 2024-12-23T11:11:50Z

zrml/neo-swaps/src/lib.rs

+            if keep.is_empty() {
+                ensure!(amount_keep.is_zero(), Error::<T>::InvalidAmountKeep);
+            } else {
+                ensure!(amount_keep < amount_buy, Error::<T>::InvalidAmountKeep);


Chralt98 · 2024-12-23T11:35:05Z

zrml/neo-swaps/src/lib.rs

+                // Ensure that numerical limits of all prices are respected.
+                for &asset in pool.assets().iter() {
+                    let spot_price = pool.calculate_spot_price(asset)?;
+                    ensure!(
+                        spot_price >= COMBO_MIN_SPOT_PRICE.saturated_into(),
+                        Error::<T>::NumericalLimits(NumericalLimitsError::SpotPriceSlippedTooLow)
+                    );
+                    ensure!(
+                        spot_price <= COMBO_MAX_SPOT_PRICE.saturated_into(),
+                        Error::<T>::NumericalLimits(NumericalLimitsError::SpotPriceSlippedTooHigh)
+                    );
+                }


This is duplicated from above.

Chralt98 · 2025-01-02T08:00:32Z

zrml/neo-swaps/src/lib.rs

                let swap_amount_out =
-                    pool.calculate_swap_amount_out_for_buy(asset_out, amount_in_minus_fees)?;
+                    pool.calculate_swap_amount_out_for_buy(buy, sell, amount_in_minus_fees)?;
                let amount_out = swap_amount_out.checked_add_res(&amount_in_minus_fees)?;


Maybe it's worth commenting here that swap_amount_out is just y(x), means the amount of assets in sell (S) that were sold for more assets of buy (B). That's why amount_out = swap_amount_out + amount_in_minus_fees, which is the buy complete set amount plus the additional amount that was received through the sale of the unwanted outcomes in sell.

Chralt98 · 2025-01-02T10:16:11Z

zrml/neo-swaps/src/tests/combo_buy.rs

+    vec![1, 3, 4],
+    5_208_333_333_333,
+    6_576_234_413_778,
+    5_000_000_000_000,


This amount_keep isn't checked, because the vector keep_indices is empty. Maybe it's better to use 0 and add a comment.

Chralt98 · 2025-01-02T10:34:28Z

zrml/neo-swaps/src/tests/combo_buy.rs

+}
+
+#[test]
+fn combo_buy_fails_on_market_not_found() {


Rename to combo_buy_fails_on_pool_not_found.

Chralt98 · 2025-01-02T13:11:27Z

zrml/neo-swaps/src/tests/deploy_combinatorial_pool.rs

+                6,
+                market_ids,
+                _10,
+                vec![20 * CENT; 5],


Comment here that the above markets produce 2 * 3 = 6 collections, instead of 5: LONG & 0, LONG & 1, LONG & 2, SHORT & 0, SHORT & 1, SHORT & 2.

Chralt98 · 2025-01-02T13:21:58Z

zrml/neo-swaps/src/tests/deploy_combinatorial_pool.rs

+        assert_noop!(
+            NeoSwaps::deploy_combinatorial_pool(
+                RuntimeOrigin::signed(BOB),
+                2,
+                vec![market_id],
+                liquidity,
+                vec![_3_4, _1_4],
+                CENT,
+                false,
+            ),
+            expected_error
+        );


Make clear to the reader that BOB has no sufficient funds as opposed to ALICE.

Chralt98 · 2025-01-02T14:00:56Z

zrml/neo-swaps/src/benchmarking.rs

+
+        // We don't care about being precise here and just deposit a huge bunch of tokens for Bob.
+        for &asset in assets.iter() {
+            let amount_for_bob = amount_buy.max(amount_buy);


This is always amount_buy.

Chralt98 · 2025-01-03T11:15:18Z

zrml/neo-swaps/src/math/types/combo_math.rs

+        let numerator = exp_sum_buy
+            .checked_add(exp_sum_sell)?
+            .checked_sub(exp_of_minus_amount_in_times_exp_sum_sell)?;


Maybe it's good to comment here that exp_sum_buy + exp_sum_sell = 1 - exp_sum_keep to arrive at the formula of the ZIP-10 document.

Chralt98 · 2025-01-13T12:49:42Z