diff --git a/README.md b/README.md index f4faf76b..0e89dcf0 100644 --- a/README.md +++ b/README.md @@ -5,8 +5,7 @@ This repository contains the implementation of The Innovation Game (TIG). ## Important Links * [TIG's tech explainer](docs/1_basics.md) -* [Getting started with Innovating](tig-algorithms/README.md) -* [Getting started with Benchmarking](tig-benchmarker/README.md) +* [Getting Started with Innovating](docs/1_basics.md) * [Challenge descriptions](tig-challenges/docs/knapsack.md) ## Repo Contents @@ -64,6 +63,166 @@ A Rust crate for verifying and computing solutions. Solutions are computed by executing an algorithm in a WASM virtual machine ([TIG's fork of wasmi](https://github.com/tig-foundation/wasmi)). +## Getting Started with Innovating + +### Setting up Private Fork + +Innovators will want to create a private fork so that they can test that their algorithm can be successfully compiled into WASM by the CI. + +1. Create private repository on GitHub +2. Create empty git repository on your local machine + ``` + mkdir tig-monorepo + cd tig-monorepo + git init + ``` +3. Setup remotes with origin pointed to your private repository + ``` + git remote add origin + git remote add public https://github.com/tig-foundation/tig-monorepo.git + ``` + +4. Pulling `blank_slate` from TIG public repository (branch with no algorithms) + ``` + git fetch public + git checkout -b blank_slate + git pull public blank_slate + ``` + +5. Push to your private repository + ``` + git push origin blank_slate + ``` + +### Checking out Existing Algorithms + +Every algorithm has its own `` with name `/`. + +Only algorithms that are successfully compiled into WASM have their branch pushed to this public repository. + +Each algorithm branch will have 2 files: +1. Rust code @ `tig-algorithms/src/.rs` +2. Wasm blob @ `tig-algorithms/wasm/.wasm` + +To pull an existing algorithm from TIG public repository, run the following command: +``` +git fetch public +git pull public +``` + +### Developing Your Algorithm + +1. Pick a challenge (``) to develop an algorithm for +2. Make a copy of an existing algorithm's rust code or `tig-algorithms//template.rs` +3. Rename the file with your own `` +4. Edit `tig-algorithms//mod.rs` to export your algorithm and test it: + ``` + pub mod ; + + #[cfg(test)] + mod tests { + use super::*; + use tig_challenges::{::*, *}; + + #[test] + fn test_() { + let difficulty = Difficulty { + // Uncomment the relevant fields. + + // -- satisfiability -- + // num_variables: 50, + // clauses_to_variables_percent: 300, + + // -- vehicle_routing -- + // num_nodes: 40, + // better_than_baseline: 250, + + // -- knapsack -- + // num_items: 50, + // better_than_baseline: 10, + }; + let seed = 0; + let challenge = Challenge::generate_instance(seed, &difficulty).unwrap(); + ::solve_challenge(&challenge).unwrap(); + } + } + ``` +5. Check that your algorithm compiles & runs: + ``` + cargo test -p tig-algorithms + ``` + +Notes: +* Do not include tests in your algorithm file. TIG will reject your algorithm submission. +* Only your algorithm's rust code gets submitted. You should not be adding dependencies to `tig-algorithms` as they will not be available when TIG compiles your algorithm + +### Locally Compiling Your Algorithm into WASM + +These steps replicate what TIG's CI does (`.github/workflows/build_algorithm.yml`): + +1. Set environment variables to match the algorithm you are compiling: + ``` + export CHALLENGE= + export ALGORITHM= + ``` +2. Compile your algorithm + ``` + cargo build -p tig-wasm --target wasm32-wasi --release --features entry-point + ``` +3. Optimise the WASM and save it into `tig-algorithms/wasm`: + ``` + mkdir -p tig-algorithms/wasm/${CHALLENGE} + wasm-opt target/wasm32-wasi/release/tig_wasm.wasm -o tig-algorithms/wasm/${CHALLENGE}/${ALGORITHM}.wasm -O2 --remove-imports + ``` + +### Testing Performance of Algorithms + +Performance testing is done by `tig-worker` in a sandboxed WASM Virtual Machine. + +**IMPORTANT**: You can compile / test existing algorithms as binary executables, but be sure to throughly vet the code beforehand for malicious routines! + +1. Pull an existing algorithm or compile your own algorithm to WASM +2. Set environment variables to match the algorithm you are testing: + ``` + export CHALLENGE= + export ALGORITHM= + ``` +3. Pick a difficulty & create `settings.json`: + ``` + { + "block_id": "", + "algorithm_id": "", + "challenge_id": "", + "player_id": "", + "difficulty": [50, 300] + } + ``` +4. Test the algorithm: + ``` + cargo run -p tig-worker --release -- settings.json tig-algorithms/wasm/${CHALLENGE}/${ALGORITHM}.wasm + ``` + +Notes: +* You can query the latest difficulty ranges via TIG's API: + ``` + query https://api.tig.foundation/play/get-block for + query https://api.tig.foundation/play/get-challenges?block_id= for qualifier_difficulties + ``` + +### Checking CI Successfully Compiles Your Algorithm + +TIG pushes all algorithms to their own branch which triggers the CI (`.github/workflows/build_algorithm.yml`). + +To trigger the CI on your private repo, your branch just needs to have a particular name: +``` +git checkout -b / +git push origin / +``` + +### Making Your Submission + +You will need to burn 0.001 ETH to make a submission. Visit https://play.tig.foundation/innovator and follow the instructions. + ## License Placeholder \ No newline at end of file diff --git a/tig-benchmarker/README.md b/tig-benchmarker/README.md deleted file mode 100644 index e79df0a0..00000000 --- a/tig-benchmarker/README.md +++ /dev/null @@ -1,3 +0,0 @@ -# Getting Started with Benchmarking - -Placeholder \ No newline at end of file diff --git a/tig-benchmarker/src/benchmarker.rs b/tig-benchmarker/src/benchmarker.rs index 1b0c8ceb..e8be7b64 100644 --- a/tig-benchmarker/src/benchmarker.rs +++ b/tig-benchmarker/src/benchmarker.rs @@ -11,7 +11,7 @@ use std::collections::{HashMap, HashSet}; use tig_api::*; use tig_structs::{config::WasmVMConfig, core::*}; use tig_utils::*; -use tig_worker::compute_solution; +use tig_worker::{compute_solution, ComputeResult}; type Result = std::result::Result; @@ -560,15 +560,13 @@ async fn do_benchmark() -> Result<()> { blob = download_wasm_blob(&job.settings.algorithm_id).await?; last_algorithm_id = job.settings.algorithm_id.clone(); } - if let Ok(solution_data) = compute_solution( + if let Ok(ComputeResult::ValidSolution(solution_data)) = compute_solution( &job.settings, nonce, blob.as_slice(), job.wasm_vm_config.max_memory, job.wasm_vm_config.max_fuel, - ) - .map_err(|e| e.to_string())? - { + ) { if solution_data.calc_solution_signature() <= job.solution_signature_threshold { let mut state = mutex().lock().await; if let Some(Some(solutions_meta_data)) = (*state) diff --git a/tig-challenges/src/lib.rs b/tig-challenges/src/lib.rs index e228256f..5a58d0a9 100644 --- a/tig-challenges/src/lib.rs +++ b/tig-challenges/src/lib.rs @@ -27,8 +27,10 @@ where } fn verify_solution(&self, solution: &T) -> Result<()>; - fn verify_solution_from_json(&self, solution: &str) -> Result> { - Ok(self.verify_solution(&serde_json::from_str(solution)?)) + fn verify_solution_from_json(&self, solution: &str) -> Result<()> { + let solution = serde_json::from_str(solution) + .map_err(|e| anyhow!("Failed to parse solution: {}", e))?; + self.verify_solution(&solution) } } diff --git a/tig-protocol/src/context.rs b/tig-protocol/src/context.rs index 55b751a0..a7ff337e 100644 --- a/tig-protocol/src/context.rs +++ b/tig-protocol/src/context.rs @@ -1,6 +1,7 @@ pub use anyhow::{Error as ContextError, Result as ContextResult}; use tig_structs::{config::*, core::*}; +#[derive(Debug, Clone, PartialEq)] pub enum SubmissionType { Algorithm, Benchmark, diff --git a/tig-wasm/src/entry_point_template.rs b/tig-wasm/src/entry_point_template.rs index f21afdb0..5786935b 100644 --- a/tig-wasm/src/entry_point_template.rs +++ b/tig-wasm/src/entry_point_template.rs @@ -7,20 +7,25 @@ use tig_utils::compress_obj; #[no_mangle] pub fn entry_point(seed: u32, difficulty: Difficulty, ptr: *mut u8, max_length: usize) { - let challenge = Challenge::generate_instance(seed, &difficulty).expect("Failed to generate challenge"); - if let Ok(Some(solution)) = {ALGORITHM}::solve_challenge(&challenge) { - if challenge.verify_solution(&solution).is_ok() { - let mut buffer = Vec::::new(); - let compressed = compress_obj(solution); - buffer.extend((compressed.len() as u32).to_be_bytes()); - buffer.extend(compressed); - if buffer.len() > max_length { - panic!("Encoded solution exceeds maximum length"); - } - - for (i, &byte) in buffer.iter().enumerate() { - unsafe { *ptr.add(i) = byte }; - } + let challenge = + Challenge::generate_instance(seed, &difficulty).expect("Failed to generate challenge"); + let (is_solution, compressed) = + if let Ok(Some(solution)) = {ALGORITHM}::solve_challenge(&challenge) { + ( + challenge.verify_solution(&solution).is_ok(), + compress_obj(solution), + ) + } else { + (false, Vec::::new()) + }; + let mut buffer = Vec::::new(); + buffer.push(is_solution as u8); + buffer.extend((compressed.len() as u32).to_be_bytes()); + buffer.extend(compressed); + for (i, &byte) in buffer.iter().enumerate() { + if i >= max_length { + break; } + unsafe { *ptr.add(i) = byte }; } } diff --git a/tig-worker/src/main.rs b/tig-worker/src/main.rs index 96bcb182..a492c373 100644 --- a/tig-worker/src/main.rs +++ b/tig-worker/src/main.rs @@ -1,161 +1,92 @@ mod worker; -use anyhow::{anyhow, Result}; use clap::{arg, Command}; -use std::{fs, path::PathBuf, process::exit}; -use tig_structs::core::{BenchmarkSettings, SolutionData}; -use tig_utils::{dejsonify, jsonify}; +use std::{fs, path::PathBuf, time::Instant}; +use tig_structs::core::BenchmarkSettings; +use tig_utils::dejsonify; fn cli() -> Command { - Command::new("rust_cli_app") - .about("CLI app to compute or verify solutions") - .subcommand_required(true) + Command::new("tig_performance_tester") + .about("Tests performance of a TIG algorithm") .arg_required_else_help(true) - .subcommand( - Command::new("compute_solution") - .about("Computes a solution") - .arg( - arg!( "Path to a settings file") - .value_parser(clap::value_parser!(PathBuf)), - ) - .arg(arg!( "A u32 nonce").value_parser(clap::value_parser!(u32))) - .arg(arg!( "Path to a wasm file").value_parser(clap::value_parser!(PathBuf))) - .arg( - arg!(--endless "Optional flag to compute solutions continuously") - .action(clap::ArgAction::SetTrue), - ) - .arg( - arg!(--fuel [FUEL] "Optional maximum fuel parameter for WASM VM") - .default_value("1000000000") - .value_parser(clap::value_parser!(u64)), - ) - .arg( - arg!(--mem [MEM] "Optional maximum memory parameter for WASM VM") - .default_value("1000000000") - .value_parser(clap::value_parser!(u64)), - ) - .arg( - arg!(--debug "Optional flag to print debug messages") - .action(clap::ArgAction::SetTrue), - ), + .arg(arg!( "Path to a settings file").value_parser(clap::value_parser!(PathBuf))) + .arg(arg!( "Path to a wasm file").value_parser(clap::value_parser!(PathBuf))) + .arg( + arg!(--nonce "Starting nonce") + .default_value("0") + .value_parser(clap::value_parser!(u32)), ) - .subcommand( - Command::new("verify_solution") - .about("Verifies a solution") - .arg( - arg!( "Path to a settings file") - .value_parser(clap::value_parser!(PathBuf)), - ) - .arg( - arg!( "Path to a solution file") - .value_parser(clap::value_parser!(PathBuf)), - ) - .arg( - arg!(--debug "Optional flag to print debug messages") - .action(clap::ArgAction::SetTrue), - ), + .arg( + arg!(--fuel [FUEL] "Optional maximum fuel parameter for WASM VM") + .default_value("1000000000") + .value_parser(clap::value_parser!(u64)), + ) + .arg( + arg!(--mem [MEM] "Optional maximum memory parameter for WASM VM") + .default_value("1000000000") + .value_parser(clap::value_parser!(u64)), ) } fn main() { let matches = cli().get_matches(); - let result = match matches.subcommand() { - Some(("compute_solution", sub_matches)) => { - let settings_path = sub_matches.get_one::("SETTINGS").unwrap(); - let nonce = *sub_matches.get_one::("NONCE").unwrap(); - let wasm_path = sub_matches.get_one::("WASM").unwrap(); - let endless = sub_matches.get_flag("endless"); - let max_fuel = *sub_matches.get_one::("fuel").unwrap(); - let max_memory = *sub_matches.get_one::("mem").unwrap(); - let debug = sub_matches.get_flag("debug"); - - compute_solution( - settings_path, - nonce, - wasm_path, - max_memory, - max_fuel, - endless, - debug, - ) - } - Some(("verify_solution", sub_matches)) => { - let settings_path = sub_matches.get_one::("SETTINGS").unwrap(); - let solution_path = sub_matches.get_one::("SOLUTION").unwrap(); - let debug = sub_matches.get_flag("debug"); + let settings_path = matches.get_one::("SETTINGS").unwrap(); + let wasm_path = matches.get_one::("WASM").unwrap(); + let nonce = *matches.get_one::("nonce").unwrap(); + let max_fuel = *matches.get_one::("fuel").unwrap(); + let max_memory = *matches.get_one::("mem").unwrap(); - verify_solution(settings_path, solution_path, debug) - } - _ => unreachable!("The CLI should prevent getting here"), - }; - match result { - Ok(_) => exit(0), - Err(e) => { - println!("Error: {}", e); - exit(1); - } - }; + test_performance(settings_path, wasm_path, nonce, max_memory, max_fuel); } -fn compute_solution( +fn test_performance( settings_path: &PathBuf, - nonce: u32, wasm_path: &PathBuf, + nonce: u32, max_memory: u64, max_fuel: u64, - endless: bool, - debug: bool, -) -> Result<()> { +) { let settings = dejsonify::( - &fs::read_to_string(settings_path) - .map_err(|e| anyhow!("Failed to read settings file: {}", e))?, + &fs::read_to_string(settings_path).expect("Failed to read settings file"), ) - .map_err(|e| anyhow!("Failed to dejsonify settings file: {}", e))?; + .expect("Failed to dejsonify settings file"); - let wasm = fs::read(wasm_path).map_err(|e| anyhow!("Failed to read wasm file: {}", e))?; + let wasm = fs::read(wasm_path).expect("Failed to read wasm file"); + println!("Algorithm: {:?}", wasm_path); + println!("Settings: {:?}", settings); + let start = Instant::now(); + let mut num_solutions = 0u32; + let mut num_errors = 0u32; let mut i = 0; + let mut last_update = 0f64; loop { - let result = - worker::compute_solution(&settings, nonce + i, wasm.as_slice(), max_memory, max_fuel)?; - match result { - Ok(solution_data) => { - println!("{}", jsonify(&solution_data)); + match worker::compute_solution(&settings, nonce + i, wasm.as_slice(), max_memory, max_fuel) + { + Ok(worker::ComputeResult::ValidSolution(_)) => { + num_solutions += 1; } - Err(e) => { - if debug { - println!("Nonce {}, no solution: {}", nonce + i, e); - } + Err(_) => { + num_errors += 1; } + _ => {} } i += 1; - if !endless { - break; - } - } - - Ok(()) -} - -fn verify_solution(settings_path: &PathBuf, solution_path: &PathBuf, debug: bool) -> Result<()> { - let settings = dejsonify::( - &fs::read_to_string(settings_path) - .map_err(|e| anyhow!("Failed to read settings file: {}", e))?, - ) - .map_err(|e| anyhow!("Failed to dejsonify settings file: {}", e))?; - let solution_data = dejsonify::( - fs::read_to_string(solution_path) - .map_err(|e| anyhow!("Failed to read solution file: {}", e))? - .as_str(), - ) - .map_err(|e| anyhow!("Failed to dejsonify solution: {}", e))?; - let result = worker::verify_solution(&settings, solution_data.nonce, &solution_data.solution)?; - if debug { - if let Err(e) = result { - println!("Solution is invalid: {}", e); - } else { - println!("Solution is valid"); + let elapsed = start.elapsed().as_micros() as f64 / 1000.0; + if elapsed >= last_update + 1000.0 { + let num_invalid_solutions = i - num_solutions - num_errors; + last_update = elapsed; + println!( + "#instances: {}, #solutions: {} ({:.1}%), #invalid_solutions: {} ({:.1}%), #errors: {} ({:.1}%), avg_time_per_solution: {}ms", + i, + num_solutions, + num_solutions as f64 / i as f64 * 100.0, + num_invalid_solutions, + num_invalid_solutions as f64 / i as f64 * 100.0, + num_errors, + num_errors as f64 / i as f64 * 100.0, + if num_solutions == 0 { 0f64 } else { elapsed / num_solutions as f64 } + ); } } - Ok(()) } diff --git a/tig-worker/src/worker.rs b/tig-worker/src/worker.rs index b6f0c795..7d96c3a0 100644 --- a/tig-worker/src/worker.rs +++ b/tig-worker/src/worker.rs @@ -1,21 +1,31 @@ use anyhow::{anyhow, Result}; use tig_challenges::{knapsack, satisfiability, vehicle_routing, ChallengeTrait}; -use tig_structs::core::*; +pub use tig_structs::core::{BenchmarkSettings, Solution, SolutionData}; use tig_utils::decompress_obj; use wasmi::{Config, Engine, Linker, Module, Store, StoreLimitsBuilder}; const BUFFER_SIZE: usize = u16::MAX as usize; +#[derive(Debug, Clone, PartialEq)] +pub enum ComputeResult { + NoSolution(SolutionData), + InvalidSolution(SolutionData), + ValidSolution(SolutionData), + SolutionTooLarge(SolutionData), +} + pub fn compute_solution( settings: &BenchmarkSettings, nonce: u32, wasm: &[u8], max_memory: u64, max_fuel: u64, -) -> Result> { - if settings.difficulty.len() != 2 { - return Err(anyhow!("Unsupported difficulty length")); - } +) -> Result { + assert_eq!( + settings.difficulty.len(), + 2, + "Unsupported difficulty length" + ); let mut config = Config::default(); config.update_runtime_signature(true); @@ -31,32 +41,31 @@ pub fn compute_solution( let mut store = Store::new(&engine, limits); store.add_fuel(max_fuel).unwrap(); let linker = Linker::new(&engine); - let module = Module::new(store.engine(), wasm) - .map_err(|e| anyhow!("Failed to instantiate module: {}", e))?; + let module = Module::new(store.engine(), wasm).expect("Failed to instantiate module"); let instance = &linker .instantiate(&mut store, &module) - .map_err(|e| anyhow!("Failed to instantiate linker: {}", e))? + .expect("Failed to instantiate linker") .start(&mut store) - .map_err(|e| anyhow!("Failed to start module: {}", e))?; + .expect("Failed to start module"); // Create memory for entry_point to write solution to let mut buffer = [0u8; BUFFER_SIZE]; let memory = instance .get_memory(&store, "memory") - .ok_or_else(|| anyhow!("Failed to find memory"))?; + .expect("Failed to find memory"); memory .write(&mut store, 0, &buffer) - .map_err(|e| anyhow!("Failed to write to memory: {}", e))?; + .expect("Failed to write to memory"); // Run algorithm let func = instance .get_func(&store, "entry_point") - .ok_or_else(|| anyhow!("Failed to find entry_point"))?; + .expect("Failed to find entry_point"); let seed = settings.calc_seed(nonce); store.set_runtime_signature(seed as u64); if let Err(e) = func .typed::<(u32, i32, i32, i32, i32), ()>(&store) - .map_err(|e| anyhow!("Failed to instantiate function: {}", e))? + .expect("Failed to instantiate function") .call( &mut store, ( @@ -68,7 +77,7 @@ pub fn compute_solution( ), ) { - return Ok(Err(anyhow!("Error occured during execution: {}", e))); + return Err(anyhow!("Error occured during execution: {}", e)); } // Get runtime signature let runtime_signature_u64 = store.get_runtime_signature(); @@ -77,87 +86,68 @@ pub fn compute_solution( // Read solution from memory memory .read(&store, 0, &mut buffer) - .map_err(|e| anyhow!("Failed to read from memory: {}", e))?; - let solution_len = u32::from_be_bytes(buffer[0..4].try_into().unwrap()) as usize; + .expect("Failed to read from memory"); + let valid_solution = buffer[0] == 1; + let solution_len = u32::from_be_bytes(buffer[1..5].try_into().unwrap()) as usize; + let mut solution_data = SolutionData { + nonce, + runtime_signature, + fuel_consumed, + solution: Solution::new(), + }; if solution_len == 0 { - return Ok(Err(anyhow!( - "No solution found (runtime_signature: {}, fuel_consumed: {})", - runtime_signature, - fuel_consumed - ))); + return Ok(ComputeResult::NoSolution(solution_data)); } - if solution_len > BUFFER_SIZE - 4 { - return Ok(Err(anyhow!( - "Solution too large (solution_size: {}, runtime_signature: {}, fuel_consumed: {})", - solution_len, - runtime_signature, - fuel_consumed - ))); + if solution_len > BUFFER_SIZE - 5 { + return Ok(ComputeResult::SolutionTooLarge(solution_data)); } - let solution = decompress_obj(&buffer[4..4 + solution_len]) - .map_err(|e| anyhow!("Failed to convert buffer to solution: {}", e.to_string()))?; + solution_data.solution = + decompress_obj(&buffer[5..5 + solution_len]).expect("Failed to convert buffer to solution"); - Ok(Ok(SolutionData { - nonce, - runtime_signature, - fuel_consumed, - solution, - })) + match valid_solution { + true => Ok(ComputeResult::ValidSolution(solution_data)), + false => Ok(ComputeResult::InvalidSolution(solution_data)), + } } pub fn verify_solution( settings: &BenchmarkSettings, nonce: u32, solution: &Solution, -) -> Result> { +) -> Result<()> { let seed = settings.calc_seed(nonce); match settings.challenge_id.as_str() { "c001" => { let challenge = satisfiability::Challenge::generate_instance_from_vec(seed, &settings.difficulty) - .map_err(|e| { - anyhow!( - "satisfiability::Challenge::generate_instance_from_vec error: {}", - e - ) - })?; + .expect("Failed to generate satisfiability instance"); match satisfiability::Solution::try_from(solution.clone()) { - Ok(solution) => Ok(challenge.verify_solution(&solution)), - Err(_) => Ok(Err(anyhow!( + Ok(solution) => challenge.verify_solution(&solution), + Err(_) => Err(anyhow!( "Invalid solution. Cannot convert to satisfiability::Solution" - ))), + )), } } "c002" => { let challenge = vehicle_routing::Challenge::generate_instance_from_vec(seed, &settings.difficulty) - .map_err(|e| { - anyhow!( - "vehicle_routing::Challenge::generate_instance_from_vec error: {}", - e - ) - })?; + .expect("Failed to generate vehicle_routing instance"); match vehicle_routing::Solution::try_from(solution.clone()) { - Ok(solution) => Ok(challenge.verify_solution(&solution)), - Err(_) => Ok(Err(anyhow!( + Ok(solution) => challenge.verify_solution(&solution), + Err(_) => Err(anyhow!( "Invalid solution. Cannot convert to vehicle_routing::Solution" - ))), + )), } } "c003" => { let challenge = knapsack::Challenge::generate_instance_from_vec(seed, &settings.difficulty) - .map_err(|e| { - anyhow!( - "knapsack::Challenge::generate_instance_from_vec error: {}", - e - ) - })?; + .expect("Failed to generate knapsack instance"); match knapsack::Solution::try_from(solution.clone()) { - Ok(solution) => Ok(challenge.verify_solution(&solution)), - Err(_) => Ok(Err(anyhow!( + Ok(solution) => challenge.verify_solution(&solution), + Err(_) => Err(anyhow!( "Invalid solution. Cannot convert to knapsack::Solution" - ))), + )), } } _ => panic!("Unknown challenge"),