Update synthesis docs

docs/synthesis/Synthesis.md

@@ -1,49 +1,162 @@
-# Synthesis
+# Core verification
 
-CoreBlocks synthesizes `Core` circuit to test how many resources it consumes as the project
-grows and more functionalities are added.
+Coreblocks is verified at several levels of abstraction. Beside of unit tests and module tests, we also
+synthesise the core to the ECP5 FPGA target, to check that it can work in reality. Performance is verified
+using synthesis results and a set of benchmarks simulated with cycle precision in cocotb. We also verify
+correctness of the core behaviour by running assembler tests from [riscv-tests](https://github.com/riscv-software-src/riscv-tests/tree/master)
+and [riscv-arch-tests](https://github.com/riscv-non-isa/riscv-arch-test).
 
-## Documentation
+These three verification steps are automatically run by CI on every commit delivered to the `master` branch. Running
+the checks in CI allow us to collect historical data, which are available in the form of the graphs
+on a dedicated [benchmark subpage](https://kuznia-rdzeni.github.io/coreblocks/dev/benchmark/).
 
-### Requirements
+In CI we use pre-built docker containers, which are publicly available on our [github page](https://github.com/orgs/kuznia-rdzeni/packages).
+In the following subsections we provide the instructions on how to manually run verification steps using these containers.
+They can be recreated using standard docker build commands:
 
-In order to perform synthesis you will need to install following tools:
-  * [yosys](https://github.com/YosysHQ/yosys)
-  * [prjtrellis](https://github.com/YosysHQ/prjtrellis)
-  * [nextpnr-ecp5](https://github.com/YosysHQ/nextpnr.git)
-
-These tools may need manual compilation from git repository, that can take some time.
-
-You can use docker images that have installed all required tools to perform synthesis:
-  * [vuush/amaranth-synth:ecp5](https://hub.docker.com/r/vuush/amaranth-synth/tags)
-
-To build the `AmaranthSynthECP5.Dockerfile` yourself use following command:
 ```
 docker build --platform linux/amd64 -t "amaranth-synth:ecp5" -f ./docker/AmaranthSynthECP5.Dockerfile .
 ```
 
-### Usage
-
-Script named `synthesize.py` is used to perform the `Core` synthesis.
-
-Example usage:
-```
-./scripts/synthesize.py --help
-./scripts/synthesize.py --platform ecp5 --verbose
-```
+## Synthesis
 
-To collect synthesis information we use script named `parse_benchmark_info.py`.
+The basic step in verification is to see if it is possible to synthesise the `Core` circuit. This allows us to
+control the level of complexity of the core. Although Coreblocks is an educational core, we want it to be practical, not theoretical,
+so it should have an acceptable maximum frequency and shouldn't use too many resources, so it can be run
+on a FPGA. The synthesis step ensures that these requirements are met. In addition, it checks whether the code that is acceptable
+for the Amaranth is also acceptable for the synthesis tools.
 
-This script parses the output of the synthesis tool and extracts the
-following information:
+The main properties collected in the synthesis step:
   - Max clock frequency
   - Number of logic cells used
   - Number of carry cells used
   - Number of RAM cells used
   - Number of DFF cells used
 
-## Benchmarks
+The configuration of the docker container is described in the `AmaranthSynthECP5.Dockerfile`, which can be found in 
+[our repo](https://github.com/orgs/kuznia-rdzeni/packages/container/package/amaranth-synth).
+
+### Manual reproduction
+
+```bash
+sudo docker pull ghcr.io/kuznia-rdzeni/amaranth-synth:ecp5-3.11
+sudo docker run -it --rm ghcr.io/kuznia-rdzeni/amaranth-synth:ecp5-3.11
+git clone --depth=1 https://github.com/kuznia-rdzeni/coreblocks.git
+cd coreblocks
+apt update
+apt install python3.11-venv
+python3 -m venv venv
+. venv/bin/activate
+python3 -m pip install --upgrade pip
+pip3 install -r requirements-dev.txt
+PYTHONHASHSEED=0 ./scripts/synthesize.py --verbose --config full
+./scripts/parse_benchmark_info.py
+cat benchmark.json
+```
+
+The main point of the above listing is the `synthesize.py` script, which creates an instance of the `Core` object with
+the configuration provided by the user and then passes it to the Amaranth to generate a Verilog description from that instance.
+This description is then processed by the Yosys and nextpnr-ecp5 to generate the ECP5 bitstream.
+
+The strength of the Coreblocks is its modularity, so we can provide different configurations with little effort. You can choose
+a configuration to synthesise using the `--config` argument to the `synthesise.py` script.
 
-For each commit on `master` branch, CI runs the synthesis and saves the parameters collected by `parse_benchmark_info` script.
+### Dependencies
 
-Graphs generated from this information are available on a dedicated [subpage](https://kuznia-rdzeni.github.io/coreblocks/dev/benchmark/).
+In order to perform synthesis we use:
+  * [yosys](https://github.com/YosysHQ/yosys) - to synthesise the Verilog generated by Amaranth up to gate level
+  * [nextpnr-ecp5](https://github.com/YosysHQ/nextpnr.git) - to perform the "Place and Route" step
+  * [prjtrellis](https://github.com/YosysHQ/prjtrellis) - provides the description of the ECP5 bitstream format
+
+## Benchmarking
+
+The maximum clock frequency determined by synthesis isn't the only measure of performance. Theoretically, there is always a
+possibility to increase Fmax by increasing the latency. To avoid the pitfall of too big latency, we have introduced the monitoring
+of instructions executed per clock cycle (IPC). This is done by simulating the core with cycle accuracy and running
+benchmarks written in C on such core. As benchmarking programs we use
+[embench](https://github.com/embench/embench-iot/tree/master).
+
+The benchmarking is done in two steps. First, we compile C programs into binary format and then run the binaries on
+simulated core. The compilation is done using [riscv-gnu-toolchain](https://github.com/riscv/riscv-gnu-toolchain), with
+glibc compiled for different architectural subsets of RISC-V extensions. The configuration of the riscv-gnu-toolchain used in
+Coreblocks is described in [riscv-toolchain.Dockerfile](https://github.com/kuznia-rdzeni/coreblocks/blob/master/docker/riscv-toolchain.Dockerfile).
+Benchmarks can be compiled once and used repeatedly as long as there is no need to add support for the new
+RISC-V extensions or the embench isn't be updated.
+
+Once we have binaries, we can execute them in simulation. This is done with [Cocotb](https://github.com/cocotb/cocotb) and
+[Verilator](https://github.com/verilator/verilator). First we generate Verilog code describing Coreblocks instance.
+Then it is passed to Verilator for compilation and Cocotb controls the execution of the program, by stubbing external
+interfaces. Compiled Verilator in a compatible version is available in [Verilator.Dockerfile](https://github.com/kuznia-rdzeni/coreblocks/blob/master/docker/Verilator.Dockerfile).
+
+### Benchmarks manual execution
+```bash
+# ========== STEP 1: Compilation ==========
+# Clone coreblocks into host file system
+git clone --depth=1 https://github.com/kuznia-rdzeni/coreblocks.git
+cd coreblocks
+git submodule update --init --recursive
+cd ..
+sudo docker pull ghcr.io/kuznia-rdzeni/riscv-toolchain:2023.10.08_v
+# Run docker with the coreblocks directory mounted into it
+sudo docker run -v ./coreblocks:/coreblocks -it --rm ghcr.io/kuznia-rdzeni/riscv-toolchain:2023.10.08_v
+cd /coreblocks/test/external/embench
+# Compilation will put binaries in the subdirectory of the /coreblocks directory, which is shared with the host
+# so that binaries survive after the docker container is closed
+make
+exit
+
+# ========== STEP 2: Execution ==========
+sudo docker pull ghcr.io/kuznia-rdzeni/verilator:v5.008-3.11
+# Run docker with the coreblocks directory mounted into it. This directory contains
+# benchmark binaries after running the first step.
+sudo docker run -v ./coreblocks:/coreblocks -it --rm ghcr.io/kuznia-rdzeni/verilator:v5.008-3.11
+apt update
+apt install python3.11-venv
+python3 -m venv venv
+. venv/bin/activate
+python3 -m pip install --upgrade pip
+cd coreblocks
+pip3 install -r requirements-dev.txt
+PYTHONHASHSEED=0 ./scripts/gen_verilog.py --verbose --config full
+./scripts/run_benchmarks.py
+```
+
+## Regression tests
+
+Regression tests should ensure that Coreblocks is complaint with RISC-V specification requirements. Tests include 
+assembler programs that tests entire RISC-V instruction set. We execute these programs in a similar way to benchmarks.
+So, as a first step, we compile the programs to the binary format and then we run them on core simulated by Verilator
+and Cocotb.
+
+### Regression tests manual execution
+```bash
+# ========== STEP 1: Compilation ==========
+# Clone coreblocks into host file system
+git clone --depth=1 https://github.com/kuznia-rdzeni/coreblocks.git
+cd coreblocks
+git submodule update --init --recursive
+cd ..
+sudo docker pull ghcr.io/kuznia-rdzeni/riscv-toolchain:2023.10.08_v
+# Run docker with the coreblocks directory mounted into it
+sudo docker run -v ./coreblocks:/coreblocks -it --rm ghcr.io/kuznia-rdzeni/riscv-toolchain:2023.10.08_v
+cd /coreblocks/test/external/riscv-tests
+# Compilation will put binaries in the subdirectory of the /coreblocks directory, which is shared with the host
+# so that binaries survive after the docker container is closed
+make
+exit
+
+# ========== STEP 2: Execution ==========
+sudo docker pull ghcr.io/kuznia-rdzeni/verilator:v5.008-3.11
+# Run docker with the coreblocks directory mounted into it. This directory contains
+# regression test binaries after running the first step.
+sudo docker run -v ./coreblocks:/coreblocks -it --rm ghcr.io/kuznia-rdzeni/verilator:v5.008-3.11
+apt update
+apt install python3.11-venv
+python3 -m venv venv
+. venv/bin/activate
+python3 -m pip install --upgrade pip
+cd coreblocks
+pip3 install -r requirements-dev.txt
+PYTHONHASHSEED=0 ./scripts/gen_verilog.py --verbose --config full
+./scripts/run_tests.py -a regression
+```