[BUG] <title> riscv-test with virtual memory enabled fails on target cv64a6_imafdc_sv39_hpdcache in write-back mode

[Bill94l]

Is there an existing CVA6 bug for this?

• I have searched the existing bug issues

Bug Description

Hi @cfuguet,

I encountered an issue while running tests for the target cv64a6_imafdc_sv39_hpdcache with the HPD-Cache configured in write-back mode. Below are the steps and observations:

Steps to Enable Write-Back Mode

To enable write-back mode, I modified the following parameters in the file core/cache_subsystem/cva6_hpdcache_subsystem.sv:

flushEntries: 4,
flushFifoDepth: 2,
wtEn: 1'b0,
wbEn: 1'b1

Observations

1. Tests Passed:
   All tests with virtual memory disabled pass successfully, including:

   • riscv-compliance-cv64a6_imafdc_sv39
   • riscv-arch-test-cv64a6_imafdc_sv39
   • riscv-tests-cv64a6_imafdc_sv39-p

2. Tests Failing:
   Tests with virtual memory enabled (e.g., riscv-tests-cv64a6_imafdc_sv39-v) fail. The simulation crashes within the design of the Verilator model, leading to a timeout.

Example Failure

For instance, when running the rv64ui-v-add test as part of the smoke-tests with the target cv64a6_imafdc_sv39_hpdcache in write-back mode, the test fails as described above.

Fri, 15 Nov 2024 10:27:25 INFO     Iteration number: 1
Fri, 15 Nov 2024 10:27:25 INFO     Processing regression test list : ../tests/testlist_riscv-tests-cv64a6_imafdc_sv39-v.yaml, test: rv64ui-v-add
Fri, 15 Nov 2024 10:27:25 INFO     Found matched tests: rv64ui-v-add, iterations:1
Fri, 15 Nov 2024 10:27:25 INFO     CVA6 Configuration is  and target is cv64a6_imafdc_sv39_hpdcache
Fri, 15 Nov 2024 10:27:25 INFO     Compiling assembly test: cva6/verif/tests/riscv-tests/isa/rv64ui/add.S
RISCV/lib/gcc/riscv-none-elf/13.1.0/../../../../riscv-none-elf/bin/ld: warning: cva6/verif/sim/out_2024-11-15/directed_asm_tests/add.o has a LOAD segment with RWX permissions
Fri, 15 Nov 2024 10:27:25 INFO     Converting to cva6/verif/sim/out_2024-11-15/directed_asm_tests/add.bin
Fri, 15 Nov 2024 10:27:25 INFO     Processing ISS setup file: cva6.yaml
Fri, 15 Nov 2024 10:27:25 INFO     Found matching ISS: spike
Fri, 15 Nov 2024 10:27:25 INFO     Target: cv64a6_imafdc_sv39_hpdcache
Fri, 15 Nov 2024 10:27:25 INFO     ISA rv64gc_zba_zbb_zbs_zbc
Fri, 15 Nov 2024 10:27:25 INFO     [spike] Running ISS simulation: make spike steps=2000000 target=cv64a6_imafdc_sv39_hpdcache variant=rv64gc_zba_zbb_zbs_zbc priv=msu elf=cva6/verif/sim/out_2024-11-15/directed_asm_tests/add.o tool_path=cva6/tools/spike/bin log=cva6/verif/sim/out_2024-11-15/spike_sim/rv64ui-v-add.cv64a6_imafdc_sv39_hpdcache.log spike_params='' &> cva6/verif/sim/out_2024-11-15/spike_sim/rv64ui-v-add.cv64a6_imafdc_sv39_hpdcache.log.iss
Fri, 15 Nov 2024 10:27:27 INFO     [spike] Running ISS simulation: cva6/verif/sim/out_2024-11-15/directed_asm_tests/add.o ...done
Fri, 15 Nov 2024 10:27:27 INFO     Processing ISS setup file: cva6.yaml
Fri, 15 Nov 2024 10:27:27 INFO     Found matching ISS: veri-testharness
Fri, 15 Nov 2024 10:27:27 INFO     Target: cv64a6_imafdc_sv39_hpdcache
Fri, 15 Nov 2024 10:27:27 INFO     ISA rv64gc_zba_zbb_zbs_zbc
Fri, 15 Nov 2024 10:27:27 INFO     [veri-testharness] Running ISS simulation: make veri-testharness target=cv64a6_imafdc_sv39_hpdcache variant=rv64gc_zba_zbb_zbs_zbc elf= cva6/verif/sim/out_2024-11-15/directed_asm_tests/add.o path_var=cva6/ tool_path=cva6/tools/spike/bin isscomp_opts="" issrun_opts="+debug_disable=1+UVM_VERBOSITY=UVM_NONE +ntb_random_seed=1" isspostrun_opts="0x0000000080000000" log=cva6/verif/sim/out_2024-11-15/veri-testharness_sim/rv64ui-v-add.cv64a6_imafdc_sv39_hpdcache.log &>cva6/verif/sim/out_2024-11-15/veri-testharness_sim/rv64ui-v-add.cv64a6_imafdc_sv39_hpdcache.log.iss
Fri, 15 Nov 2024 10:34:40 INFO     
Fri, 15 Nov 2024 10:34:40 ERROR    ERROR return code: True/2, cmd:make veri-testharness target=cv64a6_imafdc_sv39_hpdcache variant=rv64gc_zba_zbb_zbs_zbc elf= cva6/verif/sim/out_2024-11-15/directed_asm_tests/add.o path_var= cva6/ tool_path=cva6/tools/spike/bin isscomp_opts="" issrun_opts="+debug_disable=1+UVM_VERBOSITY=UVM_NONE +ntb_random_seed=1" isspostrun_opts="0x0000000080000000" log=cva6/verif/sim/out_2024-11-15/veri-testharness_sim/rv64ui-v-add.cv64a6_imafdc_sv39_hpdcache.log &> cva6/verif/sim/out_2024-11-15/veri-testharness_sim/rv64ui-v-add.cv64a6_imafdc_sv39_hpdcache.log.iss

Let me know if additional details or logs are required!

[cfuguet]

Hello @Bill94l, thank you for reporting the issue.

I will take a look on it, and let you know.

As you can see, the write-back mode of the HPDcache is not yet officially supported on the CVA6 as there is no a corresponding configuration, so you may have some issues. But anyway, It should in principle work.

[cfuguet]

Ok, I diagnosed the issue but I did not yet fixed it. It is not a very simple one. It is an integration bug: hpdcache in WB mode within the CVA6.

It is a cache coherency issue between the Icache and the Dcache (HPDcache). The runtime in the v-tests (tests with virtual memory), modifies instruction segments. These modifications are done locally in the HPDcache (because of the write-back mode), and are not visible by the instruction cache unless the HPDcache is flushed (this is not required when in write-through mode because the memory is always updated on writes). The runtime explicitly does this using a fence.i instruction, but currently this instruction only requests an invalidation of the Icache with no flush of the Dcache.

I need to modify the CVA6 controller to issue a CMO flush operation to the HPDCache in case of a fence.i instruction, or find another clever way to keep both caches coherent... I have some ideas but I will think it over a little more.

You will need to be a little patient for this one.