Passing and failing riscv-tests/debug on v0.11 - request for confirmation #869
Assignees
Comments
|
Hi @JanMatCodasip - I have the same board so might be able to repeat these tests for you but I cannot find anything in this repo explaining how to run the (a?) test suite. Maybe you or somebody else can point me in the right direction please? Thanks. Edit: oh - perhaps this (in a different repo)? |
|
@TommyMurphyTM1234 You found the tests. :-) If the README there isn't sufficient to let you run them, let me know and I'll try to make it more clear. @JanMatCodasip This list looks alright to me. When I run the HiFive1 tests (rarely), I'm generally happy as long as most of the tests are passing, because there is very little hardware out there that implements 0.11 (I only know of HiFive1). It would be nice to have a list of known not-passing tests for proper regression testing. |
@timsifive Thank you for looking at this. Unless you or @TommyMurphyTM1234 disagree with some items in the table within a week or so, I will take the table above as the base for our regression testing of v0.11 debug support. |
|
That sounds good to me. I assume you'll then commit a new skip file using the mechanism from riscv-software-src/riscv-tests#477 |
|
Sure, we can create one such file with the known failing tests on HiFive1 and commit it to riscv-tests. |
Hi @JanMatCodasip - I still plan to perform the tests for comparison with your results - hopefully today. Apologies for the delay but other stuff cropped up recently. |
Hi @timsifive - do the tests only really work if all tools (RISC-V tools, Spike, OpenOCD) are installed into Edit: OK - I just copied the Edit 2: hmmm - spoke too soon... |
|
There should be no need to have them in the system path. I have all my tools under /opt/riscv. They are in my PATH. Etrigger test is expected to fail on HiFive1 because it does not implement exception triggers. This should really be resolved by adding a property in testlib.Target the says whether etrigger is supported, checking for that in EtriggerTest, and setting the property for the spike targets. |
|
Hi @timsifive - thanks for the quick reply.
I had them in my
I'm not yet running against the HiFive1 - I'm running the smoke test against Spike.
|
|
@JanMatCodasip - maybe you can tell me exactly how you ran your tests so that I can try to replicate them? |
|
https://github.com/riscv/riscv-openocd/blob/riscv/.github/workflows/spike-openocd-tests.yml might serve as a concrete set of instructions that are run regularly. |
I tried running the 32-bit tests: as follows (all tools are on my but I get this: Most or all tests generate these Am I supposed to do something special when compiling Spike and/or OpenOCD? OpenOCD supports these interfaces: |
Sigh... The Spike pointed to by So I guess that the |
I did this - built the latest Spike using I'll try the tests against the HiFive1 hardware tomorrow hopefully. |
|
Thanks @timsifive for your assistance. |
Why not... :-) |
|
Looks like some stuff is not up to date with the toolchain post separation of the A quick |
I made the change mentioned here: and ran the tests again with these results: |
|
I'm not sure that all of the tests are running reliably in this case (HiFive1 Rev A01). For some reason, after the breakpoint is set on This is a common pattern with other "exception" results. Edit: I ran this test manually for for the RAM and flash cases and for some reason the exact same logic triggers an exception/fault in the latter case. Specifically when executing this instruction: at this point in the program: The target memory address and data to be written are identical in the RAM and flash tests so I have no idea why this causes the store access fault... |
|
There's also stuff like this: which is happening because the |
|
That should be cleaned up, but I doubt it affects anything. |
|
Hi @TommyMurphyTM1234 , thank you for your time spent on this matter.
Agreed, looks like for the newer compiler versions, the
I have compared
... and we're getting identical outcomes 😃 Thanks for this independent confirmation. |
|
So I believe this issue can be considered done, we have some sort of baseline for HiFive1 Rev A01 regressions. I will close the ticket in a couple of days, unless there are further comments. |
Sounds good. Thanks for comparing my results to yours and confirming that they're consistent.
Also sounds good. I'll take other issues (e.g. the |
MarekVCodasip
added a commit
to MarekVCodasip/riscv-tests
that referenced
this issue
Jun 30, 2023
This commit adds a list of known failing tests based on: riscv-collab/riscv-openocd#869 (comment)
|
@timsifive The corresponding test exclusion file has been submitted here: riscv-software-src/riscv-tests#485 |
As I mentioned earlier, it seems to me that some of the tests result in a failure or exception status due to problems with the test programs and/or debug sessions themselves as opposed to being genuine results from the test. I am trying to investigate but my point in mentioning this is that I don't think that the HiFive1 results that I and @JanMatCodasip have achieved are "correct" and I think that it's maybe premature to mark some of the de-facto failure/exception tests as to be excluded. |
|
My main goal in this thread was to clearly document he current state - known passing and known failing tests on HiFive1 Rev A01 - regardless if it is a test issue or DUT issue. I agree that there will most likely be issues in the test themselves that can be fixed or improved. But that can be follow-up steps, outside of this thread, and whether that will happen depends on how much time people are able to commit to this :) |
OK - that makes sense. :-) |
I just filed riscv-software-src/riscv-tests#488 for this. I'm unlikely to work on it in the foreseeable future. |
FWIW I will look at some or all of the HiFive1 failing/exception tests to see if any or them are failing because of the test program/debug session itself as opposed to the actual test failing. |
timsifive
pushed a commit
to riscv-software-src/riscv-tests
that referenced
this issue
Jul 13, 2023
* Add an exclude list for known failing Hifive1 tests This commit adds a list of known failing tests based on: riscv-collab/riscv-openocd#869 (comment) * Fix name of the HiFive1 flash target Signed-off-by: Marek Vrbka <[email protected]> --------- Signed-off-by: Marek Vrbka <[email protected]>
This was referenced Apr 20, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We have run
riscv-tests/debugon HiFive1 board (Rev A01) with RISC-V debug implementation v0.11. Each test was repeated 30 times in order to catch sporadic fails, if any.The results are in table below.
@timsifive - We would like to ask for confirmation whether you are getting the same results, so that we can establish a baseline for regression testing of the 0.11 debug support in riscv-openocd.
Thank you.
riscv-openocd commit:
e0dd44a53c9df2206d854526082c34635866ea0briscv-tests commit:
7b52ba3b7167acb4d8b38f4d4633112b4699cb26The text was updated successfully, but these errors were encountered: