RISC-V from scratch 2: Hardware layouts, linker scripts, and C runtimes

[utterances-bot]

RISC-V from scratch 2: Hardware layouts, linker scripts, and C runtimes

A post describing how C programs get to the main function. Devicetree layouts, linker scripts, minimal C runtimes, GDB and QEMU, basic RISC-V assembly, and other topics are reviewed along the way.

https://twilco.github.io/riscv-from-scratch/2019/04/27/riscv-from-scratch-2.html

[yichnal]

grep doesn't need cat. At all. It's called UUOC.

[twilco]

@StephaneFr - fixed with 9c4840e. Thanks for the feedback!

[alicelyy]

Please keep them coming!

[twilco]

@limslarmo, work has officially begun on part three - stay tuned. 😄

[yashomer1994]

hey!, I am not able to use tub option on macOS Mojave can you help me?

[twilco]

Hey again @yashomer1994. I'm not sure what you mean by "tub option" - could you give me some more information? What command are you trying to run, and what error are you getting?

[yashomer1994]

Tui*option in gdb

On Thu, 1 Aug 2019 at 1:14 AM, Tyler Wilcock ***@***.***> wrote: Hey again @yashomer1994 <https://github.com/yashomer1994>. I'm not sure what you mean by "tub option" - could you give me some more information? What command are you trying to run, and what error are you getting? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5?email_source=notifications&email_token=AEKGT3VRPQGPPBP5QX4TCR3QCHTS5A5CNFSM4HN3WXJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3ILB5Q#issuecomment-516993270>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEKGT3VD2ELTVBZIUTLA2RTQCHTS5ANCNFSM4HN3WXJA> .

-- Sent from Gmail Mobile

[twilco]

Is this the command you are trying to run?

riscv64-unknown-elf-gdb --tui a.out

The riscv64-unknown-elf-gdb executable should support the --tui flag. If this indeed the command you are trying to run, could you give me the version of your GDB installation? Here is mine:

riscv64-unknown-elf-gdb --version
GNU gdb (GDB) 8.2.90.20190228-git

[yashomer1994]

Hey, Yeah its done working. I jus relinked my riscv64-unkown-elf-gdb And its working fine.

On Thu, 1 Aug 2019 at 9:45 AM, Tyler Wilcock ***@***.***> wrote: Is this the command you are trying to run? riscv64-unknown-elf-gdb --tui a.out The riscv64-unknown-elf-gdb executable should support the --tui flag. If this indeed the command you are trying to run, could you give me the version of your GDB installation? Here is mine: riscv64-unknown-elf-gdb --version GNU gdb (GDB) 8.2.90.20190228-git — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5?email_source=notifications&email_token=AEKGT3UMBAN75U6WD7SKEIDQCJPPHA5CNFSM4HN3WXJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3JIARQ#issuecomment-517111878>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEKGT3RKVH2IGW3V45FMDYTQCJPPHANCNFSM4HN3WXJA> .

-- Sent from Gmail Mobile

[chillancezen]

hi Tyler,

I got a way to allocate the initial stack without having to define a symbol in the linker script:

.section .text
guest_kernel_entry:
    la sp, __stack_top
    //ebreak
    //j init_main
    j infinite_loop


infinite_loop:
    //wfi
    j infinite_loop




// allocate an area of memory to serve as initial stack
// 8K is supposed to be enough
.section .bss
.space 1024*8
.align 4
__stack_top:

[twilco]

Hey @chillancezen,

Looks promising — nice work!

Not sure if this is applicable in this context, but you may need greater than a 4-byte alignment (.align 4). Stacks tend to be aligned to the largest byte-size datatype that can be pushed to the stack, which for RISC-V is the long double weighing in at 16 bytes. Also, the RISC-V calling convention calls for the stack pointer to be 16-byte aligned.

For more discussion on this subject, look here in part four.

[chillancezen]

Not sure if this is applicable in this context, but you may need greater than a 4-byte alignment (.align 4). Stacks tend to be align

sure, you are right, it must be 16-bytes aligned.

[adeaarm]

Hi, is there any way to use gdb from within Eclipse to attach to the QEMU gdbserver?

[adeaarm]

Also, it would be nice to have some documentation stating what is the content of the ROM of the virt machine, placed at 0x1000.

[twilco]

Hello @adeaarm!

Hi, is there any way to use gdb from within Eclipse to attach to the QEMU gdbserver?

Yes, that should be doable. In this section of the post, we run this command:

qemu-system-riscv64 -machine virt -m 128M -gdb tcp::1234 -kernel a.out

The -gdb tcp::1234 bit tells Qemu that in addition to running a.out with the virt machine, it should also start a GDB server at localhost:1234. Then, you can use Eclipse as your GDB client and connect to this server. Check out this Stack Overflow for instructions on how to do that:

https://stackoverflow.com/a/4235095/2421349

Also, it would be nice to have some documentation stating what is the content of the ROM of the virt machine, placed at 0x1000.

Digging around a bit, I found this, which seems to imply the value of the ROM is riscv_virt_board.mrom:

https://github.com/qemu/qemu/blob/02777ac3036187077c98a05843d888b4be8c51b3/hw/riscv/virt.c#L508#L509

I wasn't able to actually find this file in the qemu repository nor the greater internet, so I decided to boot up QEMU + GDB, since GDB allows us to inspect arbitrary memory addresses.

First I compiled add.c as found in this section of post two, giving us a binary named a.out. Now let's start QEMU and GDB:

qemu-system-riscv64 -machine virt -m 128M -gdb tcp::1234 -S -kernel a.out -nographic

And then connect to this with our GDB client:

riscv64-unknown-elf-gdb --tui a.out

(gdb) target remote :1234

From here, using GDB's examine command (also see GDB output formatting), we can check out what is in the memory address range of the ROM, 0x1000 to 0x11000:

x/10000sb 0x1000

Here's a screen recording of me doing this. Note the bits I highlight.

(Edit: Realizing now that this GIF is hard to see depending on your screen size. You can try clicking/tapping on it, which should help.)

Look familiar? If you've made it through post two, it should! It appears the DTS for the virt machine is what is located in the ROM (see this section).

Hopefully this helps. Go forth and explore 🚀

[adeaarm]

Thanks for your reply @twilco. Yes, I noticed that riscv_virt_board.mrom, but I couldn't really find any documentation for it. Maybe something related to the ZSBL (Zero Stage Boot Loader) in RISC-V terminoology but I am not sure as this doesn't seem to be documented anywhere (or at least I couldn't find anywhere publicly)

[Shiva-prog123]

Hi,
Great job! I changed the above c program to get output for multiple operations like subtraction, multiplication, addition and division, used the same crt and linker but getting only the result of first arithmetic operation (here subtraction output). Is the problem with the crt0 or linker file??

[twilco]

Hi @Shiva-prog123!

It's hard to say without some more information. Can you please post your updated C program? Also, when you say "output", do you mean you're running your program under GDB and trying to print the results of additional operations (e.g. multiplication as you mentioned), but are unable to?

[Shiva-prog123]

Hi,
my c program
void main()
{
int a=20,b=17,c,d,e,f;
c=a+b;
d=a*b;
e=a-b;
f=a/b;
}
In this program I am trying to get the result of all the four operations, but getting only the result of first arithmetic operation(in this c code first operation is addition).
In the linker file PROVIDE(__stack_top = ORIGIN(ram) + LENGTH(ram)); in this line shall I define more to get other results?

[twilco]

Thanks for posting your code snippet! You may already know this, but Github supports Markdown formatting, so you can make your code look a bit nicer if you surround it in triple backticks (and optionally include the language of the snippet after the first triplet):

```c
int main() {}
```

I included backslashes to prevent the rendering, but without them it looks like:

int main()
{
    int a=20,b=17,c,d,e,f;
    c=a+b;
    d=a*b;
    e=a-b;
    f=a/b;

    return 0;
}

---

In the linker file PROVIDE(__stack_top = ORIGIN(ram) + LENGTH(ram)); in this line shall I define more to get other results?

__stack_top is unrelated to any sort of output or result. Instead, it is used by our crt0.s file to setup the stack, which you can think of as "scratch" memory for each invocation of a function. So in your code snippet, you have this line:

int a = 20, b = 17, c, d, e, f;

When the compiler sees this, it allocates memory for these local variables on the stack by decrementing (or sometimes incrementing) the stack pointer register by sizeof(int) * <NUM_INTS> bytes.

So I'm not sure exactly what you mean when you say you're only able to get the first result, but I think you may be heading the wrong direction looking at the linker script and CRT (unless something much larger is going wrong here). In the end of this post, we run our C program with GDB and set a breakpoint. I'd recommend you do the same here if you'd like to examine the results of these calculations (e.g. using GDBs print <VARIABLE_NAME> command).

Throughout this series of posts we talk about how the stack works, but this video also explains it super well.

[Shiva-prog123]

Thanks for the reply! how we can increase the stack size?

[twilco]

In our current setup, the stack will grow unbounded until we run out of physical memory. Concretely, this means we'll keep decrementing the sp register (growing the stack) until we literally cannot anymore, e.g. the CPU faults.

So, in order to increase stack size, increase the amount of physical memory available and then set __stack_top to be the address of the highest point of that memory.

It sounds like you might be interested in the Implementing a function prologue section of post 4. It's a big skip forwards, but reading it through might illuminate some of the concepts you're exploring.

[Shiva-prog123]

Thank you! I will check it out...

[Bigyin1]

Thanks for great explanation of such tricky concepts! Why does such crucial info, as memory map of qemu's machines, is not explicitly represented in it's docs, like in all MCUs docs? The only way to get it, is to read dtb or to dig in virt's source code.

[hossamfadeel]

Any help with this issue:
the following command didn't work with me
hossam@hossam:work$ qemu-system-riscv64 -machine virt -m 128M -gdb tcp::1234 -kernel a.out
qemu-system-riscv64: Some ROM regions are overlapping
These ROM regions might have been loaded by direct user request or by default.
They could be BIOS/firmware images, a guest kernel, initrd or some other file loaded into guest memory.
Check whether you intended to load all this guest code, and whether it has been built to load to the correct addresses.

The following two regions overlap (in the memory address space):
a.out ELF program header segment 0 (addresses 0x000000007ffff000 - 0x000000008000006c)
/usr/local/bin/../share/qemu/opensbi-riscv64-generic-fw_dynamic.bin (addresses 0x0000000080000000 - 0x0000000080012558)

Any suggestions??

[kevin335200]

@hossamfadeel

Some ROM regions are overlapping

Just add -bios none to ignore this warning.
Source: noteed/riscv-hello-asm#1 (comment)

[bsm1244]

It's very helpful to me, thanks. However I have a problem. There are different size of instructions in my disassembled code like this.
80000018

:
80000018: 1101 addi sp,sp,-32
8000001a: ce22 sw s0,28(sp)
8000001c: 1000 addi s0,sp,32
8000001e: 4791 li a5,4
80000020: fef42623 sw a5,-20(s0)
80000024: 47b1 li a5,12
80000026: fef42423 sw a5,-24(s0)
8000002a: fec42703 lw a4,-20(s0)
8000002e: fe842783 lw a5,-24(s0)
80000032: 97ba add a5,a5,a4
80000034: fef42223 sw a5,-28(s0)
80000038: bfcd j 8000002a <main+0x12>

how can i fix the size of instructions??

[riscvtest]

hi every one
i have problem , gdb does not stop in breakpoints it is blocked in Continuing.

can some one help me please

[riscvtest]

Is this the command you are trying to run?

riscv64-unknown-elf-gdb --tui a.out

The riscv64-unknown-elf-gdb executable should support the --tui flag. If this indeed the command you are trying to run, could you give me the version of your GDB installation? Here is mine:

riscv64-unknown-elf-gdb --version
GNU gdb (GDB) 8.2.90.20190228-git

for me i have the same problem , how can i fix this ??

-VirtualBox:/riscv_test_musl$ riscv64-unknown-linux-musl-gdb --tui test1
riscv64-unknown-linux-musl-gdb: TUI mode is not supported
-VirtualBox:/riscv_test_musl$ riscv64-unknown-linux-musl-gdb --version
GNU gdb (GDB) 10.1

[lapnd]

Hi @twilco
Is there any chance for C++ support?

[oconnor663]

I ran into the same Some ROM regions are overlapping problem as #5 (comment), and the -bios none fix suggested by @kevin335200 worked for me. Might be worth adding that to the article.

[qiujingbao]

Hi @twilco
I have questions regarding the QEMU RISC-V linker script. To my knowledge, the first instruction to be executed is located at address 0x80000000. I understand that the ENTRY directive in the linker script sets the e_entry field in the ELF file to indicate where the execution should start. My question is whether QEMU RISC-V will execute code starting at 0x80000000 or from the address specified by the ENTRY directive. Should we organize the object files so that the _start function is placed at 0x80000000? Thank you.