HandleSysMmap allocations with hints

[BlocksOnAChain]

Description
The sys_mmap system call allows the use of address hints to allocate memory at or near a requested address. Currently, op-cannon processes hinted sys_mmap calls indiscriminately in exec.HandleSysMmap, always informing the caller that it has memory at the requested address without performing any validation or tracking. This behavior poses a risk of guest memory allocations overlapping with op-cannon's internal memory spaces.

In certain scenarios, hints are commonly used. For instance, dependencies of the op-program handle some allocations by directly using runtime.mallocgc, which for large allocations will use hints.

Note
In testing the emulated program's heap is presently located at 0xc000000000, providing substantial overhead relative to cannon's arch64.HeapStart. However, blindly allowing allocations at any address and with any size is a security risk.

Recommendation
Implement bounds checking for sys_mmap calls leveraging hints to prevent memory overlaps.

--
OP labs team solution for this finding:

@Inphi - We won't be implementing bounds checking in the VM. As this issue does not affect the op-program (and Go programs in general). And Cannon is not intended to support any other types of programs.

• We will document mmap hinting in the spec per auditors suggestion.