Unable to simulate gem5 ARM O3 CPU FS with external DRAMSys Tool

[atrah22]

I could integrate DRAMSys as per the example script in gem5 (SE simulation).
However, I am failing to do so for ARM O3 CPU FS simulation.

I am trying run with ddr4-example.json fil ein DRAMSys and I changed the StoreMode to Store.

ddr4-example.json file of DRAMSys is,

{
    "simulation": {
        "addressmapping": "am_ddr4_8x4Gbx8_dimm_p1KB_brc.json",
        "mcconfig": "fr_fcfs.json",
        "memspec": "JEDEC_4Gb_DDR4-1866_8bit_A.json",
        "simconfig": "example.json",
        "simulationid": "ddr4-example",
        "tracesetup": [
            {
                "clkMhz": 200,
                "name": "example.stl"
            }
        ]
    }
}


The main change is StoreMode to Store from NoStorage in the simconfig/example.json file.

{
    "simconfig": {
        "AddressOffset": 0,
        "CheckTLM2Protocol": false,
        "DatabaseRecording": true,
        "Debug": false,
        "EnableWindowing": false,
        "PowerAnalysis": false,
        "SimulationName": "example",
        "SimulationProgressBar": true,
        "StoreMode": "Store",
        "UseMalloc": false,
        "WindowSize": 1000
    }
}

Error is:

info: Simulated platform: VExpress_GEM5_V1
warn: No dot file generated. Please install pydot to generate the dot file and pdf.
src/sim/simulate.cc:199: info: Entering event queue @ 0.  Starting simulation...
src/dev/arm/rv_ctrl.cc:176: warn: SCReg: Access to unknown device dcc0:site0:pos0:fn7:dev0

Warning: DRAM: Use the blocking mode of DRAMSys with caution! The simulated timings do not reflect the real system!
In file: /home/gem5/ext/dramsys/DRAMSys/src/libdramsys/DRAMSys/simulation/dram/Dram.cpp:273
gem5.opt: src/mem/request.hh:890: uint64_t gem5::Request::getExtraData() const: Assertion `extraDataValid()' failed.
Program aborted at tick 9703200

1. The warning is about blocking mode. How can I change configuration to non-blocking mode? and
2. Is the error related to the blocking mode?

[derchr]

Hi,
can you confirm if the provided example config (configs/example/gem5_library/dramsys/arm-hello-dramsys.py) works for you?

1. The warning regarding the blocking mode is raised, whenever a gem5 component uses atomic memory accesses instead of timing accesses. Normally this happens, when an atomic CPU is used. The O3 CPU should only ever issue timing requests. So it is unclear why you see this message.
2. Maybe. Is there some kind of component in your platform besides the O3 CPU that accesses the memory?

[atrah22]

Hello,
That's a mistake from my end. I was trying to boot OS using atomic CPU mode and then use timing CPU mode to restore the checkpoint and run the workload.
I ran now simulation (a workload from micro benchmark) with CPU timing mode including OS boot. However, I think, the simulation with the DRAMSys with gem5 is stuck or on a loop.

The same simulation(a workload from micro benchmark) with gem5 default DDR4 memory takes 27 minutes to complete whereas the simulation with gem5 + DRAMSys ddr4 memory is running for more than 100 minutes! I understand the simulation with gem5+DRAMsys will be slower but is it slower by more than 3x?

Here is my simple configuration,
In devices.py script (configs/example/arm), I added configure_dramsys in SimpleSystem class like below:

class SimpleSystem(BaseSimpleSystem):
    """
    Meant to be used with the classic memory model
    """

    def __init__(self, caches, mem_size, platform=None, **kwargs):
        super().__init__(mem_size, platform, **kwargs)

        self.membus = MemBus()
        # CPUs->PIO
        self.iobridge = Bridge(delay="50ns")

        self._caches = caches
        if self._caches:
            self.iocache = IOCache(addr_ranges=self.mem_ranges)
        else:
            self.dmabridge = Bridge(delay="50ns", ranges=self.mem_ranges)

    def connect(self):
        self.iobridge.mem_side_port = self.iobus.cpu_side_ports
        self.iobridge.cpu_side_port = self.membus.mem_side_ports

        if self._caches:
            self.iocache.mem_side = self.membus.cpu_side_ports
            self.iocache.cpu_side = self.iobus.mem_side_ports
        else:
            self.dmabridge.mem_side_port = self.membus.cpu_side_ports
            self.dmabridge.cpu_side_port = self.iobus.mem_side_ports

        if hasattr(self.realview.gic, "cpu_addr"):
            self.gic_cpu_addr = self.realview.gic.cpu_addr
        self.realview.attachOnChipIO(self.membus, self.iobridge)
        self.realview.attachIO(self.iobus)
        self.system_port = self.membus.cpu_side_ports

    def attach_pci(self, dev):
        self.realview.attachPciDevice(dev, self.iobus)

    def configure_dramsys(self, dramsys_memory):
        self.dramsys = dramsys_memory
        memory = self.dramsys
        self.mem_ranges=[AddrRange("0x80000000",size=memory.get_size())]
        self.dramsys.set_memory_range(self.mem_ranges)
        # Connect the transactor to the DRAMSys memory object
        self.membus.mem_side_ports = self.dramsys.get_mem_ports()[0][1]

In fs_bigLITTLE.py script, I tried to configure dramsys instead of using config_mem (gem5 DRAM config) in createSystem class like below:

def createSystem(
    options,
    caches,
    kernel,
    bootscript,
    machine_type="VExpress_GEM5",
    disks=[],
    mem_size=default_mem_size,
    bootloader=None,
):
    platform = ObjectList.platform_list.get(machine_type)
    m5.util.inform("Simulated platform: %s", platform.__name__)

    
    sys = devices.SimpleSystem(
        caches,
        mem_size,
        platform(),
        workload=ArmFsLinux(object_file=SysPaths.binary(kernel)),
        readfile=bootscript,
    )

    dramsys_memory = DRAMSysMem(
        configuration="/home/gem5/ext/dramsys/DRAMSys/configs/ddr4-example.json",
        recordable=True,
        resource_directory="/home/gem5/ext/dramsys/DRAMSys/configs",
        size="4GB",
    )

    #config_mem(options, sys)
    sys.configure_dramsys(dramsys_memory)

    sys.connect()

    # Attach disk images
    if disks:

        def cow_disk(image_file):
            image = CowDiskImage()
            image.child.image_file = SysPaths.disk(image_file)
            return image

        sys.disk_images = [cow_disk(f) for f in disks]
        sys.pci_vio_block = [
            PciVirtIO(vio=VirtIOBlock(image=img)) for img in sys.disk_images
        ]
        for dev in sys.pci_vio_block:
            sys.attach_pci(dev)

    sys.realview.setupBootLoader(sys, SysPaths.binary, bootloader)

    return sys

Please let me know if this is not the right way to add DRAMSys dram with gem5 for ARM O3 CPU using configs/example/arm/fs_bigLITTLE.py script.

[derchr]

Normally it should be possible to boot the OS using the Atomic CPU and then switch to the Timing CPU, even with DRAMSys. For example, running the DRAMSys examples with the Atomic CPU works fine. It results in very inaccurate timing results but this is fine for booting.

If you don't need the transaction database for analyzing the trace with our Trace Analyzer (https://github.com/tukl-msd/DRAMSys?tab=readme-ov-file#trace-analyzer-consulting-and-custom-tailored-modifications), then you can disable the database recording (try to disable it in the Pyhton script as well as in the json config). This should significantly speed up the DRAMSys simulation.

Finally, you can check if the simulation is actually stuck by enabling gem5 debug flags (for example "Cache" would be a good candidate). Is your OS Linux? Then you can also dial up the telnet connection and see the console output while booting. It should be clear then if something goes wrong.

[atrah22]

rm-hello-dramsys.p

Hello, configs/example/gem5_library/dramsys/arm-hello-dramsys.py works fine.
Ubuntu boot doesn't when memory is selected as dramsys, (configs/example/gem5_library/arm-ubuntu-run.py). arm-ubuntu-run.py is simple fs simulation and the simulation keeps running and it doesn't seem to finish.

[derchr]

Hi, I just tried to boot Ubuntu with a timing core and it seems like it's actually stuck. With the atomic core, it works. I will have a look into it.

[atrah22]

Yes, please. As I mentioned, I am facing the same issue. And this issue also persists when I try to simulate O3 ARM CPU (timing mode) using the fs_bigLITTLE.py script.

[derchr]

We identified the issue and I can confirm that DRAMSys correctly boots with the Timing as well as the O3 core.
While we are working on a fix, you can comment out line 555 in src/libdramsys/DRAMSys/controller/Controller.cpp (https://github.com/tukl-msd/DRAMSys/blob/master/src/libdramsys/DRAMSys/controller/Controller.cpp#L555).

Let me know if this fixed your issue.