Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Raspberry Pi 5 model B #21

Open
geerlingguy opened this issue Sep 28, 2023 · 186 comments
Open

Raspberry Pi 5 model B #21

geerlingguy opened this issue Sep 28, 2023 · 186 comments

Comments

@geerlingguy
Copy link
Owner

geerlingguy commented Sep 28, 2023

DSC03121

NOTE: The board used for this testing was a pre-release revision. The final production Pi 5 model B and any future revisions may have slight differences, especially as the Raspberry Pi OS image is updated.

Basic information

Linux/system information

# output of `neofetch`
       _,met$$$$$gg.          pi@pi5 
    ,g$$$$$$$$$$$$$$$P.       ------ 
  ,g$$P"     """Y$$.".        OS: Debian GNU/Linux 12 (bookworm) aarch64 
 ,$$P'              `$$$.     Host: Raspberry Pi 5 Model B Rev 1.0 
',$$P       ,ggs.     `$$b:   Kernel: 6.1.32-v8+ 
`d$$'     ,$P"'   .    $$$    Uptime: 4 mins 
 $$P      d$'     ,    $$P    Packages: 1536 (dpkg) 
 $$:      $$.   -    ,d$$'    Shell: bash 5.2.15 
 $$;      Y$b._   _,d$P'      Resolution: 1920x1080 
 Y$$.    `.`"Y$$$$P"'         Terminal: /dev/pts/0 
 `$$b      "-.__              CPU: BCM2835 (4) @ 2.400GHz 
  `Y$$                        Memory: 416MiB / 7942MiB 
   `Y$$.
     `$$b.                                            
       `Y$$b.                                         
          `"Y$b._
              `"""


# output of `uname -a`
Linux pi5 6.1.32-v8+ #1 SMP PREEMPT Sat Aug  5 07:03:33 BST 2023 aarch64 GNU/Linux

Benchmark results

CPU

Power

  • Idle power draw (at wall): 2.8 W
  • Maximum simulated power draw (stress-ng --matrix 0): 9.7 W
  • During Geekbench multicore benchmark: 7.9 W
  • During top500 HPL benchmark: 11 W (2.75 Gflops/W)

Disk

SanDisk Extreme 128GB microSD

Benchmark Result
fio 1M sequential read 91.5 MB/s
iozone 1M random read 90.09 MB/s
iozone 1M random write 62.28 MB/s
iozone 4K random read 9.86 MB/s
iozone 4K random write 4.07 MB/s

Kioxia XG8 1TB NVMe at PCIe Gen 2

Benchmark Result
fio 1M sequential read 454 MB/s
iozone 1M random read 425.57 MB/s
iozone 1M random write 401.33 MB/s
iozone 4K random read 65.51 MB/s
iozone 4K random write 159.85 MB/s

Kioxia XG8 1TB NVMe at PCIe Gen 3

Benchmark Result
fio 1M sequential read 894 MB/s
iozone 1M random read 804.39 MB/s
iozone 1M random write 777.69 MB/s
iozone 4K random read 69.76 MB/s
iozone 4K random write 194.42 MB/s

Single SanDisk Extreme PRO USB 3.1 Flash Drive

Benchmark Result
fio 1M sequential read 301 MB/s
iozone 1M random read 297.97 MB/s
iozone 1M random write 192.83 MB/s
iozone 4K random read 13.94 MB/s
iozone 4K random write 16.80 MB/s

2x SanDisk Extreme PRO USB 3.1 Flash Drives (simultaneous)

Benchmark Result
fio 1M sequential read 400 MB/s
iozone 1M random read 601.14 MB/s
iozone 1M random write 419.62 MB/s
iozone 4K random read 29.5 MB/s
iozone 4K random write 35.65 MB/s

Run benchmark on any attached storage device (e.g. eMMC, microSD, NVMe, SATA) and add results under an additional heading. Download the script with curl -o disk-benchmark.sh [URL_HERE] and run sudo DEVICE_UNDER_TEST=/dev/sda DEVICE_MOUNT_PATH=/mnt/sda1 ./disk-benchmark.sh (assuming the device is sda).

Network

Built-in Ethernet

  • iperf3 -c $SERVER_IP: 938 Mbps
  • iperf3 --reverse -c $SERVER_IP: 942 Mbps
  • iperf3 --bidir -c $SERVER_IP: 930 Mbps up, 600 Mbps down

Built-in WiFi

  • iperf3 -c $SERVER_IP: 186 Mbps
  • iperf3 --reverse -c $SERVER_IP: 207 Mbps
  • iperf3 --bidir -c $SERVER_IP: 2.15 Mbps up, 206 Mbps down

USB 3.0 Pluggable USBC-E2500 2.5 Gbps Adapter

  • iperf3 -c $SERVER_IP: 1.55 Gbps
  • iperf3 --reverse -c $SERVER_IP: 300 Mbps
  • iperf3 --bidir -c $SERVER_IP: 1.56 Gbps up, 153 Mbps down

ASUS XG-C100C 10G Network Adapter (Aquantia AQC107)

  • iperf3 -c $SERVER_IP: 5.63 Gbps
  • iperf3 --reverse -c $SERVER_IP: 6.05 Gbps
  • iperf3 --bidir -c $SERVER_IP: 4.40 Gbps up, 2.50 Gbps down

GPU

=======================================================
    glmark2 2023.01
=======================================================
    OpenGL Information
    GL_VENDOR:      Broadcom
    GL_RENDERER:    V3D 7.1.7.0
    GL_VERSION:     OpenGL ES 3.1 Mesa 24.2.4-1~bpo12+1~rpt1
    Surface Config: buf=32 r=8 g=8 b=8 a=8 depth=24 stencil=0 samples=0
    Surface Size:   800x600 windowed
=======================================================
[build] use-vbo=false: FPS: 2695 FrameTime: 0.371 ms
[build] use-vbo=true: FPS: 3518 FrameTime: 0.284 ms
[texture] texture-filter=nearest: FPS: 2765 FrameTime: 0.362 ms
[texture] texture-filter=linear: FPS: 2751 FrameTime: 0.364 ms
[texture] texture-filter=mipmap: FPS: 2847 FrameTime: 0.351 ms
[shading] shading=gouraud: FPS: 2809 FrameTime: 0.356 ms
[shading] shading=blinn-phong-inf: FPS: 2532 FrameTime: 0.395 ms
[shading] shading=phong: FPS: 2126 FrameTime: 0.471 ms
[shading] shading=cel: FPS: 2083 FrameTime: 0.480 ms
[bump] bump-render=high-poly: FPS: 1369 FrameTime: 0.730 ms
[bump] bump-render=normals: FPS: 3098 FrameTime: 0.323 ms
[bump] bump-render=height: FPS: 2887 FrameTime: 0.346 ms
[effect2d] kernel=0,1,0;1,-4,1;0,1,0;: FPS: 1180 FrameTime: 0.848 ms
[effect2d] kernel=1,1,1,1,1;1,1,1,1,1;1,1,1,1,1;: FPS: 482 FrameTime: 2.076 ms
[pulsar] light=false:quads=5:texture=false: FPS: 3000 FrameTime: 0.333 ms
[desktop] blur-radius=5:effect=blur:passes=1:separable=true:windows=4: FPS: 290 FrameTime: 3.451 ms
[desktop] effect=shadow:windows=4: FPS: 1063 FrameTime: 0.941 ms
[buffer] columns=200:interleave=false:update-dispersion=0.9:update-fraction=0.5:update-method=map: FPS: 547 FrameTime: 1.831 ms
[buffer] columns=200:interleave=false:update-dispersion=0.9:update-fraction=0.5:update-method=subdata: FPS: 533 FrameTime: 1.877 ms
[buffer] columns=200:interleave=true:update-dispersion=0.9:update-fraction=0.5:update-method=map: FPS: 584 FrameTime: 1.714 ms
[ideas] speed=duration: FPS: 2289 FrameTime: 0.437 ms
[jellyfish] <default>: FPS: 1217 FrameTime: 0.822 ms
[terrain] <default>: FPS: 77 FrameTime: 13.071 ms
[shadow] <default>: FPS: 376 FrameTime: 2.665 ms
[refract] <default>: FPS: 119 FrameTime: 8.470 ms
[conditionals] fragment-steps=0:vertex-steps=0: FPS: 3309 FrameTime: 0.302 ms
[conditionals] fragment-steps=5:vertex-steps=0: FPS: 2318 FrameTime: 0.431 ms
[conditionals] fragment-steps=0:vertex-steps=5: FPS: 3272 FrameTime: 0.306 ms
[function] fragment-complexity=low:fragment-steps=5: FPS: 2755 FrameTime: 0.363 ms
[function] fragment-complexity=medium:fragment-steps=5: FPS: 1893 FrameTime: 0.528 ms
[loop] fragment-loop=false:fragment-steps=5:vertex-steps=5: FPS: 2670 FrameTime: 0.375 ms
[loop] fragment-steps=5:fragment-uniform=false:vertex-steps=5: FPS: 2680 FrameTime: 0.373 ms
[loop] fragment-steps=5:fragment-uniform=true:vertex-steps=5: FPS: 1783 FrameTime: 0.561 ms
=======================================================
                                  glmark2 Score: 1935 
=======================================================

Compatibility reports:

Memory

tinymembench results:

Click to expand memory benchmark result
tinymembench v0.4.10 (simple benchmark for memory throughput and latency)

==========================================================================
== Memory bandwidth tests                                               ==
==                                                                      ==
== Note 1: 1MB = 1000000 bytes                                          ==
== Note 2: Results for 'copy' tests show how many bytes can be          ==
==         copied per second (adding together read and writen           ==
==         bytes would have provided twice higher numbers)              ==
== Note 3: 2-pass copy means that we are using a small temporary buffer ==
==         to first fetch data into it, and only then write it to the   ==
==         destination (source -> L1 cache, L1 cache -> destination)    ==
== Note 4: If sample standard deviation exceeds 0.1%, it is shown in    ==
==         brackets                                                     ==
==========================================================================

 C copy backwards                                     :   5676.0 MB/s (1.9%)
 C copy backwards (32 byte blocks)                    :   5649.5 MB/s (0.6%)
 C copy backwards (64 byte blocks)                    :   5688.0 MB/s (0.6%)
 C copy                                               :   4870.3 MB/s (0.2%)
 C copy prefetched (32 bytes step)                    :   4808.4 MB/s
 C copy prefetched (64 bytes step)                    :   4819.7 MB/s (0.2%)
 C 2-pass copy                                        :   4887.0 MB/s (0.6%)
 C 2-pass copy prefetched (32 bytes step)             :   4849.2 MB/s
 C 2-pass copy prefetched (64 bytes step)             :   4845.1 MB/s
 C fill                                               :  13672.2 MB/s (0.5%)
 C fill (shuffle within 16 byte blocks)               :  13655.6 MB/s (0.5%)
 C fill (shuffle within 32 byte blocks)               :  13697.6 MB/s (0.5%)
 C fill (shuffle within 64 byte blocks)               :  13703.3 MB/s (0.4%)
 NEON 64x2 COPY                                       :   4813.8 MB/s (0.1%)
 NEON 64x2x4 COPY                                     :   4832.2 MB/s (0.1%)
 NEON 64x1x4_x2 COPY                                  :   4830.8 MB/s (0.1%)
 NEON 64x2 COPY prefetch x2                           :   4298.5 MB/s
 NEON 64x2x4 COPY prefetch x1                         :   4335.5 MB/s
 NEON 64x2 COPY prefetch x1                           :   4306.5 MB/s
 NEON 64x2x4 COPY prefetch x1                         :   4336.0 MB/s
 ---
 standard memcpy                                      :   4805.4 MB/s
 standard memset                                      :  13676.9 MB/s (0.6%)
 ---
 NEON LDP/STP copy                                    :   4793.9 MB/s
 NEON LDP/STP copy pldl2strm (32 bytes step)          :   4821.9 MB/s
 NEON LDP/STP copy pldl2strm (64 bytes step)          :   4829.9 MB/s
 NEON LDP/STP copy pldl1keep (32 bytes step)          :   4811.7 MB/s
 NEON LDP/STP copy pldl1keep (64 bytes step)          :   4812.1 MB/s
 NEON LD1/ST1 copy                                    :   4818.2 MB/s
 NEON STP fill                                        :  13659.2 MB/s (0.5%)
 NEON STNP fill                                       :  13683.3 MB/s (0.4%)
 ARM LDP/STP copy                                     :   4818.7 MB/s
 ARM STP fill                                         :  13557.6 MB/s
 ARM STNP fill                                        :  13682.7 MB/s (0.4%)

==========================================================================
== Memory latency test                                                  ==
==                                                                      ==
== Average time is measured for random memory accesses in the buffers   ==
== of different sizes. The larger is the buffer, the more significant   ==
== are relative contributions of TLB, L1/L2 cache misses and SDRAM      ==
== accesses. For extremely large buffer sizes we are expecting to see   ==
== page table walk with several requests to SDRAM for almost every      ==
== memory access (though 64MiB is not nearly large enough to experience ==
== this effect to its fullest).                                         ==
==                                                                      ==
== Note 1: All the numbers are representing extra time, which needs to  ==
==         be added to L1 cache latency. The cycle timings for L1 cache ==
==         latency can be usually found in the processor documentation. ==
== Note 2: Dual random read means that we are simultaneously performing ==
==         two independent memory accesses at a time. In the case if    ==
==         the memory subsystem can't handle multiple outstanding       ==
==         requests, dual random read has the same timings as two       ==
==         single reads performed one after another.                    ==
==========================================================================

block size : single random read / dual random read
      1024 :    0.0 ns          /     0.0 ns 
      2048 :    0.0 ns          /     0.0 ns 
      4096 :    0.0 ns          /     0.0 ns 
      8192 :    0.0 ns          /     0.0 ns 
     16384 :    0.0 ns          /     0.0 ns 
     32768 :    0.0 ns          /     0.0 ns 
     65536 :    0.0 ns          /     0.0 ns 
    131072 :    1.1 ns          /     1.5 ns 
    262144 :    1.6 ns          /     2.0 ns 
    524288 :    1.9 ns          /     2.2 ns 
   1048576 :    8.3 ns          /    11.2 ns 
   2097152 :   12.2 ns          /    14.4 ns 
   4194304 :   54.9 ns          /    84.1 ns 
   8388608 :   86.8 ns          /   119.4 ns 
  16777216 :  103.0 ns          /   131.6 ns 
  33554432 :  112.9 ns          /   138.3 ns 
  67108864 :  118.7 ns          /   142.2 ns 

Phoronix Test Suite

Results from pi-general-benchmark.sh:

  • pts/encode-mp3: 11.824 sec
  • pts/x264 4K: 4.20 fps
  • pts/x264 1080p: 18.00 fps
  • pts/phpbench: 437085
  • pts/build-linux-kernel (defconfig): 2282.952 sec

Other Data

Crypto performance as measured by OpenSSL (see sbc-bench ARMv8 Crypto Extensions):

pi@pi5:~ $ openssl speed -elapsed -evp aes-256-cbc
You have chosen to measure elapsed time instead of user CPU time.
Doing AES-256-CBC for 3s on 16 size blocks: 109452832 AES-256-CBC's in 3.00s
Doing AES-256-CBC for 3s on 64 size blocks: 48954244 AES-256-CBC's in 3.00s
Doing AES-256-CBC for 3s on 256 size blocks: 14899089 AES-256-CBC's in 3.00s
Doing AES-256-CBC for 3s on 1024 size blocks: 3922332 AES-256-CBC's in 3.00s
Doing AES-256-CBC for 3s on 8192 size blocks: 500147 AES-256-CBC's in 3.00s
Doing AES-256-CBC for 3s on 16384 size blocks: 250524 AES-256-CBC's in 3.00s
version: 3.0.9
built on: Tue Jun 27 11:03:08 2023 UTC
options: bn(64,64)
compiler: gcc -fPIC -pthread -Wa,--noexecstack -Wall -fzero-call-used-regs=used-gpr -DOPENSSL_TLS_SECURITY_LEVEL=2 -Wa,--noexecstack -g -O2 -ffile-prefix-map=/build/openssl-3EYFhe/openssl-3.0.9=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
CPUINFO: OPENSSL_armcap=0xbd
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
AES-256-CBC     583748.44k  1044357.21k  1271388.93k  1338822.66k  1365734.74k  1368195.07k

PTP Hardware Timestamping support via Cadence GEM:

pi@pi5:~ $ dmesg | grep ptp
[    4.692928] pps pps0: new PPS source ptp0
[    4.692979] macb 1f00100000.ethernet: gem-ptp-timer ptp clock registered.
pi@pi5:~ $ ls /dev/ptp*
/dev/ptp0
pi@pi5:~ $ cat /sys/class/ptp/ptp0/clock_name 
gem-ptp-timer
pi@pi5:~ $ ethtool -T eth0
Time stamping parameters for eth0:
Capabilities:
  hardware-transmit
  software-transmit
  hardware-receive
  software-receive
  software-system-clock
  hardware-raw-clock
PTP Hardware Clock: 0
Hardware Transmit Timestamp Modes:
  off
  on
  onestep-sync
Hardware Receive Filter Modes:
  none
  all
@geerlingguy
Copy link
Owner Author

geerlingguy commented Sep 28, 2023

Note: Most benchmarks were run with 16k page size; Geekbench 6 would not work with 16k so I rebooted on 4k page size kernel8.

See video: Rasbperry Pi 5: EVERYTHING you need to know

And blog post: Testing PCIe on the Raspberry Pi 5

@geerlingguy
Copy link
Owner Author

sbc-bench results: ThomasKaiser/sbc-bench#77

@ThomasKaiser
Copy link

Can you please post output from lsusb and lspci?

@geerlingguy
Copy link
Owner Author

lsusb:

pi@pi5:~ $ sudo lsusb
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
pi@pi5:~ $ sudo lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/2p, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/2p, 480M

lspci:

pi@pi5:~ $ sudo lspci -vvvv
0001:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21) (prog-if 00 [Normal decode])
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 40
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	Memory behind bridge: 00000000-005fffff [size=6M] [32-bit]
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled] [64-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [48] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x4, ASPM L1, Exit Latency L1 <2us
			ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
		LnkCtl:	ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x4
			TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt+
		RootCap: CRSVisible+
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
			 10BitTagComp- 10BitTagReq- OBFF Via WAKE#, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- LN System CLS Not Supported, TPHComp- ExtTPHComp- ARIFwd+
			 AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled, ARIFwd-
			 AtomicOpsCtl: ReqEn- EgressBlck-
		LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS+
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported, DRS-
			 DownstreamComp: Link Up - Present
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
		RootCmd: CERptEn+ NFERptEn+ FERptEn+
		RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
			 FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
		ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
	Capabilities: [160 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
	Capabilities: [240 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=8us PortTPowerOnTime=10us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=1us LTR1.2_Threshold=0ns
		L1SubCtl2: T_PwrOn=10us
	Capabilities: [300 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Kernel driver in use: pcieport

0001:01:00.0 Ethernet controller: Device 1de4:0001
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 40
	Region 0: Memory at 1f00410000 (32-bit, non-prefetchable) [size=16K]
	Region 1: Memory at 1f00000000 (32-bit, non-prefetchable) [virtual] [size=4M]
	Region 2: Memory at 1f00400000 (32-bit, non-prefetchable) [size=64K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2- AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [70] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0W
		DevCtl:	CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <2us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x4
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP- LTR-
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- TPHComp- ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
			 AtomicOpsCtl: ReqEn-
		LnkCap2: Supported Link Speeds: 2.5-5GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [b0] MSI-X: Enable+ Count=61 Masked-
		Vector table: BAR=0 offset=00000000
		PBA: BAR=0 offset=00002000
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Kernel driver in use: rp1

@geerlingguy
Copy link
Owner Author

Note that the above lspci was run when I had nothing attached to the 'ext' port—it's currently booting off a microSD card in the official case, and working with the external connector with the Pi inside the case is a bit tricky :)

@rgov
Copy link

rgov commented Sep 28, 2023

The new UART header is presumably to make it easier to attach the Pi Debug Probe. Do we get GPIO 14/15 as a second UART, then? Can we use the UART header for something other than the serial console?

@ThomasKaiser
Copy link

Thank you. I'm just trying to figure out how the whole PCIe setup looks like...

Does the 'external' PCIe lane originate from the SoC or from RP1? When you force PCIe to Gen3 does this affect only the external 'slot' or also interconnection with RP1? Which USB controller is inside the thing?

The NIC has the RPi vendor ID (1de4:0001) but what about USB? Maybe some USB peripherals need to be connected for the controller showing up on the PCIe bus?

@will127534
Copy link

They got their own PCIe Vender ID! : D
Also interesting to see that most interface is packed in a single device and named as ethernet device. I wonder how is the CSI camera sending back the data through it.

@geerlingguy
Copy link
Owner Author

The new UART header is presumably to make it easier to attach the Pi Debug Probe. Do we get GPIO 14/15 as a second UART, then? Can we use the UART header for something other than the serial console?

@rgov you still get the separate UART via RP1 on GPIO 14/15, that one is configured as always via config.txt — the UART header is direct into the BCM2711 SoC.

Does the 'external' PCIe lane originate from the SoC or from RP1? When you force PCIe to Gen3 does this affect only the external 'slot' or also interconnection with RP1? Which USB controller is inside the thing?

@ThomasKaiser ext PCIe from SoC (so when you set dtparam=pciex1_gen=3 that only affects the external PCIe lane, not RP1).

The NIC has the RPi vendor ID (1de4:0001) but what about USB? Maybe some USB peripherals need to be connected for the controller showing up on the PCIe bus?

With a 1TB SSD (SanDisk Extreme) attached:

pi@pi5:~ $ lsusb
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 002: ID 0781:5588 SanDisk Corp. Extreme Pro
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
pi@pi5:~ $ sudo lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/2p, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
    |__ Port 1: Dev 2, If 0, Class=Mass Storage, Driver=uas, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/2p, 480M
pi@pi5:~ $ lspci
0001:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21)
0001:01:00.0 Ethernet controller: Device 1de4:0001

Not sure what magic they're pulling, it all seems to route through RP1 on those x4 lanes.

@DamonHD
Copy link

DamonHD commented Sep 28, 2023

Thanks for the idle power value: important for running off-grid, though still higher than my current 3B at about 1W...

https://www.earth.org.uk/note-on-Raspberry-Pi-3-setup.html

And a short note from me on the RPi5 power: http://www.earth.org.uk/note-on-site-technicals-76.html#2023-09-28

@cleverca22
Copy link

https://gist.github.com/cleverca22/97e46998b5fbd4e2f6772127c0cd034a
device tree in here, where you can see all of the fun bits in the rp1, including an adc!

@ThomasKaiser
Copy link

https://gist.github.com/cleverca22/97e46998b5fbd4e2f6772127c0cd034a

Thanks for this. So in the RP1 Ethernet is Cadence MACB/GEM IP and USB the usual Synopsys DesignWare 3 stuff while the SoC itself also seems to contain a RGMII capable GMAC (if RPi Trading Ltd. comes up with a new CM5 pinout they could maybe use this other Gigabit Ethernet as well)

@cleverca22
Copy link

ah, good catch, i missed that

@lukts30
Copy link

lukts30 commented Sep 28, 2023

Looks like it is still limited to USB OTG 2.0 like the Pi4 [1].
Rock 3/4 have USB OTG 3.0 but Rock 5 only has USB OTG 2.0 like all the raspberrypi's.

I need a SBC with USB OTG 3.0 for a fast piSCSI USB Drive.

@ThomasKaiser
Copy link

@geerlingguy have you tried to power the RPi 5 with something else than their new 5V/5A power brick?

Curious whether it supports higher profiles with supply voltages exceeding 5V or whether this limitation is a hard one? Also curious whether debugfs is enabled and something shows up below /sys/kernel/debug/usb/ that hints on which USB PD chip they use. On Rock 5B for example you get detailed negotiation status/process when doing this:

cat /sys/kernel/debug/usb/fusb302-4-0022 /sys/kernel/debug/usb/tcpm-4-0022 | sort

Also curious whether USB-C details show up when doing something like grep "" /sys/class/typec/port0/* 2>/dev/null?

And in general providing dmesg output would be great since answering a bunch of questions without us annoying you :)

@hexwab
Copy link

hexwab commented Sep 28, 2023

See video: Rasbperry Pi 5: EVERYTHING you need to know

Graph at 9:25 looks suspect: you appear to be using a cumulative plot. So for example the blue "108.9" point is actually at y position 108.9+54.9=163.8. Screenshot: https://sphere.chronosempire.org.uk/~HEx/shots/latency.png

Here's what I'd expect it to look like: https://sphere.chronosempire.org.uk/~HEx/pi5.svg

Code:

import matplotlib.pyplot as plt
from matplotlib import ticker

size = range(0,17)
boards = {
    'Pi 4': [0.0,0.0,0.0,0.0,0.0,0.0,4.7,7.2,10.3,11.9,22.7,80.9,108.9,129.4,139.8,145.1,156.5],
    'Pi 5': [0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.1,1.6,1.9,8.3,12.2,54.9,86.8,103.0,112.9,118.7]
}

plt.rcParams["figure.figsize"] = (12,8)
fig, ax = plt.subplots()
ax.xaxis.set_major_formatter(ticker.FuncFormatter(lambda n,pos: "%d"%(2**n)))
ax.grid()
for q in boards.keys():
    ax.plot(size, boards[q], marker='.', markeredgecolor='black',label=q)
    ax.fill_between(size, boards[q])
    for xy in zip(size,boards[q]):
  	    ax.annotate('%.1f' % xy[1], xy=xy, textcoords='data')
ax.legend(loc='upper left')
ax.set_title('Memory latency, single random read')
ax.set_xlabel('Size (KiB)')
ax.set_ylabel('Latency (ns)')
plt.savefig("pi5.svg")

@jack-ma
Copy link

jack-ma commented Sep 28, 2023

Looks like it is still limited to USB OTG 2.0 like the Pi4 [1]. Rock 3/4 have USB OTG 3.0 but Rock 5 only has USB OTG 2.0 like all the raspberrypi's.

@lukts30

ROCK 5A and ROCK 5B both has USB OTG 3.0, ROCK 5B OTG is at USB C. ROCK 5A OTG is at the upper double layer USB 3.0. Please check the Tech Spec section: https://radxa.com/rock5a/

@AndrewRentschler
Copy link

Now lets just hope there is plenty of supply!

@geerlingguy
Copy link
Owner Author

geerlingguy commented Sep 28, 2023

Graph at 9:25 looks suspect: you appear to be using a cumulative plot. So for example the blue "108.9" point is actually at y position 108.9+54.9=163.8.

@hexwab - Oh my! You're correct. Chalk that up to 'was up too late benchmarking' :O

I'm updating the graph and wording in my blog post.

@geerlingguy
Copy link
Owner Author

geerlingguy commented Sep 28, 2023

@ThomasKaiser - Full dmesg output is below the fold here:

Click to show full `dmesg` output
pi@pi5:~ $ dmesg
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x414fd0b1]
[    0.000000] Linux version 6.1.47-v8_16k+ (pi@pi5) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT Thu Sep 21 21:07:03 CDT 2023
[    0.000000] random: crng init done
[    0.000000] Machine model: Raspberry Pi 5 Model B Rev 1.0
[    0.000000] efi: UEFI not found.
[    0.000000] Reserved memory: created CMA memory pool at 0x000000001a000000, size 320 MiB
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x00000000ffffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   [mem 0x0000000100000000-0x00000001ffffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x000000000007ffff]
[    0.000000]   node   0: [mem 0x0000000000080000-0x000000003fbfffff]
[    0.000000]   node   0: [mem 0x0000000040000000-0x00000001ffffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x00000001ffffffff]
[    0.000000] On node 0, zone DMA: 256 pages in unavailable ranges
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: MIGRATE_INFO_TYPE not supported.
[    0.000000] psci: SMC Calling Convention v1.2
[    0.000000] percpu: Embedded 14 pages/cpu s176872 r8192 d44312 u229376
[    0.000000] pcpu-alloc: s176872 r8192 d44312 u229376 alloc=14*16384
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: detected: Virtualization Host Extensions
[    0.000000] CPU features: detected: Hardware dirty bit management
[    0.000000] CPU features: detected: Spectre-v4
[    0.000000] CPU features: detected: Spectre-BHB
[    0.000000] CPU features: kernel page table isolation forced ON by KASLR
[    0.000000] CPU features: detected: Kernel page table isolation (KPTI)
[    0.000000] alternatives: applying boot alternatives
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 521984
[    0.000000] Kernel command line: coherent_pool=1M 8250.nr_uarts=1 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1  smsc95xx.macaddr=B8:27:EB:AE:74:3B vc_mem.mem_base=0x3fc00000 vc_mem.mem_size=0x40000000  console=ttyAMA10,115200 console=tty1 root=PARTUUID=22c2ce57-02 rootfstype=ext4 fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles cfg80211.ieee80211_regdom=US
[    0.000000] Unknown kernel command line parameters "splash", will be passed to user space.
[    0.000000] Dentry cache hash table entries: 1048576 (order: 9, 8388608 bytes, linear)
[    0.000000] Inode-cache hash table entries: 524288 (order: 8, 4194304 bytes, linear)
[    0.000000] mem auto-init: stack:all(zero), heap alloc:off, heap free:off
[    0.000000] software IO TLB: area num 4.
[    0.000000] software IO TLB: mapped [mem 0x00000000fbffc000-0x00000000ffffc000] (64MB)
[    0.000000] Memory: 7915024K/8384512K available (12288K kernel code, 2156K rwdata, 4160K rodata, 4416K init, 1211K bss, 141808K reserved, 327680K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] ftrace: allocating 39682 entries in 39 pages
[    0.000000] ftrace: allocated 39 pages with 4 groups
[    0.000000] trace event string verifier disabled
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu: 	RCU event tracing is enabled.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=4.
[    0.000000] 	Trampoline variant of Tasks RCU enabled.
[    0.000000] 	Rude variant of Tasks RCU enabled.
[    0.000000] 	Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] Root IRQ handler: gic_handle_irq
[    0.000000] GIC: Using split EOI/Deactivate mode
[    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000] arch_timer: cp15 timer(s) running at 54.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xc743ce346, max_idle_ns: 440795203123 ns
[    0.000000] sched_clock: 56 bits at 54MHz, resolution 18ns, wraps every 4398046511102ns
[    0.000149] Console: colour dummy device 80x25
[    0.000171] printk: console [tty1] enabled
[    0.000188] Calibrating delay loop (skipped), value calculated using timer frequency.. 108.00 BogoMIPS (lpj=216000)
[    0.000194] pid_max: default: 32768 minimum: 301
[    0.000219] LSM: Security Framework initializing
[    0.000283] Mount-cache hash table entries: 16384 (order: 3, 131072 bytes, linear)
[    0.000308] Mountpoint-cache hash table entries: 16384 (order: 3, 131072 bytes, linear)
[    0.000739] cgroup: Disabling memory control group subsystem
[    0.001289] cblist_init_generic: Setting adjustable number of callback queues.
[    0.001294] cblist_init_generic: Setting shift to 2 and lim to 1.
[    0.001342] cblist_init_generic: Setting adjustable number of callback queues.
[    0.001344] cblist_init_generic: Setting shift to 2 and lim to 1.
[    0.001380] cblist_init_generic: Setting adjustable number of callback queues.
[    0.001382] cblist_init_generic: Setting shift to 2 and lim to 1.
[    0.001478] rcu: Hierarchical SRCU implementation.
[    0.001481] rcu: 	Max phase no-delay instances is 1000.
[    0.002117] EFI services will not be available.
[    0.002252] smp: Bringing up secondary CPUs ...
[    0.002511] Detected PIPT I-cache on CPU1
[    0.002562] CPU1: Booted secondary processor 0x0000000100 [0x414fd0b1]
[    0.002849] Detected PIPT I-cache on CPU2
[    0.002888] CPU2: Booted secondary processor 0x0000000200 [0x414fd0b1]
[    0.003160] Detected PIPT I-cache on CPU3
[    0.003195] CPU3: Booted secondary processor 0x0000000300 [0x414fd0b1]
[    0.003236] smp: Brought up 1 node, 4 CPUs
[    0.003241] SMP: Total of 4 processors activated.
[    0.003244] CPU features: detected: 32-bit EL0 Support
[    0.003247] CPU features: detected: Data cache clean to the PoU not required for I/D coherence
[    0.003250] CPU features: detected: Common not Private translations
[    0.003253] CPU features: detected: CRC32 instructions
[    0.003256] CPU features: detected: RCpc load-acquire (LDAPR)
[    0.003258] CPU features: detected: LSE atomic instructions
[    0.003261] CPU features: detected: Privileged Access Never
[    0.003263] CPU features: detected: RAS Extension Support
[    0.003267] CPU features: detected: Speculative Store Bypassing Safe (SSBS)
[    0.003308] CPU: All CPU(s) started at EL2
[    0.003311] alternatives: applying system-wide alternatives
[    0.005173] devtmpfs: initialized
[    0.008148] Enabled cp15_barrier support
[    0.008156] Enabled setend support
[    0.008224] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.008232] futex hash table entries: 1024 (order: 2, 65536 bytes, linear)
[    0.009195] pinctrl core: initialized pinctrl subsystem
[    0.009424] DMI not present or invalid.
[    0.009623] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.011107] DMA: preallocated 1024 KiB GFP_KERNEL pool for atomic allocations
[    0.011168] DMA: preallocated 1024 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.011255] DMA: preallocated 1024 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.011283] audit: initializing netlink subsys (disabled)
[    0.011358] audit: type=2000 audit(0.008:1): state=initialized audit_enabled=0 res=1
[    0.011515] thermal_sys: Registered thermal governor 'step_wise'
[    0.011535] cpuidle: using governor menu
[    0.011620] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.011668] ASID allocator initialised with 32768 entries
[    0.011955] Serial: AMBA PL011 UART driver
[    0.014047] bcm2835-mbox 107c013880.mailbox: mailbox enabled
[    0.014885] 107d001000.serial: ttyAMA10 at MMIO 0x107d001000 (irq = 15, base_baud = 0) is a PL011 rev2
[    0.014925] printk: console [ttyAMA10] enabled
[    0.024897] raspberrypi-firmware soc:firmware: Attached to firmware from 2023-09-13T10:37:06, variant start_cd
[    0.028899] raspberrypi-firmware soc:firmware: Firmware hash is 9d49431600000000000000000000000000000000
[    0.038434] KASLR enabled
[    0.047902] bcm2835-dma 1000010000.dma: DMA legacy API manager, dmachans=0x1
[    0.049194] iommu: Default domain type: Translated 
[    0.049198] iommu: DMA domain TLB invalidation policy: strict mode 
[    0.049308] SCSI subsystem initialized
[    0.049372] usbcore: registered new interface driver usbfs
[    0.049383] usbcore: registered new interface driver hub
[    0.049397] usbcore: registered new device driver usb
[    0.049519] usb_phy_generic phy: supply vcc not found, using dummy regulator
[    0.049617] pps_core: LinuxPPS API ver. 1 registered
[    0.049621] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <[email protected]>
[    0.049627] PTP clock support registered
[    0.049896] vgaarb: loaded
[    0.050040] clocksource: Switched to clocksource arch_sys_counter
[    0.050181] VFS: Disk quotas dquot_6.6.0
[    0.050202] VFS: Dquot-cache hash table entries: 2048 (order 0, 16384 bytes)
[    0.050246] FS-Cache: Loaded
[    0.050280] CacheFiles: Loaded
[    0.052351] NET: Registered PF_INET protocol family
[    0.052524] IP idents hash table entries: 131072 (order: 6, 1048576 bytes, linear)
[    0.054948] tcp_listen_portaddr_hash hash table entries: 4096 (order: 2, 65536 bytes, linear)
[    0.054996] Table-perturb hash table entries: 65536 (order: 4, 262144 bytes, linear)
[    0.055004] TCP established hash table entries: 65536 (order: 5, 524288 bytes, linear)
[    0.055275] TCP bind hash table entries: 65536 (order: 7, 2097152 bytes, linear)
[    0.057047] TCP: Hash tables configured (established 65536 bind 65536)
[    0.057113] UDP hash table entries: 4096 (order: 3, 131072 bytes, linear)
[    0.057214] UDP-Lite hash table entries: 4096 (order: 3, 131072 bytes, linear)
[    0.057378] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.057549] RPC: Registered named UNIX socket transport module.
[    0.057552] RPC: Registered udp transport module.
[    0.057555] RPC: Registered tcp transport module.
[    0.057557] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.057565] PCI: CLS 0 bytes, default 64
[    0.058133] hw perfevents: enabled with armv8_cortex_a76 PMU driver, 7 counters available
[    0.058274] kvm [1]: IPA Size Limit: 40 bits
[    0.058287] kvm [1]: GICV region size/alignment is unsafe, using trapping (reduced performance)
[    0.058344] kvm [1]: vgic interrupt IRQ9
[    0.058398] kvm [1]: VHE mode initialized successfully
[    0.350104] Initialise system trusted keyrings
[    0.350198] workingset: timestamp_bits=46 max_order=19 bucket_order=0
[    0.352123] zbud: loaded
[    0.352721] NFS: Registering the id_resolver key type
[    0.352734] Key type id_resolver registered
[    0.352737] Key type id_legacy registered
[    0.352759] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    0.352763] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
[    0.352820] F2FS not supported on PAGE_SIZE(16384) != 4096
[    0.352903] Key type asymmetric registered
[    0.352908] Asymmetric key parser 'x509' registered
[    0.352929] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[    0.352983] io scheduler mq-deadline registered
[    0.352988] io scheduler kyber registered
[    0.353598] irq_brcmstb_l2: registered L2 intc (/soc/interrupt-controller@7c502000, parent irq: 31)
[    0.353768] irq_brcmstb_l2: registered L2 intc (/soc/intc@7d503000, parent irq: 32)
[    0.353908] irq_brcmstb_l2: registered L2 intc (/soc/intc@7d508380, parent irq: 33)
[    0.354049] irq_brcmstb_l2: registered L2 intc (/soc/intc@7d508400, parent irq: 34)
[    0.354189] irq_brcmstb_l2: registered L2 intc (/soc/interrupt-controller@7d510600, parent irq: 35)
[    0.354327] irq_brcmstb_l2: registered L2 intc (/soc/intc@7d517b00, parent irq: 36)
[    0.356135] gpio-38 (RP1 RUN pin): hogged as output/high
[    0.390899] Serial: 8250/16550 driver, 1 ports, IRQ sharing enabled
[    0.391566] 107d50c000.serial: ttyS0 at MMIO 0x107d50c000 (irq = 38, base_baud = 6000000) is a 16550A
[    0.391621] serial serial0: tty port ttyS0 registered
[    0.392258] iproc-rng200 107d208000.rng: hwrng registered
[    0.392422] vc-mem: phys_addr:0x00000000 mem_base=0x3fc00000 mem_size:0x40000000(1024 MiB)
[    0.392794] rpi-gpiomem 107d508500.gpiomem: window base 0x107d508500 size 0x00000040
[    0.392832] rpi-gpiomem 107d508500.gpiomem: initialised 1 regions as /dev/gpiomem1
[    0.392910] rpi-gpiomem 107d517c00.gpiomem: window base 0x107d517c00 size 0x00000040
[    0.392944] rpi-gpiomem 107d517c00.gpiomem: initialised 1 regions as /dev/gpiomem2
[    0.393021] rpi-gpiomem 107d504100.gpiomem: window base 0x107d504100 size 0x00000020
[    0.393052] rpi-gpiomem 107d504100.gpiomem: initialised 1 regions as /dev/gpiomem3
[    0.393128] rpi-gpiomem 107d510700.gpiomem: window base 0x107d510700 size 0x00000020
[    0.393158] rpi-gpiomem 107d510700.gpiomem: initialised 1 regions as /dev/gpiomem4
[    0.393817] bcm2712-iommu-cache 1000005b00.iommuc: bcm2712_iommu_cache_probe
[    0.396256] brd: module loaded
[    0.398145] loop: module loaded
[    0.398400] bcm2835-power bcm2835-power: Broadcom BCM2835 power domains driver
[    0.398617] Loading iSCSI transport class v2.0-870.
[    0.399802] usbcore: registered new interface driver r8152
[    0.399817] usbcore: registered new interface driver lan78xx
[    0.399829] usbcore: registered new interface driver smsc95xx
[    0.399984] dwc_otg: version 3.00a 10-AUG-2012 (platform bus)
[    0.400077] dwc_otg: FIQ enabled
[    0.400080] dwc_otg: NAK holdoff enabled
[    0.400082] dwc_otg: FIQ split-transaction FSM enabled
[    0.400086] Module dwc_common_port init
[    0.400389] usbcore: registered new interface driver uas
[    0.400408] usbcore: registered new interface driver usb-storage
[    0.400643] mousedev: PS/2 mouse device common for all mice
[    0.408441] rpi-rtc soc:rpi_rtc: registered as rtc0
[    0.409906] rpi-rtc soc:rpi_rtc: setting system clock to 2023-09-28T15:58:43 UTC (1695916723)
[    0.410716] bcm2835-wdt bcm2835-wdt: Poweroff handler already present!
[    0.410721] bcm2835-wdt bcm2835-wdt: Broadcom BCM2835 watchdog timer
[    0.413721] sdhci: Secure Digital Host Controller Interface driver
[    0.413725] sdhci: Copyright(c) Pierre Ossman
[    0.413910] sdhci-pltfm: SDHCI platform and OF driver helper
[    0.414426] ledtrig-cpu: registered to indicate activity on CPUs
[    0.414468] SMCCC: SOC_ID: ARCH_SOC_ID not implemented, skipping ....
[    0.414481] hid: raw HID events driver (C) Jiri Kosina
[    0.414512] usbcore: registered new interface driver usbhid
[    0.414515] usbhid: USB HID core driver
[    0.414734] NET: Registered PF_PACKET protocol family
[    0.414768] Key type dns_resolver registered
[    0.415074] registered taskstats version 1
[    0.415097] Loading compiled-in X.509 certificates
[    0.415274] Key type .fscrypt registered
[    0.415277] Key type fscrypt-provisioning registered
[    0.417659] brcm-pcie 1000110000.pcie: host bridge /axi/pcie@110000 ranges:
[    0.417670] brcm-pcie 1000110000.pcie:   No bus range found for /axi/pcie@110000, using [bus 00-ff]
[    0.417700] brcm-pcie 1000110000.pcie:      MEM 0x1b00000000..0x1bfffffffb -> 0x0000000000
[    0.417714] brcm-pcie 1000110000.pcie:      MEM 0x1800000000..0x1affffffff -> 0x0400000000
[    0.417733] brcm-pcie 1000110000.pcie:   IB MEM 0x0000000000..0x0fffffffff -> 0x1000000000
[    0.418933] brcm-pcie 1000110000.pcie: setting SCB_ACCESS_EN, READ_UR_MODE, MAX_BURST_SIZE
[    0.418943] brcm-pcie 1000110000.pcie: Forcing gen 3
[    0.418991] brcm-pcie 1000110000.pcie: PCI host bridge to bus 0000:00
[    0.418996] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.419000] pci_bus 0000:00: root bus resource [mem 0x1b00000000-0x1bfffffffb] (bus address [0x00000000-0xfffffffb])
[    0.419004] pci_bus 0000:00: root bus resource [mem 0x1800000000-0x1affffffff pref] (bus address [0x400000000-0x6ffffffff])
[    0.419026] pci 0000:00:00.0: [14e4:2712] type 01 class 0x060400
[    0.419097] pci 0000:00:00.0: PME# supported from D0 D3hot
[    0.420058] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    0.738043] brcm-pcie 1000110000.pcie: link down
[    0.742719] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[    0.742729] pci 0000:00:00.0: PCI bridge to [bus 01]
[    0.742742] pci 0000:00:00.0: Max Payload Size set to  512/ 512 (was  128), Max Read Rq  512
[    0.742906] pcieport 0000:00:00.0: PME: Signaling with IRQ 39
[    0.743017] pcieport 0000:00:00.0: AER: enabled with IRQ 39
[    0.743190] pci_bus 0000:01: busn_res: [bus 01] is released
[    0.743261] pci_bus 0000:00: busn_res: [bus 00-ff] is released
[    0.743542] brcm-pcie 1000120000.pcie: host bridge /axi/pcie@120000 ranges:
[    0.743550] brcm-pcie 1000120000.pcie:   No bus range found for /axi/pcie@120000, using [bus 00-ff]
[    0.743578] brcm-pcie 1000120000.pcie:      MEM 0x1f00000000..0x1ffffffffb -> 0x0000000000
[    0.743592] brcm-pcie 1000120000.pcie:      MEM 0x1c00000000..0x1effffffff -> 0x0400000000
[    0.743622] brcm-pcie 1000120000.pcie:   IB MEM 0x1f00000000..0x1f003fffff -> 0x0000000000
[    0.743637] brcm-pcie 1000120000.pcie:   IB MEM 0x0000000000..0x0fffffffff -> 0x1000000000
[    0.744826] brcm-pcie 1000120000.pcie: setting SCB_ACCESS_EN, READ_UR_MODE, MAX_BURST_SIZE
[    0.744837] brcm-pcie 1000120000.pcie: Forcing gen 2
[    0.744880] brcm-pcie 1000120000.pcie: PCI host bridge to bus 0001:00
[    0.744884] pci_bus 0001:00: root bus resource [bus 00-ff]
[    0.744889] pci_bus 0001:00: root bus resource [mem 0x1f00000000-0x1ffffffffb] (bus address [0x00000000-0xfffffffb])
[    0.744893] pci_bus 0001:00: root bus resource [mem 0x1c00000000-0x1effffffff pref] (bus address [0x400000000-0x6ffffffff])
[    0.744912] pci 0001:00:00.0: [14e4:2712] type 01 class 0x060400
[    0.744979] pci 0001:00:00.0: PME# supported from D0 D3hot
[    0.745812] pci 0001:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    0.774045] brcm-pcie 1000120000.pcie: link up, 5.0 GT/s PCIe x4 (!SSC)
[    0.774072] pci 0001:01:00.0: [1de4:0001] type 00 class 0x020000
[    0.774092] pci 0001:01:00.0: reg 0x10: [mem 0xffffc000-0xffffffff]
[    0.774103] pci 0001:01:00.0: reg 0x14: [mem 0xffc00000-0xffffffff]
[    0.774113] pci 0001:01:00.0: reg 0x18: [mem 0xffff0000-0xffffffff]
[    0.774209] pci 0001:01:00.0: supports D1
[    0.774212] pci 0001:01:00.0: PME# supported from D0 D1 D3hot D3cold
[    0.786057] pci_bus 0001:01: busn_res: [bus 01-ff] end is updated to 01
[    0.786068] pci 0001:00:00.0: BAR 8: assigned [mem 0x1f00000000-0x1f005fffff]
[    0.786073] pci 0001:01:00.0: BAR 1: assigned [mem 0x1f00000000-0x1f003fffff]
[    0.786079] pci 0001:01:00.0: BAR 2: assigned [mem 0x1f00400000-0x1f0040ffff]
[    0.786086] pci 0001:01:00.0: BAR 0: assigned [mem 0x1f00410000-0x1f00413fff]
[    0.786093] pci 0001:00:00.0: PCI bridge to [bus 01]
[    0.786098] pci 0001:00:00.0:   bridge window [mem 0x1f00000000-0x1f005fffff]
[    0.786107] pci 0001:00:00.0: Max Payload Size set to  256/ 512 (was  128), Max Read Rq  512
[    0.786119] pci 0001:01:00.0: Max Payload Size set to  256/ 256 (was  128), Max Read Rq  512
[    0.786255] pcieport 0001:00:00.0: enabling device (0000 -> 0002)
[    0.786306] pcieport 0001:00:00.0: PME: Signaling with IRQ 40
[    0.786408] pcieport 0001:00:00.0: AER: enabled with IRQ 40
[    0.786596] rp1 0001:01:00.0: bar0 len 0x4000, start 0x1f00410000, end 0x1f00413fff, flags, 0x40200
[    0.786601] rp1 0001:01:00.0: bar1 len 0x400000, start 0x1f00000000, end 0x1f003fffff, flags, 0x40200
[    0.786614] rp1 0001:01:00.0: enabling device (0000 -> 0002)
[    0.787931] rp1 0001:01:00.0: chip_id 0x20001927
[    0.811314] macb 1f00100000.ethernet eth0: Cadence GEM rev 0x00070109 at 0x1f00100000 irq 108 (b8:27:eb:ae:74:3b)
[    0.812141] dw_axi_dmac_platform 1f00188000.dma: DesignWare AXI DMA Controller, 8 channels
[    0.812774] xhci-hcd xhci-hcd.0: xHCI Host Controller
[    0.812787] xhci-hcd xhci-hcd.0: new USB bus registered, assigned bus number 1
[    0.813249] xhci-hcd xhci-hcd.0: hcc params 0x0240fe6d hci version 0x110 quirks 0x0000000000010810
[    0.813267] xhci-hcd xhci-hcd.0: irq 133, io mem 0x1f00200000
[    0.813361] xhci-hcd xhci-hcd.0: xHCI Host Controller
[    0.813365] xhci-hcd xhci-hcd.0: new USB bus registered, assigned bus number 2
[    0.813370] xhci-hcd xhci-hcd.0: Host supports USB 3.0 SuperSpeed
[    0.813469] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.01
[    0.813474] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    0.813478] usb usb1: Product: xHCI Host Controller
[    0.813481] usb usb1: Manufacturer: Linux 6.1.47-v8_16k+ xhci-hcd
[    0.813485] usb usb1: SerialNumber: xhci-hcd.0
[    0.813648] hub 1-0:1.0: USB hub found
[    0.813669] hub 1-0:1.0: 2 ports detected
[    0.813910] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.01
[    0.813915] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    0.813918] usb usb2: Product: xHCI Host Controller
[    0.813921] usb usb2: Manufacturer: Linux 6.1.47-v8_16k+ xhci-hcd
[    0.813924] usb usb2: SerialNumber: xhci-hcd.0
[    0.814066] hub 2-0:1.0: USB hub found
[    0.814085] hub 2-0:1.0: 1 port detected
[    0.814772] xhci-hcd xhci-hcd.1: xHCI Host Controller
[    0.814778] xhci-hcd xhci-hcd.1: new USB bus registered, assigned bus number 3
[    0.815239] xhci-hcd xhci-hcd.1: hcc params 0x0240fe6d hci version 0x110 quirks 0x0000000000010810
[    0.815258] xhci-hcd xhci-hcd.1: irq 138, io mem 0x1f00300000
[    0.815352] xhci-hcd xhci-hcd.1: xHCI Host Controller
[    0.815356] xhci-hcd xhci-hcd.1: new USB bus registered, assigned bus number 4
[    0.815361] xhci-hcd xhci-hcd.1: Host supports USB 3.0 SuperSpeed
[    0.815444] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.01
[    0.815448] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    0.815452] usb usb3: Product: xHCI Host Controller
[    0.815455] usb usb3: Manufacturer: Linux 6.1.47-v8_16k+ xhci-hcd
[    0.815458] usb usb3: SerialNumber: xhci-hcd.1
[    0.815597] hub 3-0:1.0: USB hub found
[    0.815616] hub 3-0:1.0: 2 ports detected
[    0.815845] usb usb4: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.01
[    0.815849] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    0.815853] usb usb4: Product: xHCI Host Controller
[    0.815856] usb usb4: Manufacturer: Linux 6.1.47-v8_16k+ xhci-hcd
[    0.815859] usb usb4: SerialNumber: xhci-hcd.1
[    0.815988] hub 4-0:1.0: USB hub found
[    0.816005] hub 4-0:1.0: 1 port detected
[    0.816480] rpi-gpiomem 1f000d0000.gpiomem: window base 0x1f000d0000 size 0x00030000
[    0.816519] rpi-gpiomem 1f000d0000.gpiomem: initialised 1 regions as /dev/gpiomem0
[    0.817224] bcm2712-iommu 1000005100.iommu: bcm2712_iommu_init: DEBUG_INFO = 0x20804774
[    0.817655] platform 1000880000.pisp_be: bcm2712_iommu_probe_device: MMU 1000005100.iommu
[    0.817659] platform 1000880000.pisp_be: bcm2712_iommu_device_group: MMU 1000005100.iommu
[    0.817665] platform 1000880000.pisp_be: Adding to iommu group 0
[    0.817714] platform 1000880000.pisp_be: bcm2712_iommu_attach_dev: MMU 1000005100.iommu
[    0.817722] bcm2712-iommu 1000005100.iommu: bcm2712_iommu_probe: Success
[    0.818422] bcm2712-iommu 1000005200.iommu: bcm2712_iommu_init: DEBUG_INFO = 0x20804774
[    0.818788] platform axi:gpu: bcm2712_iommu_probe_device: MMU 1000005200.iommu
[    0.818791] platform axi:gpu: bcm2712_iommu_device_group: MMU 1000005200.iommu
[    0.818796] platform axi:gpu: Adding to iommu group 1
[    0.818857] platform axi:gpu: bcm2712_iommu_attach_dev: MMU 1000005200.iommu
[    0.818864] bcm2712-iommu 1000005200.iommu: bcm2712_iommu_probe: Success
[    0.819408] bcm2712-iommu 1000005280.iommu: bcm2712_iommu_init: DEBUG_INFO = 0x20804774
[    0.819802] bcm2712-iommu 1000005280.iommu: bcm2712_iommu_probe: Success
[    0.820052] sdhci-brcmstb 1000fff000.mmc: there is not valid maps for state default
[    0.820458] mmc0: CQHCI version 5.10
[    0.820965] mmc1: CQHCI version 5.10
[    0.822144] of_cfs_init
[    0.822183] of_cfs_init: OK
[    0.854045] mmc0: SDHCI controller on 1000fff000.mmc [1000fff000.mmc] using ADMA 64-bit
[    0.953059] mmc0: new ultra high speed SDR104 SDXC card at address e624
[    0.953381] mmcblk0: mmc0:e624 SR128 119 GiB 
[    0.954871]  mmcblk0: p1 p2
[    0.954987] mmcblk0: mmc0:e624 SR128 119 GiB (quirks 0x00004000)
[    1.008591] mmc1: SDHCI controller on 1001100000.mmc [1001100000.mmc] using ADMA 64-bit
[    1.026882] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Quota mode: none.
[    1.026904] VFS: Mounted root (ext4 filesystem) readonly on device 179:2.
[    1.032354] devtmpfs: mounted
[    1.034883] Freeing unused kernel memory: 4416K
[    1.034932] Run /sbin/init as init process
[    1.034935]   with arguments:
[    1.034937]     /sbin/init
[    1.034940]     splash
[    1.034942]   with environment:
[    1.034944]     HOME=/
[    1.034946]     TERM=linux
[    1.041762] mmc1: new ultra high speed DDR50 SDIO card at address 0001
[    1.290247] usb 2-1: new SuperSpeed USB device number 2 using xhci-hcd
[    1.315245] usb 2-1: New USB device found, idVendor=0781, idProduct=5588, bcdDevice= 1.00
[    1.315251] usb 2-1: New USB device strings: Mfr=2, Product=3, SerialNumber=1
[    1.315255] usb 2-1: Product: SanDisk 3.2 Gen 1
[    1.315259] usb 2-1: Manufacturer: SanDisk
[    1.315262] usb 2-1: SerialNumber: 51B456789E2A
[    1.317859] scsi host0: uas
[    1.318412] scsi 0:0:0:0: Direct-Access     SanDisk  3.2 Gen 1        0    PQ: 0 ANSI: 6
[    1.324453] sd 0:0:0:0: [sda] 1930199040 512-byte logical blocks: (988 GB/920 GiB)
[    1.324606] sd 0:0:0:0: [sda] Write Protect is off
[    1.324611] sd 0:0:0:0: [sda] Mode Sense: 43 00 00 00
[    1.324795] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    1.324998] sd 0:0:0:0: [sda] Preferred minimum I/O size 512 bytes
[    1.325001] sd 0:0:0:0: [sda] Optimal transfer size 33553920 bytes
[    1.327323]  sda: sda1
[    1.327411] sd 0:0:0:0: [sda] Attached SCSI disk
[    1.331343] NET: Registered PF_INET6 protocol family
[    1.331790] Segment Routing with IPv6
[    1.331799] In-situ OAM (IOAM) with IPv6
[    1.365988] systemd[1]: systemd 252.12-1~deb12u1 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    1.366009] systemd[1]: Detected architecture arm64.
[    1.376868] systemd[1]: Hostname set to <pi5>.
[    1.435106] uart-pl011 107d001000.serial: no DMA platform data
[    1.685649] systemd[1]: Queued start job for default target graphical.target.
[    1.751486] systemd[1]: Created slice system-getty.slice - Slice /system/getty.
[    1.752118] systemd[1]: Created slice system-modprobe.slice - Slice /system/modprobe.
[    1.752637] systemd[1]: Created slice system-serial\x2dgetty.slice - Slice /system/serial-getty.
[    1.753154] systemd[1]: Created slice system-systemd\x2dfsck.slice - Slice /system/systemd-fsck.
[    1.753530] systemd[1]: Created slice user.slice - User and Session Slice.
[    1.753658] systemd[1]: Started systemd-ask-password-wall.path - Forward Password Requests to Wall Directory Watch.
[    1.753939] systemd[1]: Set up automount proc-sys-fs-binfmt_misc.automount - Arbitrary Executable File Formats File System Automount Point.
[    1.754018] systemd[1]: Reached target integritysetup.target - Local Integrity Protected Volumes.
[    1.754098] systemd[1]: Reached target nss-user-lookup.target - User and Group Name Lookups.
[    1.754141] systemd[1]: Reached target slices.target - Slice Units.
[    1.754170] systemd[1]: Reached target swap.target - Swaps.
[    1.754207] systemd[1]: Reached target veritysetup.target - Local Verity Protected Volumes.
[    1.754422] systemd[1]: Listening on systemd-fsckd.socket - fsck to fsckd communication Socket.
[    1.754515] systemd[1]: Listening on systemd-initctl.socket - initctl Compatibility Named Pipe.
[    1.754954] systemd[1]: Listening on systemd-journald-audit.socket - Journal Audit Socket.
[    1.755160] systemd[1]: Listening on systemd-journald-dev-log.socket - Journal Socket (/dev/log).
[    1.755378] systemd[1]: Listening on systemd-journald.socket - Journal Socket.
[    1.759238] systemd[1]: Listening on systemd-udevd-control.socket - udev Control Socket.
[    1.759425] systemd[1]: Listening on systemd-udevd-kernel.socket - udev Kernel Socket.
[    1.759653] systemd[1]: dev-hugepages.mount - Huge Pages File System was skipped because of an unmet condition check (ConditionPathExists=/sys/kernel/mm/hugepages).
[    1.760935] systemd[1]: Mounting dev-mqueue.mount - POSIX Message Queue File System...
[    1.762471] systemd[1]: Mounting sys-kernel-debug.mount - Kernel Debug File System...
[    1.764144] systemd[1]: Mounting sys-kernel-tracing.mount - Kernel Trace File System...
[    1.764495] systemd[1]: auth-rpcgss-module.service - Kernel Module supporting RPCSEC_GSS was skipped because of an unmet condition check (ConditionPathExists=/etc/krb5.keytab).
[    1.767301] systemd[1]: Starting fake-hwclock.service - Restore / save the current clock...
[    1.770593] systemd[1]: Starting keyboard-setup.service - Set the console keyboard layout...
[    1.772226] systemd[1]: Starting kmod-static-nodes.service - Create List of Static Device Nodes...
[    1.774152] systemd[1]: Starting [email protected] - Load Kernel Module configfs...
[    1.776062] systemd[1]: Starting modprobe@dm_mod.service - Load Kernel Module dm_mod...
[    1.778020] systemd[1]: Starting [email protected] - Load Kernel Module drm...
[    1.780032] systemd[1]: Starting modprobe@efi_pstore.service - Load Kernel Module efi_pstore...
[    1.781944] systemd[1]: Starting [email protected] - Load Kernel Module fuse...
[    1.783969] systemd[1]: Starting [email protected] - Load Kernel Module loop...
[    1.786083] systemd[1]: Starting systemd-fsck-root.service - File System Check on Root Device...
[    1.789218] systemd[1]: Starting systemd-journald.service - Journal Service...
[    1.806451] systemd[1]: Starting systemd-modules-load.service - Load Kernel Modules...
[    1.808608] systemd[1]: Starting systemd-udev-trigger.service - Coldplug All udev Devices...
[    1.811861] systemd[1]: Mounted dev-mqueue.mount - POSIX Message Queue File System.
[    1.812108] systemd[1]: Mounted sys-kernel-debug.mount - Kernel Debug File System.
[    1.812288] systemd[1]: Mounted sys-kernel-tracing.mount - Kernel Trace File System.
[    1.812733] systemd[1]: Finished kmod-static-nodes.service - Create List of Static Device Nodes.
[    1.813527] systemd[1]: [email protected]: Deactivated successfully.
[    1.813742] device-mapper: ioctl: 4.47.0-ioctl (2022-07-28) initialised: [email protected]
[    1.813864] systemd[1]: Finished [email protected] - Load Kernel Module configfs.
[    1.814567] systemd[1]: modprobe@efi_pstore.service: Deactivated successfully.
[    1.814843] systemd[1]: Finished modprobe@efi_pstore.service - Load Kernel Module efi_pstore.
[    1.815414] systemd[1]: [email protected]: Deactivated successfully.
[    1.815733] systemd[1]: Finished [email protected] - Load Kernel Module loop.
[    1.818134] systemd[1]: Mounting sys-kernel-config.mount - Kernel Configuration File System...
[    1.821315] systemd[1]: Started systemd-fsckd.service - File System Check Daemon to report status.
[    1.825126] systemd[1]: modprobe@dm_mod.service: Deactivated successfully.
[    1.825389] systemd[1]: Finished modprobe@dm_mod.service - Load Kernel Module dm_mod.
[    1.825722] systemd[1]: Mounted sys-kernel-config.mount - Kernel Configuration File System.
[    1.825951] systemd[1]: systemd-repart.service - Repartition Root Disk was skipped because no trigger condition checks were met.
[    1.831203] fuse: init (API version 7.37)
[    1.832160] systemd[1]: [email protected]: Deactivated successfully.
[    1.832400] systemd[1]: Finished [email protected] - Load Kernel Module fuse.
[    1.834488] systemd[1]: Mounting sys-fs-fuse-connections.mount - FUSE Control File System...
[    1.837609] systemd[1]: Mounted sys-fs-fuse-connections.mount - FUSE Control File System.
[    1.850583] i2c_dev: i2c /dev entries driver
[    1.851648] systemd[1]: Finished systemd-modules-load.service - Load Kernel Modules.
[    1.853385] systemd[1]: Starting systemd-sysctl.service - Apply Kernel Variables...
[    1.876209] systemd[1]: Finished fake-hwclock.service - Restore / save the current clock.
[    1.877671] systemd[1]: [email protected]: Deactivated successfully.
[    1.877919] systemd[1]: Finished [email protected] - Load Kernel Module drm.
[    1.886219] systemd[1]: Started systemd-journald.service - Journal Service.
[    2.059813] EXT4-fs (mmcblk0p2): re-mounted. Quota mode: none.
[    2.075644] systemd-journald[146]: Received client request to flush runtime journal.
[    2.885798] input: pwr_button as /devices/platform/pwr_button/input/input0
[    2.894851] [drm] Initialized v3d 1.0.0 20180419 for 1002000000.v3d on minor 0
[    2.908319] mc: Linux media interface: v0.10
[    2.923824] spi-bcm2835 107d004000.spi: no tx-dma configuration found - not using dma mode
[    2.932866] brcmstb-i2c 107d508200.i2c:  @200000hz registered in interrupt mode
[    2.939066] videodev: Linux video capture interface: v2.00
[    2.941042] brcmstb-i2c 107d508280.i2c:  @200000hz registered in interrupt mode
[    3.011467] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[    3.021292] cfg80211: Loaded X.509 cert '[email protected]: 577e021cb980e0e820821ba7b54b4961b8b4fadf'
[    3.021619] cfg80211: Loaded X.509 cert '[email protected]: 3abbc6ec146e09d1b6016ab9d6cf71dd233f0328'
[    3.021932] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[    3.062663] pispbe 1000880000.pisp_be: bcm2712_iommu_of_xlate: MMU 1000005100.iommu
[    3.062990] pispbe 1000880000.pisp_be: pispbe_probe: HW version:  0x02252700
[    3.062999] pispbe 1000880000.pisp_be: pispbe_probe: BatchStatus: 0x00000000
[    3.063003] pispbe 1000880000.pisp_be: pispbe_probe: Status:      0x00000000
[    3.063007] pispbe 1000880000.pisp_be: Register nodes for group 0
[    3.063160] pispbe 1000880000.pisp_be: input device node registered as /dev/video20
[    3.063942] pispbe 1000880000.pisp_be: tdn_input device node registered as /dev/video21
[    3.065843] pispbe 1000880000.pisp_be: stitch_input device node registered as /dev/video22
[    3.069295] brcmfmac: F1 signature read @0x18000000=0x15264345
[    3.071733] pispbe 1000880000.pisp_be: hog_output device node registered as /dev/video23
[    3.073181] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[    3.073472] rpivid_hevc: module is from the staging directory, the quality is unknown, you have been warned.
[    3.074795] brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43455-sdio.raspberrypi,5-model-b.bin failed with error -2
[    3.078941] pispbe 1000880000.pisp_be: output0 device node registered as /dev/video24
[    3.079821] pispbe 1000880000.pisp_be: output1 device node registered as /dev/video25
[    3.080025] pispbe 1000880000.pisp_be: tdn_output device node registered as /dev/video26
[    3.080167] pispbe 1000880000.pisp_be: stitch_output device node registered as /dev/video27
[    3.080345] pispbe 1000880000.pisp_be: config device node registered as /dev/video28
[    3.081164] usbcore: registered new interface driver brcmfmac
[    3.082823] rpivid 1000800000.codec: Device registered as /dev/video19
[    3.083201] pispbe 1000880000.pisp_be: Register nodes for group 1
[    3.083953] pispbe 1000880000.pisp_be: input device node registered as /dev/video29
[    3.084819] pispbe 1000880000.pisp_be: tdn_input device node registered as /dev/video30
[    3.099928] pispbe 1000880000.pisp_be: stitch_input device node registered as /dev/video31
[    3.100748] pispbe 1000880000.pisp_be: hog_output device node registered as /dev/video32
[    3.101365] pispbe 1000880000.pisp_be: output0 device node registered as /dev/video33
[    3.102020] pispbe 1000880000.pisp_be: output1 device node registered as /dev/video34
[    3.102646] pispbe 1000880000.pisp_be: tdn_output device node registered as /dev/video35
[    3.103785] pispbe 1000880000.pisp_be: stitch_output device node registered as /dev/video36
[    3.109100] pispbe 1000880000.pisp_be: config device node registered as /dev/video37
[    3.236508] Bluetooth: Core ver 2.22
[    3.236551] NET: Registered PF_BLUETOOTH protocol family
[    3.236555] Bluetooth: HCI device and connection manager initialized
[    3.236564] Bluetooth: HCI socket layer initialized
[    3.236569] Bluetooth: L2CAP socket layer initialized
[    3.236578] Bluetooth: SCO socket layer initialized
[    3.239104] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Nov  1 2021 00:37:25 version 7.45.241 (1a2f2fa CY) FWID 01-703fd60
[    3.261423] Bluetooth: HCI UART driver ver 2.3
[    3.261434] Bluetooth: HCI UART protocol H4 registered
[    3.261490] Bluetooth: HCI UART protocol Three-wire (H5) registered
[    3.271661] hci_uart_bcm serial0-0: supply vbat not found, using dummy regulator
[    3.273719] Bluetooth: HCI UART protocol Broadcom registered
[    3.280617] hci_uart_bcm serial0-0: supply vddio not found, using dummy regulator
[    3.342833] sd 0:0:0:0: Attached scsi generic sg0 type 0
[    3.642648] Bluetooth: hci0: BCM: chip id 107
[    3.642943] Bluetooth: hci0: BCM: features 0x2f
[    3.644125] Bluetooth: hci0: BCM4345C0
[    3.644134] Bluetooth: hci0: BCM4345C0 (003.001.025) build 0000
[    3.644236] vc4-drm axi:gpu: bcm2712_iommu_of_xlate: MMU 1000005200.iommu
[    3.648079] vc4-drm axi:gpu: bound 107c580000.hvs (ops vc4_hvs_ops [vc4])
[    3.651794] Bluetooth: hci0: BCM4345C0 'brcm/BCM4345C0.hcd' Patch
[    3.652204] Registered IR keymap rc-cec
[    3.652265] rc rc0: vc4-hdmi-0 as /devices/platform/soc/107c701400.hdmi/rc/rc0
[    3.652331] input: vc4-hdmi-0 as /devices/platform/soc/107c701400.hdmi/rc/rc0/input1
[    3.655594] vc4-drm axi:gpu: bound 107c701400.hdmi (ops vc4_hdmi_ops [vc4])
[    3.660710] Registered IR keymap rc-cec
[    3.660784] rc rc1: vc4-hdmi-1 as /devices/platform/soc/107c706400.hdmi/rc/rc1
[    3.660842] input: vc4-hdmi-1 as /devices/platform/soc/107c706400.hdmi/rc/rc1/input2
[    3.684290] vc4-drm axi:gpu: bound 107c706400.hdmi (ops vc4_hdmi_ops [vc4])
[    3.684494] vc4-drm axi:gpu: bound 107c500000.mop (ops vc4_txp_ops [vc4])
[    3.684609] vc4-drm axi:gpu: bound 107c501000.moplet (ops vc4_txp_ops [vc4])
[    3.684724] vc4-drm axi:gpu: bound 107c410000.pixelvalve (ops vc4_crtc_ops [vc4])
[    3.684830] vc4-drm axi:gpu: bound 107c411000.pixelvalve (ops vc4_crtc_ops [vc4])
[    3.685790] [drm] Initialized vc4 0.0.0 20140616 for axi:gpu on minor 1
[    3.689887] vc4-drm axi:gpu: [drm] Cannot find any crtc or sizes
[    3.693899] vc4-drm axi:gpu: [drm] Cannot find any crtc or sizes
[    3.697761] vc4-drm axi:gpu: [drm] Cannot find any crtc or sizes
[    4.055827] Adding 102384k swap on /var/swap.  Priority:-2 extents:1 across:102384k SSFS
[    4.338765] Bluetooth: hci0: BCM: features 0x2f
[    4.340161] Bluetooth: hci0: BCM43455 37.4MHz Raspberry Pi 3+-0190
[    4.340166] Bluetooth: hci0: BCM4345C0 (003.001.025) build 0382
[    4.340480] Bluetooth: hci0: BCM: Using default device address (43:45:c0:00:1f:ac)
[    4.631212] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[    4.631221] Bluetooth: BNEP filters: protocol multicast
[    4.631230] Bluetooth: BNEP socket layer initialized
[    4.637365] Bluetooth: MGMT ver 1.22
[    4.643890] NET: Registered PF_ALG protocol family
[    5.147720] macb 1f00100000.ethernet eth0: PHY [1f00100000.ethernet-ffffffff:01] driver [Broadcom BCM54213PE] (irq=POLL)
[    5.147729] macb 1f00100000.ethernet eth0: configuring for phy/rgmii-id link mode
[    5.150205] pps pps0: new PPS source ptp0
[    5.150274] macb 1f00100000.ethernet: gem-ptp-timer ptp clock registered.
[    5.184682] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save enabled
[    7.710745] Bluetooth: RFCOMM TTY layer initialized
[    7.710760] Bluetooth: RFCOMM socket layer initialized
[    7.710771] Bluetooth: RFCOMM ver 1.11
[    9.247310] macb 1f00100000.ethernet eth0: Link is Up - 1Gbps/Full - flow control tx
[    9.247348] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   10.406380] EXT4-fs (sda1): mounted filesystem with ordered data mode. Quota mode: none.
[   11.049112] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[   14.132840] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[   14.139623] Bridge firewalling registered
[   14.290571] Initializing XFRM netlink socket

@FCLC
Copy link

FCLC commented Sep 28, 2023

Following up on https://mast.hpc.social/deck/@[email protected]/111143860188995259

2 questions:

CPU: What ISA extensions are supported by the chip? the A76 should be an Aarch64 chip that implements baseline Arm v8.2 with select extensions such as the Dot product extensions from V8.4 the speculative load/store memory extensions from V8.5 amongst others (lscpu -e should do it? at worse cat /proc/cpuinfo >> ~/cpuinfo_dump.txt) Answered here #21 (comment)

GPU/Multimedia ASIC: What formats/profile does the HEVC decoder support? is it 8bit 420 only? does it implement 10bit (required for "proper" HDR)?
Easiest way to check with be vainfo, but that assumes that VAAPI is implemented and that it's installed

@Headcrabed
Copy link

Hello, can you test glmark2 and glmark2-es2 score and do a simple comparison between rpi4b?

@hexwab
Copy link

hexwab commented Sep 28, 2023

@hexwab - Oh my! You're correct. Chalk that up to 'was up too late benchmarking' :O

I'm updating the graph and wording in my blog post.

Cheers. While I'm nitpicking: have you considered SVG graphs (rather than JPEG of all things)?

@ThomasKaiser
Copy link

What ISA extensions are supported by the chip?

Either grep for "Flags" here or see there: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp

@lukasMega
Copy link

lukasMega commented Sep 28, 2023

@geerlingguy, do you plan to focus more on "tweaking" in order to achieve the lowest possible idle power consumption? (like disabling hdmi, wifi, bluetooth, undervolting, …)
BTW, thanks for your great report and video about RPi 5!

//edit: I like how they did indicator of memory version (8G, 4G, 2G, 1G) it's like <input type="radio" value="8G" checked /> in real life.
//edit2: Why there are 2022 ?! It's written on the board under USB 2 header. It was produced in 2022?

@DamonHD
Copy link

DamonHD commented Sep 28, 2023

I would certainly like to know that set of tweaks for my application... B^>

@FCLC
Copy link

FCLC commented Sep 28, 2023

What ISA extensions are supported by the chip?

Either grep for "Flags" here or see there: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp

Gotcha, so looks like we get
neon (ASIMD),
FP16 (FPHP + asimdhp or "Advanced SIMD half precision),
FP64 FMA( asimdrdm or Rounding Double Multiply Accumulate/Subtract)
SIMD dot products (asimddp)

@ThomasKaiser
Copy link

device tree in here, where you can see all of the fun bits in the rp1, including an adc!

BTW: sbc-bench output collected by Jeff shows lm-sensors output:

rp1_adc-isa-0000
in1:           1.47 V  
in2:           2.53 V  
in3:           1.41 V  
in4:           1.38 V  
temp1:        +55.5 C

Also accessible via sysfs. There it's hwmon1:

/sys/class/hwmon/hwmon1/in1_input:1475
/sys/class/hwmon/hwmon1/in1_raw:1824
/sys/class/hwmon/hwmon1/in2_input:2539
/sys/class/hwmon/hwmon1/in2_raw:3149
/sys/class/hwmon/hwmon1/in3_input:1413
/sys/class/hwmon/hwmon1/in3_raw:1544
/sys/class/hwmon/hwmon1/in4_input:1385
/sys/class/hwmon/hwmon1/in4_raw:1522
/sys/class/hwmon/hwmon1/name:rp1_adc
/sys/class/hwmon/hwmon1/temp1_input:51985
/sys/class/hwmon/hwmon1/temp1_raw:823
/sys/class/hwmon/hwmon1/uevent:OF_NAME=adc
/sys/class/hwmon/hwmon1/uevent:OF_FULLNAME=/axi/pcie@120000/rp1/adc@c8000
/sys/class/hwmon/hwmon1/uevent:OF_COMPATIBLE_0=raspberrypi,rp1-adc
/sys/class/hwmon/hwmon1/uevent:OF_COMPATIBLE_N=1

@geerlingguy
Copy link
Owner Author

Hello, can you test glmark2 and glmark2-es2 score and do a simple comparison between rpi4b?

@Headcrabed - mentioned in my video (linked earlier), GLMark2 scores:

  • 117 fullscreen
  • 905 windowed

@theofficialgman
Copy link

Threadx is now open source https://www.theregister.com/AMP/2023/11/28/microsoft_opens_sources_threadx/

Hopefully raspberry pi LTD follows suit and open sources their implementation so we can have a truly open bootstack.

@fanoush
Copy link

fanoush commented Nov 29, 2023

Hopefully raspberry pi LTD follows suit and open sources their implementation so we can have a truly open bootstack.

there is no rationale for such conclusion

discussed also here https://forums.raspberrypi.com/viewtopic.php?t=360475

@StuartIanNaylor
Copy link

Hopefully raspberry pi LTD follows suit and open sources their implementation so we can have a truly open bootstack.

Haven't read the forum but its not Raspberry the Video-Core loader is prop of Broadcom and likely fat chance.

@theofficialgman
Copy link

theofficialgman commented Nov 29, 2023

Hopefully raspberry pi LTD follows suit and open sources their implementation so we can have a truly open bootstack.

there is no rationale for such conclusion

discussed also here https://forums.raspberrypi.com/viewtopic.php?t=360475

I've read the forum and issues linked there. It seems that upstream threadx being open source now places the only barrier on raspberry pi open sourcing their implementation on broadcom, rather than broadcom and microsoft (who wholly own Express Logic Inc). Lets hope broadcom rebases their version on upstream MIT licensed threadx and decides to release their changes under a compatible license. As you can see from the blog post from eclipse https://eclipse-foundation.blog/2023/11/21/introducing-eclipse-threadx/ they do want a future when all efforts are merged upstream.

@fanoush
Copy link

fanoush commented Nov 29, 2023

and decides to release their changes under a compatible license.

Again, there is no reason for this. If Microsoft opensources Windows will Adobe make Photoshop opensource (because it runs on top of Windows)?

@theofficialgman
Copy link

theofficialgman commented Nov 29, 2023

and decides to release their changes under a compatible license.

Again, there is no reason for this. If Microsoft opensources Windows will Adobe make Photoshop opensource?

clearly you are not a software developer and do not understand this relationship. Your analogy is extremely misguided. adobe photoshop is not based on Windows and is not an implementation of Windows which your analogy would imply.

the bootloader that runs on the broadcom GPU is an implementation based on ThreadX with Broadcom proprietary patches applied on top. They are not separate software such as Windows and Photoshop. Now that the base ThreadX has an MIT licensed version, assuming rpi/broadcom rebase their changes ontop of the upstream MIT licensed ThreadX, the only blocker is rpi/broadcom's own code. Previously, in order to release their the pi bootloader as FOSS, they would have to obtain permission from Microsoft (which they would never give). That is no longer the case.

@dwillmore
Copy link

dwillmore commented Nov 29, 2023 via email

@theofficialgman
Copy link

theofficialgman commented Nov 29, 2023

No, ThreadX is an OS and the firmware of the Rpi boards is an application that runs on ThreadX. fanoush was precisely correct in their analogy.

incorrect. the stage 2 bootloader is the threadx OS with broadcom patches applied on top and is not simply an application that runs on an unmodified threadx OS. As you can see from the official supported architectures of threadx https://github.com/azure-rtos/threadx/#supported-architecture-ports , the broadcom gpu architectures are not official supported architectures. Thus, broadcom's patches involve at least partially, changes necessary to threadX OS itself to run on their gpu architectures. Of course broadcom has developed applications that run ontop of the threadx OS and those have always been wholly owned by rpi/broadcom.

@Headcrabed
Copy link

@theofficialgman Any news from upstream maintainer about that schci patch?

@ThomasKaiser
Copy link

ThomasKaiser commented Nov 29, 2023

Hopefully raspberry pi LTD follows suit and open sources their implementation so we can have a truly open bootstack.

Not really sure how decisions for the future affect decisions being made prior to even BroadCom having bought Alphamosaic: https://forums.raspberrypi.com/viewtopic.php?p=1460642#p1460642

And we should always remember: Raspberry Pi Ltd. is a catch basin for VideoCore guys. When Eben Upton (CEO), Gordon Hollingworth (CTO software) and James Adams (CTO hardware) talk in an interview about what 'we did together in the past' then mostly they refer to a situation where they were Broadcom employees (all three for example members of the VC4 design team and Adams even becoming BroadCom employee by acquisition of his former employer and VideoCore inventor Alphamosaic).

@applecuckoo
Copy link

@Headcrabed Maybe it's a good idea to try to send something on that linux-mmc mailing list? That email is [email protected], maybe Adrian will respond there.

@marcelgsteiger
Copy link

marcelgsteiger commented Dec 4, 2023

MD Raid10 on two SDXC Cards 128GB Samsung Pro Plus w/USB Adapter or (one of them) in SD cardslot

RPI5B w/4 GiB RAM
Array created with:
mdadm --create /dev/mdx --level=10 --metadata=1.2 --raid-devices=2 --layout=o2 --chunk=2M --bitmap=internal
both cards in USB adapter: benchmarkresult.txt
very fast sequential speeds, random write 1M is faster (compared with the SanDisk Extreme SD) despite the fact that it has to write to 2 mirror devices.
Both cards in Samsung USB adapters supplied with the cards.

One card moved to internal SD slot: benchmarkresult_sdcardreader.txt
Random I/O is comparable, but sequential I/O is only one third. The built-in SD card slot appears to be a major bottleneck here.
In practice, this might not be very important - the cases in which iozone shows extraordinary results are normally better handled by application side caching / buffering.

These tests show that we can mirror the SD card with a second card via USB adapter - so we get better availability without losing performance. In my application, I mirror the bootfs partition as well (using RAID 1 w/v0.90 metadata), so both cards are interchangeable and can freely be moved betwenn USB adapter and SD cardslot, and the system always remains bootable as long as at least one of the cards is operational.

@marcelgsteiger
Copy link

marcelgsteiger commented Dec 6, 2023

MD Raid10 on two Samsung Portable SSD T7 1TB

RPI5B w/4GB RAM
Array created with:
mdadm --create /dev/mdx --level=10 --metadata=1.2 --raid-devices=2 --layout=o2 --chunk=2M --bitmap=internal
samsung_t7_raid.txt

@k-koeberl

This comment was marked as off-topic.

@applecuckoo

This comment was marked as off-topic.

@k-koeberl

This comment was marked as off-topic.

@k-koeberl

This comment was marked as off-topic.

@ThomasKaiser

This comment was marked as off-topic.

@k-koeberl

This comment was marked as off-topic.

@theofficialgman

This comment was marked as off-topic.

@k-koeberl

This comment was marked as off-topic.

@geerlingguy
Copy link
Owner Author

Going to close up the previous discussion as it's a support-related question and not particularly related to the test data for this board.

@Headcrabed
Copy link

@theofficialgman We could improve sd card performance by enabling sd cqe support: https://forums.raspberrypi.com/viewtopic.php?t=367459

@geerlingguy
Copy link
Owner Author

After testing the NUMA Emulation patch (and finding the performance impact is... mixed at best?), I also want to do some testing with A2 cards (as mentioned in the link in the comment above), so I'll enable the queuing and run some tests on my SanDisk Extreme 256GB A2 card.

@geerlingguy
Copy link
Owner Author

geerlingguy commented Jul 7, 2024

Using the following benchmark with a 256 GB SanDisk Extreme A2 microSD card:

wget https://raw.githubusercontent.com/geerlingguy/pi-cluster/master/benchmarks/disk-benchmark.sh
chmod +x disk-benchmark.sh
sudo MOUNT_PATH=/ TEST_SIZE=1g ./disk-benchmark.sh

To enable command queueing, I placed dtparam=sd_cqe inside /boot/firmware/config.txt and rebooted.

Before

pi@pi5:~ $ dmesg | grep mmc0
[    2.546753] mmc0: SDHCI controller on 1000fff000.mmc [1000fff000.mmc] using ADMA 64-bit
[    2.652792] mmc0: new ultra high speed SDR104 SDXC card at address aaaa
[    2.659679] mmcblk0: mmc0:aaaa SN256 238 GiB
[    2.668229] mmcblk0: mmc0:aaaa SN256 238 GiB (quirks 0x00004000)

After

pi@pi5:~ $ dmesg | grep mmc0
[    2.506000] mmc0: CQHCI version 5.10
[    2.547683] mmc0: SDHCI controller on 1000fff000.mmc [1000fff000.mmc] using ADMA 64-bit
[    2.653376] mmc0: Command Queue Engine enabled, 31 tags
[    2.658629] mmc0: new ultra high speed SDR104 SDXC card at address aaaa
[    2.665452] mmcblk0: mmc0:aaaa SN256 238 GiB
[    2.706796] mmcblk0: mmc0:aaaa SN256 238 GiB (quirks 0x00004000)
Benchmark Before After
iozone 4K random read 9.54 MB/s 10.44 MB/s
iozone 4K random write 4.49 MB/s 13.87 MB/s
iozone 1M random read 90.10 MB/s 89.59 MB/s
iozone 1M random write 67.71 MB/s 66.51 MB/s
iozone 1M sequential read 89.69 MB/s 89.59 MB/s
iozone 1M sequential write 71.27 MB/s 69.30 MB/s

Also, running with agnostics (the Raspberry Pi Diagnostics SD Card test, sudo apt install agnostics):

Before

pi@pi5:~ $ sh /usr/share/agnostics/sdtest.sh
Run 1
prepare-file;0;0;64000;125
seq-write;0;0;64440;125
rand-4k-write;0;0;4553;1138
rand-4k-read;12038;3009;0;0
Sequential write speed 64440 KB/sec (target 10000) - PASS
Random write speed 1138 IOPS (target 500) - PASS
Random read speed 3009 IOPS (target 1500) - PASS

After

pi@pi5:~ $ sh /usr/share/agnostics/sdtest.sh
Run 1
prepare-file;0;0;62834;122
seq-write;0;0;64440;125
rand-4k-write;0;0;14911;3727
rand-4k-read;23003;5750;0;0
Sequential write speed 64440 KB/sec (target 10000) - PASS
Random write speed 3727 IOPS (target 500) - PASS
Random read speed 5750 IOPS (target 1500) - PASS

@platima
Copy link
Contributor

platima commented Jul 7, 2024 via email

@kmm0
Copy link

kmm0 commented Oct 26, 2024

Late to the game as I just saw that the Pi 5 supports Command Queueing.

I only had two A2 cards laying around.

Samsung Evo 128gb A2, doesn't seem to be supported, no mmc0: Command Queue Engine enabled, 31 tags after adding dtparam=sd_cqe and no difference in speed before and after.

SanDisk Extreme 64gb A2, this card worked

before

Run 1
prepare-file;0;0;30714;59
seq-write;0;0;74346;18
seq-read;40261;9;0;0
rand-4k-write;0;0;4175;1043
rand-4k-read;9829;2457;0;0
Sequential write speed 74346 KB/sec (target 10000) - PASS
Random write speed 1043 IOPS (target 500) - PASS
Random read speed 2457 IOPS (target 1500) - PASS

After
prepare-file;0;0;39942;78
seq-write;0;0;29942;7
seq-read;92077;22;0;0
rand-4k-write;0;0;11645;2911
rand-4k-read;24220;6055;0;0
Sequential write speed 29942 KB/sec (target 10000) - PASS
Random write speed 2911 IOPS (target 500) - PASS
Random read speed 6055 IOPS (target 1500) - PASS

@geerlingguy
Copy link
Owner Author

geerlingguy commented Dec 2, 2024

Raspberry Pi is working on some Pi SDRAM timing adjustments, which benefit both the Pi 5 and Pi 4.

I've been re-testing things like Geekbench (see linked blog post for those results), showing an 8-20% speedup depending on the specific benchmark. HPL also benefits, and here are tinymembench results:

tinymembench v0.4.10 (simple benchmark for memory throughput and latency)

==========================================================================
== Memory bandwidth tests                                               ==
==                                                                      ==
== Note 1: 1MB = 1000000 bytes                                          ==
== Note 2: Results for 'copy' tests show how many bytes can be          ==
==         copied per second (adding together read and writen           ==
==         bytes would have provided twice higher numbers)              ==
== Note 3: 2-pass copy means that we are using a small temporary buffer ==
==         to first fetch data into it, and only then write it to the   ==
==         destination (source -> L1 cache, L1 cache -> destination)    ==
== Note 4: If sample standard deviation exceeds 0.1%, it is shown in    ==
==         brackets                                                     ==
==========================================================================

 C copy backwards                                     :   6345.9 MB/s (0.4%)
 C copy backwards (32 byte blocks)                    :   6247.0 MB/s
 C copy backwards (64 byte blocks)                    :   6250.1 MB/s
 C copy                                               :   6138.4 MB/s (0.1%)
 C copy prefetched (32 bytes step)                    :   6144.8 MB/s
 C copy prefetched (64 bytes step)                    :   6146.3 MB/s
 C 2-pass copy                                        :   5727.1 MB/s (0.3%)
 C 2-pass copy prefetched (32 bytes step)             :   6178.2 MB/s (0.4%)
 C 2-pass copy prefetched (64 bytes step)             :   6186.5 MB/s (0.2%)
 C fill                                               :   9348.8 MB/s (0.5%)
 C fill (shuffle within 16 byte blocks)               :   9343.2 MB/s (0.4%)
 C fill (shuffle within 32 byte blocks)               :   9347.1 MB/s (0.3%)
 C fill (shuffle within 64 byte blocks)               :   9335.3 MB/s (0.2%)
 NEON 64x2 COPY                                       :   6068.6 MB/s
 NEON 64x2x4 COPY                                     :   6081.9 MB/s
 NEON 64x1x4_x2 COPY                                  :   6090.8 MB/s
 NEON 64x2 COPY prefetch x2                           :   5908.0 MB/s
 NEON 64x2x4 COPY prefetch x1                         :   6017.4 MB/s
 NEON 64x2 COPY prefetch x1                           :   5911.9 MB/s
 NEON 64x2x4 COPY prefetch x1                         :   6035.6 MB/s
 ---
 standard memcpy                                      :   6108.1 MB/s
 standard memset                                      :   9359.8 MB/s (0.5%)
 ---
 NEON LDP/STP copy                                    :   6098.9 MB/s
 NEON LDP/STP copy pldl2strm (32 bytes step)          :   6210.6 MB/s
 NEON LDP/STP copy pldl2strm (64 bytes step)          :   6214.1 MB/s
 NEON LDP/STP copy pldl1keep (32 bytes step)          :   6212.3 MB/s
 NEON LDP/STP copy pldl1keep (64 bytes step)          :   6215.8 MB/s
 NEON LD1/ST1 copy                                    :   6091.9 MB/s
 NEON STP fill                                        :   9344.3 MB/s (0.6%)
 NEON STNP fill                                       :   9335.9 MB/s
 ARM LDP/STP copy                                     :   6105.8 MB/s (0.2%)
 ARM STP fill                                         :   9323.0 MB/s (0.4%)
 ARM STNP fill                                        :   9317.7 MB/s (0.6%)

==========================================================================
== Memory latency test                                                  ==
==                                                                      ==
== Average time is measured for random memory accesses in the buffers   ==
== of different sizes. The larger is the buffer, the more significant   ==
== are relative contributions of TLB, L1/L2 cache misses and SDRAM      ==
== accesses. For extremely large buffer sizes we are expecting to see   ==
== page table walk with several requests to SDRAM for almost every      ==
== memory access (though 64MiB is not nearly large enough to experience ==
== this effect to its fullest).                                         ==
==                                                                      ==
== Note 1: All the numbers are representing extra time, which needs to  ==
==         be added to L1 cache latency. The cycle timings for L1 cache ==
==         latency can be usually found in the processor documentation. ==
== Note 2: Dual random read means that we are simultaneously performing ==
==         two independent memory accesses at a time. In the case if    ==
==         the memory subsystem can't handle multiple outstanding       ==
==         requests, dual random read has the same timings as two       ==
==         single reads performed one after another.                    ==
==========================================================================

block size : single random read / dual random read
      1024 :    0.0 ns          /     0.0 ns 
      2048 :    0.0 ns          /     0.0 ns 
      4096 :    0.0 ns          /     0.0 ns 
      8192 :    0.0 ns          /     0.0 ns 
     16384 :    0.0 ns          /     0.0 ns 
     32768 :    0.0 ns          /     0.0 ns 
     65536 :    0.0 ns          /     0.0 ns 
    131072 :    1.1 ns          /     1.5 ns 
    262144 :    1.6 ns          /     2.0 ns 
    524288 :    1.9 ns          /     2.2 ns 
   1048576 :    8.3 ns          /    11.2 ns 
   2097152 :   12.2 ns          /    14.5 ns 
   4194304 :   48.5 ns          /    72.1 ns 
   8388608 :   75.7 ns          /    99.8 ns 
  16777216 :   89.4 ns          /   108.8 ns 
  33554432 :   98.4 ns          /   113.6 ns 
  67108864 :  103.5 ns          /   116.7 ns 

Strange seeing it go from 13 GB/sec to 9 GB/sec for memset...

Edit: re-testing just editing sudo rpi-eeprom-config -e and commenting the SDRAM_BANKLOW setting (and rebooting), I got:

 standard memcpy                                      :   6139.6 MB/s
 standard memset                                      :  13196.5 MB/s (0.8%)

So something definitely slows down the writes at least for synthetic memory benchmarks. I think someone in the Pi forum thread mentioned the same:

Geekbench 6 is indeed faster than ever before. So is Google Octane v2. Passmark performs extremely good as well, except for the "Memory Write" sub test, which regresses from ~11500 MB/s to 8000 something. Sysbench memory read performance is also better than ever (very nice latency, both average and max), while the write test corroborates Passmark's result, showing a severe regression in write performance.

@geerlingguy
Copy link
Owner Author

With the new tweaks hitting apt, I'm going to re-run some of the benchmarks (especially HPL) to see how efficiency is improved for multicore tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests