Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kernel panic with ath10k #283

Open
dseomn opened this issue Dec 5, 2021 · 4 comments
Open

Kernel panic with ath10k #283

dseomn opened this issue Dec 5, 2021 · 4 comments

Comments

@dseomn
Copy link

dseomn commented Dec 5, 2021

PCEngines firmware version
v4.14.0.5

APU variant
apu2e4

OS and OS Version
Debian GNU/Linux 10 (buster)

Affected component(s), peripheral(s) or functionality
The entire system crashes.

Brief summary
The kernel panics occasionally. The watchdog still works though, so it gets rebooted and then works.

How reproducible
I think it happens a bit less than once per week.

How to reproduce
I haven't noticed any pattern of when it happens.

Expected behavior
The kernel shouldn't panic.

Actual behavior
The kernel panics. Here's a log from the serial console the most recent time it happened:

[86737.652272] wlp1s0.lan.l3: failed to remove key (0, a0:af:bd:e9:cc:c6) from hardware (-110)
[259531.353847] wlp5s0: failed to remove key (0, 14:c1:4e:b7:b5:d2) from hardware (-110)
[432016.150262] wlp1s0.guest.h2: failed to remove key (2, ff:ff:ff:ff:ff:ff) from hardware (-11)
[518728.502333] wlp1s0.lan.l3: failed to remove key (0, a0:af:bd:e9:cc:c6) from hardware (-110)
[1005496.119589] wlp1s0.lan.l3: failed to remove key (0, a0:af:bd:e9:cc:c6) from hardware (-11)
[1295278.270050] wlp1s0.lan.l3: failed to remove key (0, a0:af:bd:e9:cc:c6) from hardware (-11)
[1296309.161775] wlp1s0.lan.l3: failed to remove key (0, a0:af:bd:e9:cc:c6) from hardware (-110)
[1471067.906042] wlp1s0.lan.l3: failed to remove key (0, a0:af:bd:e9:cc:c6) from hardware (-11)
[1555503.955474] wlp5s0: failed to remove key (0, 14:c1:4e:b7:b5:d2) from hardware (-110)
[1589501.336457] wlp1s0: failed to remove key (0, 14:c1:4e:b7:b5:d2) from hardware (-11)
[1589502.759769] BUG: unable to handle kernel NULL pointer dereference at 0000000000000fc0
[1589502.767792] PGD 0 P4D 0 
[1589502.770522] Oops: 0002 [#1] SMP NOPTI
[1589502.774376] CPU: 3 PID: 19198 Comm: kworker/3:0 Tainted: G        W   E     4.19.0-18-amd64 #1 Debian 4.19.208-1
[1589502.784721] Hardware name: PC Engines apu2/apu2, BIOS v4.14.0.5 10/11/2021
[1589502.791874] Workqueue: events_freezable ieee80211_restart_work [mac80211]
[1589502.798856] RIP: 0010:_raw_spin_lock_bh+0x17/0x30
[1589502.803741] Code: 0f 1f 44 00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 0f 1f 44 00 00 65 81 05 10 e2 cd 4b 00 02 00 00 31 c0 ba 01 00 00 00 <f0> 0f b1 17 85 c0 75 01 c3 89 c6 e8 d9 33 99 ff 66 90 c3 66 0f 1f
[1589502.822666] RSP: 0018:ffff8eca2ab83cc0 EFLAGS: 00010246
[1589502.828074] RAX: 0000000000000000 RBX: 0000000000000fc0 RCX: 0000000000000000
[1589502.835389] RDX: 0000000000000001 RSI: ffff8eca27e1f4c8 RDI: 0000000000000fc0
[1589502.842702] RBP: ffff8eca27e1f3a8 R08: 0000000000000000 R09: 0000000000000000
[1589502.850019] R10: ffff8ec935c7e0f8 R11: 00000000000001ff R12: ffffffffc0b66030
[1589502.857332] R13: 0000000000000000 R14: ffff8eca26dc0ff8 R15: ffff8eca27e1e8c0
[1589502.864650] FS:  0000000000000000(0000) GS:ffff8eca2ab80000(0000) knlGS:0000000000000000
[1589502.872915] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1589502.878841] CR2: 0000000000000fc0 CR3: 00000001291ec000 CR4: 00000000000406e0
[1589502.886148] Call Trace:
[1589502.888780]  <IRQ>
[1589502.891026]  ath10k_wmi_tx_beacons_iter+0x24/0x140 [ath10k_core]
[1589502.897253]  ? ath10k_wmi_cmd_send_nowait+0xa0/0xa0 [ath10k_core]
[1589502.903622]  __iterate_interfaces+0x72/0x110 [mac80211]
[1589502.909136]  ieee80211_iterate_active_interfaces_atomic+0xd/0x10 [mac80211]
[1589502.916313]  ath10k_wmi_op_ep_tx_credits+0x20/0x40 [ath10k_core]
[1589502.922533]  ath10k_htc_process_trailer+0x1af/0x280 [ath10k_core]
[1589502.928853]  ath10k_htc_rx_completion_handler+0x97/0x130 [ath10k_core]
[1589502.935592]  ? ath10k_htc_process_trailer+0x280/0x280 [ath10k_core]
[1589502.942051]  ath10k_pci_process_rx_cb+0x174/0x1b0 [ath10k_pci]
[1589502.948112]  ath10k_ce_per_engine_service+0x93/0x100 [ath10k_core]
[1589502.954508]  ath10k_ce_per_engine_service_any+0x75/0xb0 [ath10k_core]
[1589502.961136]  ath10k_pci_napi_poll+0x44/0x100 [ath10k_pci]
[1589502.966724]  net_rx_action+0x149/0x3b0
[1589502.970666]  __do_softirq+0xde/0x2d8
[1589502.974439]  irq_exit+0xba/0xc0
[1589502.977772]  do_IRQ+0x7f/0xe0
[1589502.980933]  common_interrupt+0xf/0xf
[1589502.984778]  </IRQ>
[1589502.987094] RIP: 0010:ath10k_add_interface+0x79/0x1180 [ath10k_core]
[1589502.993628] Code: 89 e1 31 c0 48 c7 83 48 01 00 00 00 00 00 00 48 83 e7 f8 48 c7 83 a8 03 00 00 00 00 00 00 48 29 f9 81 c1 68 02 00 00 c1 e9 03 <f3> 48 ab 48 8b 83 30 01 00 00 48 85 c0 74 0c 48 8d 50 18 48 89 50
[1589503.012553] RSP: 0018:ffffb16bc0a5fd60 EFLAGS: 00000212 ORIG_RAX: ffffffffffffffda
[1589503.020301] RAX: 0000000000000000 RBX: ffff8eca27e1f3a8 RCX: 000000000000001f
[1589503.027616] RDX: ffff8eca237c0000 RSI: ffff8eca27e1f3a8 RDI: ffff8eca27e1f660
[1589503.034930] RBP: ffff8eca26dc0780 R08: 0000000000200000 R09: 0000000000000000
[1589503.042245] R10: 000000000000001f R11: ffff8eca2605a100 R12: ffff8eca27e1f4f0
[1589503.049558] R13: ffff8eca26dc2540 R14: ffff8eca24609f00 R15: ffff8eca26dc15a0
[1589503.056950]  drv_add_interface+0x4c/0x130 [mac80211]
[1589503.062208]  ieee80211_reconfig+0x226/0x1420 [mac80211]
[1589503.067622]  ? rcu_exp_wait_wake+0x240/0x240
[1589503.072080]  ? try_to_del_timer_sync+0x4d/0x80
[1589503.076780]  ieee80211_restart_work+0xbb/0xe0 [mac80211]
[1589503.082286]  process_one_work+0x1a7/0x3a0
[1589503.086484]  worker_thread+0x30/0x390
[1589503.090333]  ? create_worker+0x1a0/0x1a0
[1589503.094439]  kthread+0x112/0x130
[1589503.097856]  ? kthread_bind+0x30/0x30
[1589503.101702]  ret_from_fork+0x22/0x40
[1589503.105463] Modules linked in: cdc_ether usbnet r8152 mii cmac ctr ccm arc4 8021q garp mrp bridge stp llc wireguard(E) ip6_udp_tunnel udp_tunnel sr_mod cdrom nft_nat amd64_edac_mod ath10k_pci edac_mce_amd kvm_amd ath10k_core kvm evdev nft_fib_inet ath nft_fib_ipv4 nft_fib_ipv6 mac80211 irqbypass nft_fib uas crct10dif_pclmul crc32_pclmul leds_apu cfg80211 usb_storage ghash_clmulni_intel pcspkr sp5100_tco k10temp ccp sg fam15h_power rfkill rng_core nft_masq_ipv4 nft_masq button pcc_cpufreq nft_chain_nat_ipv4 nf_nat_ipv4 acpi_cpufreq nf_nat nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c nf_tables_set nf_tables nfnetlink ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic fscrypto ecb sd_mod crc32c_intel ahci libahci sdhci_pci libata cqhci sdhci xhci_pci xhci_hcd mmc_core ehci_pci
[1589503.176812]  ehci_hcd scsi_mod aesni_intel igb usbcore aes_x86_64 crypto_simd cryptd glue_helper i2c_piix4 i2c_algo_bit usb_common dca thermal
[1589503.189799] CR2: 0000000000000fc0
[1589503.193307] ---[ end trace 90a3696deb151c81 ]---
[1589503.198112] RIP: 0010:_raw_spin_lock_bh+0x17/0x30
[1589503.202998] Code: 0f 1f 44 00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 0f 1f 44 00 00 65 81 05 10 e2 cd 4b 00 02 00 00 31 c0 ba 01 00 00 00 <f0> 0f b1 17 85 c0 75 01 c3 89 c6 e8 d9 33 99 ff 66 90 c3 66 0f 1f
[1589503.221920] RSP: 0018:ffff8eca2ab83cc0 EFLAGS: 00010246
[1589503.227330] RAX: 0000000000000000 RBX: 0000000000000fc0 RCX: 0000000000000000
[1589503.234644] RDX: 0000000000000001 RSI: ffff8eca27e1f4c8 RDI: 0000000000000fc0
[1589503.241959] RBP: ffff8eca27e1f3a8 R08: 0000000000000000 R09: 0000000000000000
[1589503.249273] R10: ffff8ec935c7e0f8 R11: 00000000000001ff R12: ffffffffc0b66030
[1589503.256589] R13: 0000000000000000 R14: ffff8eca26dc0ff8 R15: ffff8eca27e1e8c0
[1589503.263905] FS:  0000000000000000(0000) GS:ffff8eca2ab80000(0000) knlGS:0000000000000000
[1589503.272170] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1589503.278100] CR2: 0000000000000fc0 CR3: 00000001291ec000 CR4: 00000000000406e0
[1589503.285415] Kernel panic - not syncing: Fatal exception in interrupt
[1589503.292021] Kernel Offset: 0x32c00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[1589503.302986] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
[1589503.310754] ------------[ cut here ]------------
[1589503.315555] sched: Unexpected reschedule of offline CPU#0!
[1589503.321235] WARNING: CPU: 3 PID: 19198 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x34/0x40
[1589503.330884] Modules linked in: cdc_ether usbnet r8152 mii cmac ctr ccm arc4 8021q garp mrp bridge stp llc wireguard(E) ip6_udp_tunnel udp_tunnel sr_mod cdrom nft_nat amd64_edac_mod ath10k_pci edac_mce_amd kvm_amd ath10k_core kvm evdev nft_fib_inet ath nft_fib_ipv4 nft_fib_ipv6 mac80211 irqbypass nft_fib uas crct10dif_pclmul crc32_pclmul leds_apu cfg80211 usb_storage ghash_clmulni_intel pcspkr sp5100_tco k10temp ccp sg fam15h_power rfkill rng_core nft_masq_ipv4 nft_masq button pcc_cpufreq nft_chain_nat_ipv4 nf_nat_ipv4 acpi_cpufreq nf_nat nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c nf_tables_set nf_tables nfnetlink ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic fscrypto ecb sd_mod crc32c_intel ahci libahci sdhci_pci libata cqhci sdhci xhci_pci xhci_hcd mmc_core ehci_pci
[1589503.402194]  ehci_hcd scsi_mod aesni_intel igb usbcore aes_x86_64 crypto_simd cryptd glue_helper i2c_piix4 i2c_algo_bit usb_common dca thermal
[1589503.415161] CPU: 3 PID: 19198 Comm: kworker/3:0 Tainted: G      D W   E     4.19.0-18-amd64 #1 Debian 4.19.208-1
[1589503.425500] Hardware name: PC Engines apu2/apu2, BIOS v4.14.0.5 10/11/2021
[1589503.432636] Workqueue: events_freezable ieee80211_restart_work [mac80211]
[1589503.439608] RIP: 0010:native_smp_send_reschedule+0x34/0x40
[1589503.445273] Code: 05 61 58 0e 01 73 15 48 8b 05 b8 93 ea 00 be fd 00 00 00 48 8b 40 30 e9 ea a7 9b 00 89 fe 48 c7 c7 80 d9 a4 b4 e8 95 c3 6b 00 <0f> 0b c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 8b 05 ed a3 5a
[1589503.464198] RSP: 0018:ffff8eca2ab83730 EFLAGS: 00010086
[1589503.469608] RAX: 0000000000000000 RBX: ffff8eca2aa22140 RCX: 0000000000000006
[1589503.476922] RDX: 0000000000000007 RSI: 0000000000000086 RDI: ffff8eca2ab966b0
[1589503.484236] RBP: ffff8eca24745a00 R08: 0000000000000518 R09: 000000000000000f
[1589503.491551] R10: 0000000000000000 R11: ffffffffb51fa6ef R12: ffff8eca2474615c
[1589503.498866] R13: ffff8eca2ab83780 R14: 0000000000000046 R15: 0000000000000000
[1589503.506181] FS:  0000000000000000(0000) GS:ffff8eca2ab80000(0000) knlGS:0000000000000000
[1589503.514447] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1589503.520375] CR2: 0000000000000fc0 CR3: 00000001291ec000 CR4: 00000000000406e0
[1589503.527682] Call Trace:
[1589503.530321]  <IRQ>
[1589503.532523]  check_preempt_curr+0x7a/0x90
[1589503.536720]  ttwu_do_wakeup+0x19/0x140
[1589503.540652]  try_to_wake_up+0x1d8/0x470
[1589503.544680]  autoremove_wake_function+0x11/0x50
[1589503.549398]  __wake_up_common+0x7a/0x190
[1589503.553506]  __wake_up_common_lock+0x7c/0xc0
[1589503.557962]  ? tick_sched_do_timer+0x60/0x60
[1589503.562414]  irq_work_run_list+0x4d/0x70
[1589503.566526]  update_process_times+0x4a/0x60
[1589503.570908]  tick_sched_handle+0x22/0x60
[1589503.575013]  tick_sched_timer+0x37/0x70
[1589503.579037]  __hrtimer_run_queues+0x100/0x280
[1589503.583580]  hrtimer_interrupt+0x100/0x210
[1589503.587859]  ? up+0x12/0x50
[1589503.590843]  smp_apic_timer_interrupt+0x6a/0x140
[1589503.595640]  apic_timer_interrupt+0xf/0x20
[1589503.599926] RIP: 0010:panic+0x204/0x24a
[1589503.603951] Code: eb a6 83 3d 9d cf ee 00 00 74 05 e8 46 8e 99 ff 48 c7 c6 00 1e 1f b5 48 c7 c7 20 74 a5 b4 e8 91 56 00 00 fb 66 0f 1f 44 00 00 <31> db e8 05 73 a3 ff 4c 39 eb 7c 1d 41 83 f4 01 48 8b 05 45 cf ee
[1589503.622876] RSP: 0018:ffff8eca2ab83ab0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[1589503.630624] RAX: 0000000000000046 RBX: 0000000000000000 RCX: 0000000000000006
[1589503.637931] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff8eca2ab966b0
[1589503.645244] RBP: ffff8eca2ab83b20 R08: 0000000000000516 R09: 000000000000000f
[1589503.652550] R10: 0000000000000000 R11: ffffffffb51fa6ef R12: 0000000000000000
[1589503.659856] R13: 0000000000000000 R14: 0000000000000009 R15: 0000000000000001
[1589503.667166]  ? apic_timer_interrupt+0xa/0x20
[1589503.671632]  oops_end.cold.9+0xc/0x18
[1589503.675481]  no_context+0x1be/0x380
[1589503.679156]  __do_page_fault+0xb2/0x4f0
[1589503.683179]  page_fault+0x1e/0x30
[1589503.686687] RIP: 0010:_raw_spin_lock_bh+0x17/0x30
[1589503.691570] Code: 0f 1f 44 00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 0f 1f 44 00 00 65 81 05 10 e2 cd 4b 00 02 00 00 31 c0 ba 01 00 00 00 <f0> 0f b1 17 85 c0 75 01 c3 89 c6 e8 d9 33 99 ff 66 90 c3 66 0f 1f
[1589503.710495] RSP: 0018:ffff8eca2ab83cc0 EFLAGS: 00010246
[1589503.715903] RAX: 0000000000000000 RBX: 0000000000000fc0 RCX: 0000000000000000
[1589503.723210] RDX: 0000000000000001 RSI: ffff8eca27e1f4c8 RDI: 0000000000000fc0
[1589503.730526] RBP: ffff8eca27e1f3a8 R08: 0000000000000000 R09: 0000000000000000
[1589503.737840] R10: ffff8ec935c7e0f8 R11: 00000000000001ff R12: ffffffffc0b66030
[1589503.745154] R13: 0000000000000000 R14: ffff8eca26dc0ff8 R15: ffff8eca27e1e8c0
[1589503.752512]  ? ath10k_wmi_cmd_send_nowait+0xa0/0xa0 [ath10k_core]
[1589503.758830]  ath10k_wmi_tx_beacons_iter+0x24/0x140 [ath10k_core]
[1589503.765059]  ? ath10k_wmi_cmd_send_nowait+0xa0/0xa0 [ath10k_core]
[1589503.771421]  __iterate_interfaces+0x72/0x110 [mac80211]
[1589503.776920]  ieee80211_iterate_active_interfaces_atomic+0xd/0x10 [mac80211]
[1589503.784098]  ath10k_wmi_op_ep_tx_credits+0x20/0x40 [ath10k_core]
[1589503.790319]  ath10k_htc_process_trailer+0x1af/0x280 [ath10k_core]
[1589503.796628]  ath10k_htc_rx_completion_handler+0x97/0x130 [ath10k_core]
[1589503.803371]  ? ath10k_htc_process_trailer+0x280/0x280 [ath10k_core]
[1589503.809827]  ath10k_pci_process_rx_cb+0x174/0x1b0 [ath10k_pci]
[1589503.815880]  ath10k_ce_per_engine_service+0x93/0x100 [ath10k_core]
[1589503.822275]  ath10k_ce_per_engine_service_any+0x75/0xb0 [ath10k_core]
[1589503.828902]  ath10k_pci_napi_poll+0x44/0x100 [ath10k_pci]
[1589503.834491]  net_rx_action+0x149/0x3b0
[1589503.838434]  __do_softirq+0xde/0x2d8
[1589503.842203]  irq_exit+0xba/0xc0
[1589503.845536]  do_IRQ+0x7f/0xe0
[1589503.848692]  common_interrupt+0xf/0xf
[1589503.852543]  </IRQ>
[1589503.854853] RIP: 0010:ath10k_add_interface+0x79/0x1180 [ath10k_core]
[1589503.861386] Code: 89 e1 31 c0 48 c7 83 48 01 00 00 00 00 00 00 48 83 e7 f8 48 c7 83 a8 03 00 00 00 00 00 00 48 29 f9 81 c1 68 02 00 00 c1 e9 03 <f3> 48 ab 48 8b 83 30 01 00 00 48 85 c0 74 0c 48 8d 50 18 48 89 50
[1589503.880310] RSP: 0018:ffffb16bc0a5fd60 EFLAGS: 00000212 ORIG_RAX: ffffffffffffffda
[1589503.888058] RAX: 0000000000000000 RBX: ffff8eca27e1f3a8 RCX: 000000000000001f
[1589503.895365] RDX: ffff8eca237c0000 RSI: ffff8eca27e1f3a8 RDI: ffff8eca27e1f660
[1589503.902680] RBP: ffff8eca26dc0780 R08: 0000000000200000 R09: 0000000000000000
[1589503.909996] R10: 000000000000001f R11: ffff8eca2605a100 R12: ffff8eca27e1f4f0
[1589503.917309] R13: ffff8eca26dc2540 R14: ffff8eca24609f00 R15: ffff8eca26dc15a0
[1589503.924710]  drv_add_interface+0x4c/0x130 [mac80211]
[1589503.929955]  ieee80211_reconfig+0x226/0x1420 [mac80211]
[1589503.935369]  ? rcu_exp_wait_wake+0x240/0x240
[1589503.939821]  ? try_to_del_timer_sync+0x4d/0x80
[1589503.944521]  ieee80211_restart_work+0xbb/0xe0 [mac80211]
[1589503.950015]  process_one_work+0x1a7/0x3a0
[1589503.954208]  worker_thread+0x30/0x390
[1589503.958056]  ? create_worker+0x1a0/0x1a0
[1589503.962163]  kthread+0x112/0x130
[1589503.965579]  ? kthread_bind+0x30/0x30
[1589503.969427]  ret_from_fork+0x22/0x40
[1589503.973188] ---[ end trace 90a3696deb151c82 ]---
[1589503.977994] ------------[ cut here ]------------
[1589503.982795] sched: Unexpected reschedule of offline CPU#1!
[1589503.988473] WARNING: CPU: 3 PID: 19198 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x34/0x40
[1589503.998125] Modules linked in: cdc_ether usbnet r8152 mii cmac ctr ccm arc4 8021q garp mrp bridge stp llc wireguard(E) ip6_udp_tunnel udp_tunnel sr_mod cdrom nft_nat amd64_edac_mod ath10k_pci edac_mce_amd kvm_amd ath10k_core kvm evdev nft_fib_inet ath nft_fib_ipv4 nft_fib_ipv6 mac80211 irqbypass nft_fib uas crct10dif_pclmul crc32_pclmul leds_apu cfg80211 usb_storage ghash_clmulni_intel pcspkr sp5100_tco k10temp ccp sg fam15h_power rfkill rng_core nft_masq_ipv4 nft_masq button pcc_cpufreq nft_chain_nat_ipv4 nf_nat_ipv4 acpi_cpufreq nf_nat nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c nf_tables_set nf_tables nfnetlink ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic fscrypto ecb sd_mod crc32c_intel ahci libahci sdhci_pci libata cqhci sdhci xhci_pci xhci_hcd mmc_core ehci_pci
[1589504.069417]  ehci_hcd scsi_mod aesni_intel igb usbcore aes_x86_64 crypto_simd cryptd glue_helper i2c_piix4 i2c_algo_bit usb_common dca thermal
[1589504.082380] CPU: 3 PID: 19198 Comm: kworker/3:0 Tainted: G      D W   E     4.19.0-18-amd64 #1 Debian 4.19.208-1
[1589504.092721] Hardware name: PC Engines apu2/apu2, BIOS v4.14.0.5 10/11/2021
[1589504.099844] Workqueue: events_freezable ieee80211_restart_work [mac80211]
[1589504.106819] RIP: 0010:native_smp_send_reschedule+0x34/0x40
[1589504.112485] Code: 05 61 58 0e 01 73 15 48 8b 05 b8 93 ea 00 be fd 00 00 00 48 8b 40 30 e9 ea a7 9b 00 89 fe 48 c7 c7 80 d9 a4 b4 e8 95 c3 6b 00 <0f> 0b c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 8b 05 ed a3 5a
[1589504.131410] RSP: 0018:ffff8eca2ab83698 EFLAGS: 00010082
[1589504.136819] RAX: 0000000000000000 RBX: ffff8eca2aaa2140 RCX: 0000000000000006
[1589504.144134] RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff8eca2ab966b0
[1589504.151449] RBP: ffff8eca229a8000 R08: 0000000000000579 R09: 000000000000000f
[1589504.158763] R10: 0000000000000000 R11: ffffffffb51fa6ef R12: ffff8eca229a875c
[1589504.166078] R13: ffff8eca2ab836e8 R14: 0000000000000046 R15: 0000000000000001
[1589504.173394] FS:  0000000000000000(0000) GS:ffff8eca2ab80000(0000) knlGS:0000000000000000
[1589504.181661] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1589504.187589] CR2: 0000000000000fc0 CR3: 00000001291ec000 CR4: 00000000000406e0
[1589504.194903] Call Trace:
[1589504.197538]  <IRQ>
[1589504.199744]  check_preempt_curr+0x7a/0x90
[1589504.203939]  ttwu_do_wakeup+0x19/0x140
[1589504.207874]  try_to_wake_up+0x1d8/0x470
[1589504.211897]  __wake_up_common+0x7a/0x190
[1589504.216010]  ep_poll_callback+0x143/0x2e0
[1589504.220209]  __wake_up_common+0x7a/0x190
[1589504.224325]  __wake_up_common_lock+0x7c/0xc0
[1589504.228779]  ? tick_sched_do_timer+0x60/0x60
[1589504.233232]  irq_work_run_list+0x4d/0x70
[1589504.237340]  update_process_times+0x4a/0x60
[1589504.241707]  tick_sched_handle+0x22/0x60
[1589504.245814]  tick_sched_timer+0x37/0x70
[1589504.249839]  __hrtimer_run_queues+0x100/0x280
[1589504.254382]  hrtimer_interrupt+0x100/0x210
[1589504.258668]  ? up+0x12/0x50
[1589504.261651]  smp_apic_timer_interrupt+0x6a/0x140
[1589504.266450]  apic_timer_interrupt+0xf/0x20
[1589504.270740] RIP: 0010:panic+0x204/0x24a
[1589504.274762] Code: eb a6 83 3d 9d cf ee 00 00 74 05 e8 46 8e 99 ff 48 c7 c6 00 1e 1f b5 48 c7 c7 20 74 a5 b4 e8 91 56 00 00 fb 66 0f 1f 44 00 00 <31> db e8 05 73 a3 ff 4c 39 eb 7c 1d 41 83 f4 01 48 8b 05 45 cf ee
[1589504.293687] RSP: 0018:ffff8eca2ab83ab0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[1589504.301433] RAX: 0000000000000046 RBX: 0000000000000000 RCX: 0000000000000006
[1589504.308740] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff8eca2ab966b0
[1589504.316056] RBP: ffff8eca2ab83b20 R08: 0000000000000516 R09: 000000000000000f
[1589504.323370] R10: 0000000000000000 R11: ffffffffb51fa6ef R12: 0000000000000000
[1589504.330684] R13: 0000000000000000 R14: 0000000000000009 R15: 0000000000000001
[1589504.338003]  ? apic_timer_interrupt+0xa/0x20
[1589504.342465]  oops_end.cold.9+0xc/0x18
[1589504.346314]  no_context+0x1be/0x380
[1589504.349989]  __do_page_fault+0xb2/0x4f0
[1589504.354014]  page_fault+0x1e/0x30
[1589504.357520] RIP: 0010:_raw_spin_lock_bh+0x17/0x30
[1589504.362406] Code: 0f 1f 44 00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 0f 1f 44 00 00 65 81 05 10 e2 cd 4b 00 02 00 00 31 c0 ba 01 00 00 00 <f0> 0f b1 17 85 c0 75 01 c3 89 c6 e8 d9 33 99 ff 66 90 c3 66 0f 1f
[1589504.381340] RSP: 0018:ffff8eca2ab83cc0 EFLAGS: 00010246
[1589504.386748] RAX: 0000000000000000 RBX: 0000000000000fc0 RCX: 0000000000000000
[1589504.394063] RDX: 0000000000000001 RSI: ffff8eca27e1f4c8 RDI: 0000000000000fc0
[1589504.401369] RBP: ffff8eca27e1f3a8 R08: 0000000000000000 R09: 0000000000000000
[1589504.408683] R10: ffff8ec935c7e0f8 R11: 00000000000001ff R12: ffffffffc0b66030
[1589504.415990] R13: 0000000000000000 R14: ffff8eca26dc0ff8 R15: ffff8eca27e1e8c0
[1589504.423343]  ? ath10k_wmi_cmd_send_nowait+0xa0/0xa0 [ath10k_core]
[1589504.429656]  ath10k_wmi_tx_beacons_iter+0x24/0x140 [ath10k_core]
[1589504.435884]  ? ath10k_wmi_cmd_send_nowait+0xa0/0xa0 [ath10k_core]
[1589504.442248]  __iterate_interfaces+0x72/0x110 [mac80211]
[1589504.447746]  ieee80211_iterate_active_interfaces_atomic+0xd/0x10 [mac80211]
[1589504.454924]  ath10k_wmi_op_ep_tx_credits+0x20/0x40 [ath10k_core]
[1589504.461145]  ath10k_htc_process_trailer+0x1af/0x280 [ath10k_core]
[1589504.467456]  ath10k_htc_rx_completion_handler+0x97/0x130 [ath10k_core]
[1589504.474196]  ? ath10k_htc_process_trailer+0x280/0x280 [ath10k_core]
[1589504.480652]  ath10k_pci_process_rx_cb+0x174/0x1b0 [ath10k_pci]
[1589504.486707]  ath10k_ce_per_engine_service+0x93/0x100 [ath10k_core]
[1589504.493109]  ath10k_ce_per_engine_service_any+0x75/0xb0 [ath10k_core]
[1589504.499739]  ath10k_pci_napi_poll+0x44/0x100 [ath10k_pci]
[1589504.505325]  net_rx_action+0x149/0x3b0
[1589504.509268]  __do_softirq+0xde/0x2d8
[1589504.513029]  irq_exit+0xba/0xc0
[1589504.516352]  do_IRQ+0x7f/0xe0
[1589504.519511]  common_interrupt+0xf/0xf
[1589504.523352]  </IRQ>
[1589504.525663] RIP: 0010:ath10k_add_interface+0x79/0x1180 [ath10k_core]
[1589504.532195] Code: 89 e1 31 c0 48 c7 83 48 01 00 00 00 00 00 00 48 83 e7 f8 48 c7 83 a8 03 00 00 00 00 00 00 48 29 f9 81 c1 68 02 00 00 c1 e9 03 <f3> 48 ab 48 8b 83 30 01 00 00 48 85 c0 74 0c 48 8d 50 18 48 89 50
[1589504.551120] RSP: 0018:ffffb16bc0a5fd60 EFLAGS: 00000212 ORIG_RAX: ffffffffffffffda
[1589504.558868] RAX: 0000000000000000 RBX: ffff8eca27e1f3a8 RCX: 000000000000001f
[1589504.566183] RDX: ffff8eca237c0000 RSI: ffff8eca27e1f3a8 RDI: ffff8eca27e1f660
[1589504.573498] RBP: ffff8eca26dc0780 R08: 0000000000200000 R09: 0000000000000000
[1589504.580814] R10: 000000000000001f R11: ffff8eca2605a100 R12: ffff8eca27e1f4f0
[1589504.588128] R13: ffff8eca26dc2540 R14: ffff8eca24609f00 R15: ffff8eca26dc15a0
[1589504.595528]  drv_add_interface+0x4c/0x130 [mac80211]
[1589504.600775]  ieee80211_reconfig+0x226/0x1420 [mac80211]
[1589504.606187]  ? rcu_exp_wait_wake+0x240/0x240
[1589504.610639]  ? try_to_del_timer_sync+0x4d/0x80
[1589504.615338]  ieee80211_restart_work+0xbb/0xe0 [mac80211]
[1589504.620834]  process_one_work+0x1a7/0x3a0
[1589504.625026]  worker_thread+0x30/0x390
[1589504.628874]  ? create_worker+0x1a0/0x1a0
[1589504.632983]  kthread+0x112/0x130
[1589504.636397]  ? kthread_bind+0x30/0x30
[1589504.640245]  ret_from_fork+0x22/0x40
[1589504.644006] ---[ end trace 90a3696deb151c83 ]---

Solutions you've tried
The watchdog timer turns this from a major issue into a tolerable one.
dseomn added a commit to dseomn/system-configs that referenced this issue Dec 5, 2021
@dseomn
Decrease watchdog timeout
b0d3a9d 
pcengines/apu2-documentation#283 is still
happening and this will hopefully make it less annoying.
@miczyg1
Copy link
Member

miczyg1 commented Dec 6, 2021

@dseomn as per our documentation of mPCIe modules: https://github.com/pcengines/apu2-documentation/blob/master/docs/mpcie_modules.md
you may want to:

  • try passing amd_iommu=off to the kernel cmdline OR
  • disable the IOMMU in the apu BIOS setup

Then see if it helps

@dseomn
Copy link
Author

dseomn commented Dec 7, 2021

Thanks for the link, I just added amd_iommu=off. I'll update here if it panics again.

@dseomn
Copy link
Author

dseomn commented Dec 17, 2021

It panicked again, this time with amd_iommu=off.

@metux
Copy link

metux commented Feb 21, 2023

Null pointer dereference: looks like a kernel bug in ath driver - please report to the corresponding kernel maintainer.
In your case - Debian kernel - the right place is the Debain Kernel Team.

Note that HW vendors usually neither have the capacity nor the knowledge to give SW or OS support.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@metux @dseomn @miczyg1