Fresh installation unstable, entire system crashes intermittently and xenstored/xencommons not working consistently

[Glegori]

Hello Mr. Lengyel

I have done a fresh install of Ubuntu and followed the installation instructions on https://drakvuf.com/ to try to get your software installed. When I get to the first reboot of Xen the xenstored.service driver is not functioning. I get a failure to bind message in systemctl status but it is hard to get output logs as something is causing hanging frequently(display on, no time update, not on network when I look in my switch). Sometimes the xenstored.service will come back only for the xencommons service to not work.

Initially I thought it may be a resource thing so I increased the memory value of Dom0 and pinned more VCPU but it did not alter the crashing issues in any significant way.

I tried to full fresh install again to see if it was something I added or modified later on such as doing some of the things listed in this link. And am getting the same issues.

I have installed your software to a functioning state on another device for a different project but do not have access to that device any longer.

Don't know what else to try please give me some advice.

[Glegori]

Note the crashing is only a Xen thing when I boot into the normal Ubuntu side it does not crash.

[tklengyel]

Unfortunately without logs/serial access figuring out what's going wrong is largely impossible. You could try updating the system bios but no guarantee that will fix anything.

[Glegori]

journalctlOutput.txt
I was able to nab the journalctl but I have not had any experience with digging through it, I was able to expand resources for Dom0 to temporarily get some relief from most of the crashing(still happens but less often)

and I was able to find the issue with xenstored I think it had to do with some of the drivers not matching up. Will me commenting out the differences and changing SSCI in the following image break anything?

[Glegori]

When I get a clean restart off and it is able to correctly shutdown xenstored and xencommons it is able to start itself back up but when it does not get a clean shutdown things that are required for startup are still marked as in use and not able to be used.

It is now just the issues with crashing. When I installed Xen with packet manager it works fine with no crashing but when clear the Xen installation and install via the DrakVUF submodules I get this crashing issue. Is this something you would be able to help me with or should I report it in a report on the Xen wiki.

I am just learning this lowlevel systems stuff as I dont have a great background in it but rather networking and highlevel coding so I apologize for any ineptitude.

[Glegori]

Hello,

I failed to add the log file from last comment. Is it something you are able to point me in the right direction on?

Mar 08 11:41:10 Malstudy kernel: page dumped because: page still charged to cgroup
Mar 08 11:41:10 Malstudy kernel: Modules linked in: rfcomm ccm cmac algif_hash algif_skcipher af_alg bridge stp llc snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel snd_sof_intel_hda_mlink soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_generic_allocation bnep snd_hda_codec_realtek soundwire_bus snd_hda_codec_generic intel_uncore_frequency_common ledtrig_audio snd_hda_codec_hdmi snd_soc_core snd_compress ac97_bus snd_pcm_dmaengine snd_hda_intel intel_rapl_msr crct10dif_pclmul snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec polyval_clmulni polyval_generic ghash_clmulni_intel cryptd sha256_ssse3 snd_hda_core sha1_ssse3 snd_hwdep snd_pcm snd_seq_midi binfmt_misc snd_seq_midi_event nouveau iwlmvm wmi_bmof joydev serio_raw snd_rawmidi input_leds nls_iso8859_1 mxm_wmi mac80211 drm_ttm_helper ttm libarc4 btusb snd_seq btrtl cmdlinepart btbcm drm_display_helper btintel spi_nor btmtk snd_seq_device
Mar 08 11:41:10 Malstudy kernel:  processor_thermal_device_pci cec mtd ee1004 iwlwifi snd_timer bluetooth rc_core mei_me processor_thermal_device ecdh_generic processor_thermal_rfim drm_kms_helper processor_thermal_mbox ecc processor_thermal_rapl mei i2c_algo_bit snd cfg80211 soundcore intel_rapl_common int3403_thermal int340x_thermal_zone int3400_thermal acpi_thermal_rel acpi_tad mac_hid sch_fq_codel xen_pciback xen_netback xen_blkback xen_gntalloc xen_gntdev xen_evtchn msr parport_pc ppdev lp parport drm efi_pstore xenfs xen_privcmd ip_tables x_tables autofs4 hid_generic usbhid hid dm_snapshot dm_bufio nvme nvme_core ahci nvme_common libahci intel_lpss_pci intel_lpss crc32_pclmul psmouse i2c_i801 r8169 i2c_smbus spi_intel_pci realtek spi_intel idma64 xhci_pci xhci_pci_renesas vmd video wmi pinctrl_alderlake
Mar 08 11:41:10 Malstudy kernel: CPU: 0 PID: 2559 Comm: cut Tainted: G    B              6.5.0-25-generic #25~22.04.1-Ubuntu
Mar 08 11:41:10 Malstudy kernel: Hardware name: Acer Nitro N50-640/Nitro N50-640, BIOS R01-A4 02/21/2022
Mar 08 11:41:10 Malstudy kernel: Call Trace:
Mar 08 11:41:10 Malstudy kernel:  <TASK>
Mar 08 11:41:10 Malstudy kernel:  dump_stack_lvl+0x48/0x70
Mar 08 11:41:10 Malstudy kernel:  dump_stack+0x10/0x20
Mar 08 11:41:10 Malstudy kernel:  bad_page+0x76/0x120
Mar 08 11:41:10 Malstudy kernel:  __rmqueue_pcplist+0x149/0x1d0
Mar 08 11:41:10 Malstudy kernel:  ? _raw_spin_unlock+0xe/0x40
Mar 08 11:41:10 Malstudy kernel:  rmqueue+0x37c/0xf10
Mar 08 11:41:10 Malstudy kernel:  get_page_from_freelist+0x10b/0x4c0
Mar 08 11:41:10 Malstudy kernel:  __alloc_pages+0x1e7/0x350
Mar 08 11:41:10 Malstudy kernel:  alloc_pages+0x91/0x1a0
Mar 08 11:41:10 Malstudy kernel:  __get_free_pages+0x11/0x50
Mar 08 11:41:10 Malstudy kernel:  tlb_remove_table+0x11e/0x2b0
Mar 08 11:41:10 Malstudy kernel:  ___pte_free_tlb+0x45/0x70
Mar 08 11:41:10 Malstudy kernel:  free_pud_range+0x234/0x4a0
Mar 08 11:41:10 Malstudy kernel:  free_p4d_range+0xa7/0x210
Mar 08 11:41:10 Malstudy kernel:  free_pgd_range+0x18a/0x1b0
Mar 08 11:41:10 Malstudy kernel:  shift_arg_pages+0x15e/0x200
Mar 08 11:41:10 Malstudy kernel:  setup_arg_pages+0x236/0x3d0
Mar 08 11:41:10 Malstudy kernel:  load_elf_binary+0x3cd/0xec0
Mar 08 11:41:10 Malstudy kernel:  ? ima_bprm_check+0xa2/0xe0
Mar 08 11:41:10 Malstudy kernel:  search_binary_handler+0xd7/0x260
Mar 08 11:41:10 Malstudy kernel:  exec_binprm+0x59/0x1c0
Mar 08 11:41:10 Malstudy kernel:  bprm_execve.part.0+0x198/0x250
Mar 08 11:41:10 Malstudy kernel:  bprm_execve+0x5d/0xa0
Mar 08 11:41:10 Malstudy kernel:  do_execveat_common.isra.0+0x1a9/0x260
Mar 08 11:41:10 Malstudy kernel:  __x64_sys_execve+0x37/0x60
Mar 08 11:41:10 Malstudy kernel:  do_syscall_64+0x58/0x90
Mar 08 11:41:10 Malstudy kernel:  ? exc_page_fault+0x94/0x1b0
Mar 08 11:41:10 Malstudy kernel:  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
Mar 08 11:41:10 Malstudy kernel: RIP: 0033:0x7676e5eeb08b
Mar 08 11:41:10 Malstudy kernel: Code: Unable to access opcode bytes at 0x7676e5eeb061.
Mar 08 11:41:10 Malstudy kernel: RSP: 002b:00007ffdc32bb778 EFLAGS: 00000246 ORIG_RAX: 000000000000003b
Mar 08 11:41:10 Malstudy kernel: RAX: ffffffffffffffda RBX: 0000653f616eb700 RCX: 00007676e5eeb08b
Mar 08 11:41:10 Malstudy kernel: RDX: 0000653f616eb848 RSI: 0000653f616eb700 RDI: 0000653f616eb9a0
Mar 08 11:41:10 Malstudy kernel: RBP: 0000653f5fe6c027 R08: 0000653f5fe6c1ff R09: 0000653f616ff855
Mar 08 11:41:10 Malstudy kernel: R10: 0000000000000002 R11: 0000000000000246 R12: 0000653f616eb848
Mar 08 11:41:10 Malstudy kernel: R13: 00007ffdc32bb868 R14: 0000653f616eb848 R15: 0000653f616eb9a0
Mar 08 11:41:10 Malstudy kernel:  </TASK>
Mar 08 11:41:10 Malstudy kernel: BUG: Bad page state in process cut  pfn:157de2
Mar 08 11:41:27 Malstudy sudo[3048]: malstudy : TTY=pts/0 ; PWD=/home/malstudy ; USER=root ; COMMAND=/usr/local/sbin/xl create windowsSnap.cfg
Mar 08 11:41:27 Malstudy sudo[3048]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)

CrashingOutput.txt

[tklengyel]

What version of Ubuntu are you using and what is your kernel version? You might want to give a shot to our pre-built deb packages instead of compiling your own, the CI automatically pushes the latest builds to https://github.com/tklengyel/drakvuf-builds

[Glegori]

I am running Ubuntu 22.04.4 with kernel 6.5.0-25-generic. Even with the precompiled version I am having the same issue.