index.json

[{"content":"I use atuin to keep a persistent and searchable bash history on my machine. Unfortunately, I was noticing periodic 5-10s of lag sometimes between entering a command, and seeing its response. A quick search turned up this Github issue, pointing to a ZFS bug involving sqlite (used by atuin) and ftruncate.\nA helpful comment alluded to a solution by storing the atuin sqlite db on tmpfs, and syncing that sqlite DB to ZFS using litestream. I had heard of litestream but had never used it, and wanted to duplicate the proposed fix.\nSetup Ensure you have atuin, litestream, and screen installed.\nThe standard atuin install suggests placing this in .bashrc (if using bash-preexec):\n[[ -f ~/.bash-preexec.sh ]] \u0026amp;\u0026amp; source ~/.bash-preexec.sh eval \u0026#34;$(atuin init bash)\u0026#34; I created an additional bash function that on the first bash session\nCreates a tmp dir Replicates the atuin db to tmpfs Setups up a background process running litestream to replicate the tmpfs sqlite db back to zfs The script function fix_atuin_zfs() { # Can remove after https://github.com/openzfs/zfs/issues/14290 gets fixed. # Also need to set ~/.config/atuin/config.toml:db_path = /tmp/$USER-atuin-db/history.db # (replace $USER with your username in config.toml) tmpfs_db_path=\u0026#34;/tmp/$USER-atuin-db\u0026#34; tmpfs_db_file=\u0026#34;$tmpfs_db_path/history.db\u0026#34; litestream_backup_path=\u0026#34;$HOME/.local/share/atuin/history-db-litestream\u0026#34; # Only run on first bash session for this user if mkdir \u0026#34;$tmpfs_db_path\u0026#34; 2\u0026gt;/dev/null; then # Need to copy over the DB to tmp dir + run litestream if [ -d \u0026#34;$litestream_backup_path\u0026#34; ]; then # We\u0026#39;ve already been using litestream, use it as the source of truth for history.db litestream restore -o \u0026#34;$tmpfs_db_file\u0026#34; \u0026#34;file://$litestream_backup_path\u0026#34; \u0026gt; /dev/null 2\u0026gt;\u0026amp;1 else # Migrate over the initial history.db from atuin to tmpfs cp ~/.local/share/atuin/history.db* \u0026#34;$tmpfs_db_path/\u0026#34; fi # Run litestream replication in the background screen -S litestream-atuin-$USER -dm litestream replicate \u0026#34;$tmpfs_db_file\u0026#34; \u0026#34;file://$litestream_backup_path\u0026#34; fi } fix_atuin_zfs [[ -f ~/.bash-preexec.sh ]] \u0026amp;\u0026amp; source ~/.bash-preexec.sh eval \u0026#34;$(atuin init bash)\u0026#34; A quick script, and hopefully not one that needs to last too long before ZFS gets a bugfix.\nMigrating Away Once ZFS is fixed, you can remove this from you .bashrc, kill the existing litestream process, and then do a final replication from the litestream backup to your normal atuin db:\nlitestream restore -o ~/.local/share/atuin/history.db ~/.local/share/atuin/history-db-litestream ","permalink":"https://branchmispredictor.github.io/post/fixing-atuin-lag-on-zfs/","summary":"I use atuin to keep a persistent and searchable bash history on my machine. Unfortunately, I was noticing periodic 5-10s of lag sometimes between entering a command, and seeing its response. A quick search turned up this Github issue, pointing to a ZFS bug involving sqlite (used by atuin) and ftruncate.\nA helpful comment alluded to a solution by storing the atuin sqlite db on tmpfs, and syncing that sqlite DB to ZFS using litestream.","title":"Fixing Atuin Lag on Zfs"},{"content":"This is an initcpio hook that loads your root pool zfs password from a usb stick, as an alternative to modifying the zfs initcpio hook as mentioned in https://wiki.archlinux.org/title/Install_Arch_Linux_on_ZFS#Loading_password_from_USB-Stick.\nThis hook can also be used in ZFSBootMenu, meaning the same code and same usb can unlock your root pool in ZFSBootMenu and Linux itself. This solution does not store your key in initramfs.\nGenerate the Password + Write to the USB Generate your zfs password and put it on a usb drive using DD, see the first two steps: https://wiki.archlinux.org/title/Install_Arch_Linux_on_ZFS#Loading_password_from_USB-Stick.\nCreate The Initcpio Hook Create /etc/initcpio/hooks/rpoolusb and edit usb and pool (and if necessary the size/location of your password on the usb in the dd command):\n#!/bin/ash run_hook() { echo \u0026#34;[rpoolusb]\u0026#34; usb=\u0026#34;/dev/disk/by-id/usb-Kingston_DataTraveler_3.0-0:0\u0026#34; pool=\u0026#34;rpool\u0026#34; mkdir /zfs-key-files mount -t tmpfs key_store /zfs-key-files if [ -e \u0026#34;$usb\u0026#34; ]; then echo \u0026#34;Loading key\u0026#34; dd if=\u0026#34;$usb\u0026#34; of=\u0026#34;/zfs-key-files/$pool\u0026#34; bs=32 count=1 else echo \u0026#34;Could not find usb!\u0026#34; sleep 5s fi } run_cleanuphook() { echo \u0026#34;[rpoolusb]\u0026#34; } Create /etc/initcpio/install/rpoolusb:\n#!/bin/bash build() { map add_binary \\ udevadm map add_file \\ /lib/udev/rules.d/60-zvol.rules \\ /lib/udev/rules.d/69-vdev.rules \\ /lib/udev/rules.d/90-zfs.rules \\ /lib/libgcc_s.so.1 add_runscript } help() { cat\u0026lt;\u0026lt;HELPEOF This hook allows you to load a zfs key from usb. HELPEOF } Set the Keylocation for Your ZFS Pool sudo zfs set -u keylocation=file:///zfs-key-files/rpool rpool Install the Hook for linux Add the hook to your mkinitcpio config at /etc/mkinitcpio.conf, example:\nHOOKS=(base udev autodetect modconf kms keyboard keymap consolefont block rpoolusb zfs filesystems) Regenerate initcpio: sudo mkinitcpio -P\nInstall the Hook for ZFSBootMenu Ensure you are using ZFSBootMenu with InitCPIO: true following https://wiki.archlinux.org/title/Install_Arch_Linux_on_ZFS#Generate_a_ZFSBootMenu_image.\nYou may have to add the following to /etc/zfsbootmenu/config.yaml:\nGlobal: ... InitCPIOHookDirs: - /etc/zfsbootmenu/initcpio - /etc/initcpio - /usr/lib/initcpio Add the rpoolusb hook to /etc/zfsbootmenu/mkinitcpio.conf, example:\nHOOKS=(base udev autodetect modconf block filesystems keyboard rpoolusb zfsbootmenu) Regenerate your zfsbootmenu with sudo generate-zbm\nReferences gist.github.com/Soulsuke/arch_on_zfs.txt Arch Wiki: Install Arch Linux on ZFS ","permalink":"https://branchmispredictor.github.io/post/encrypted-zfs-root-key-on-usb-using-arch/","summary":"This is an initcpio hook that loads your root pool zfs password from a usb stick, as an alternative to modifying the zfs initcpio hook as mentioned in https://wiki.archlinux.org/title/Install_Arch_Linux_on_ZFS#Loading_password_from_USB-Stick.\nThis hook can also be used in ZFSBootMenu, meaning the same code and same usb can unlock your root pool in ZFSBootMenu and Linux itself. This solution does not store your key in initramfs.\nGenerate the Password + Write to the USB Generate your zfs password and put it on a usb drive using DD, see the first two steps: https://wiki.","title":"Encrypted ZFS Root Key on USB using Arch"},{"content":"After updating my Arch Linux server to kernel version 5.14.14, all of the drives attached through a LSI SAS Card (LSISAS2116) disappeared.\nTL;DR: The fix was to add mpt3sas.max_queue_depth=10000 as a kernel param.\nInvestigation The lines in question via dmesg:\nmpt2sas_cm0: sense pool(0x(____ptrval____)) - dma(0x10d000000): depth(29868), element_size(96), pool_size (2800 kB) mpt2sas_cm0: sense pool(0x(____ptrval____))- dma(0x10d000000): depth(29868),element_size(96), pool_size(0 kB) mpt2sas_cm0: reply pool(0x(____ptrval____)) - dma(0x10d400000): depth(30191), frame_size(128), pool_size(3773 kB) mpt2sas_cm0: config page(0x(____ptrval____)) - dma(0x10cfc9000): size(512) mpt2sas_cm0: Allocated physical memory: size(66932 kB) mpt2sas_cm0: Current Controller Queue Depth(29865),Max Controller Queue Depth(32455) mpt2sas_cm0: Scatter Gather Elements per IO(128) random: fast init done mpt2sas_cm0: LSISAS2116: FWVersion(20.00.07.00), ChipRevision(0x02), BiosVersion(07.39.02.00) mpt2sas_cm0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ) scsi host10: Fusion MPT SAS Host blk-mq: reduced tag depth to 10240 mpt2sas_cm0: sending port enable !! mpt3sas 0000:02:00.0: can\u0026#39;t disable ASPM; OS doesn\u0026#39;t have ASPM control mpt2sas_cm1: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (12138696 kB) mpt2sas_cm0: hba_port entry: (____ptrval____), port: 255 is added to hba_port list mpt2sas_cm0: host_add: handle(0x0001), sas_addr(0x500062b20028efc0), phys(16) mpt2sas_cm0: expander_add: handle(0x0011), parent(0x0001), sas_addr(0x5001438022c73126), phys(37) expander-10:0: add: handle(0x0011), sas_addr(0x5001438022c73126) ------------[ cut here ]------------ WARNING: CPU: 6 PID: 128 at mm/page_alloc.c:5367 __alloc_pages+0x210/0x230 Modules linked in: crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd mpt3sas(+) \u0026gt; CPU: 6 PID: 128 Comm: kworker/u16:3 Not tainted 5.14.14-arch1-1 #1 3033d1ff40825c3916f6297baa2f1c356df9db26 Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./Z68XP-UD4, BIOS U1L 03/06/2013 Workqueue: fw_event_mpt2sas0 _firmware_event_work [mpt3sas] RIP: 0010:__alloc_pages+0x210/0x230 Code: 8d 54 24 08 44 89 ee 89 ef 89 6c 24 04 c6 44 24 28 00 48 89 5c 24 10 e8 0e f1 ff ff 49 89 c4 e9 f0 fe ff ff 40 80 e5 3\u0026gt; RSP: 0018:ffffb75980507938 EFLAGS: 00010246 RAX: 0000000000000000 RBX: 00000000000074a9 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 000000000000000b RDI: 0000000000040dc0 RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000140 R11: 0000000000000000 R12: 0000000000000000 R13: 000000000000000b R14: 0000000000577ec0 R15: 0000000000000140 FS: 0000000000000000(0000) GS:ffff9ecd93380000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f7e38cd1bf0 CR3: 00000002c9410002 CR4: 00000000000606e0 Call Trace: kmalloc_large_node+0x40/0xb0 __kmalloc_node+0x305/0x3b0 sbitmap_init_node+0x7f/0x1e0 scsi_alloc_sdev+0x2b2/0x350 scsi_probe_and_add_lun+0x9b9/0xdf0 __scsi_scan_target+0xa2/0x5a0 ? __pm_runtime_resume+0x58/0x80 scsi_scan_target+0xd7/0xf0 sas_rphy_add+0x126/0x170 [scsi_transport_sas d494b7ec27527bf0e737feb713a5a5b6ecd159d2] mpt3sas_transport_port_add+0x316/0x4a0 [mpt3sas c250154050dc3e3e76222987e48f6fd033eada1e] _scsih_add_device.constprop.0+0x441/0x550 [mpt3sas c250154050dc3e3e76222987e48f6fd033eada1e] _firmware_event_work+0xe17/0x1ed0 [mpt3sas c250154050dc3e3e76222987e48f6fd033eada1e] ? psi_task_switch+0xc2/0x1f0 ? __switch_to_asm+0x42/0x70 ? finish_task_switch.isra.0+0xaa/0x290 process_one_work+0x1e3/0x3b0 worker_thread+0x50/0x3b0 ? process_one_work+0x3b0/0x3b0 kthread+0x132/0x160 ? set_kthread_struct+0x40/0x40 ret_from_fork+0x22/0x30 ---[ end trace 79fad68877b70d38 ]--- scsi_alloc_sdev: Allocation failure during SCSI scanning, some SCSI devices might not be configured end_device-10:0:0: add: handle(0x0012), sas_addr(0x5001438022c7310c) scsi_alloc_sdev: Allocation failure during SCSI scanning, some SCSI devices might not be configured end_device-10:0:1: add: handle(0x0013), sas_addr(0x5001438022c7310d) scsi_alloc_sdev: Allocation failure during SCSI scanning, some SCSI devices might not be configured end_device-10:0:2: add: handle(0x0014), sas_addr(0x5001438022c7310e) scsi_alloc_sdev: Allocation failure during SCSI scanning, some SCSI devices might not be configured end_device-10:0:3: add: handle(0x0015), sas_addr(0x5001438022c7310f) scsi_alloc_sdev: Allocation failure during SCSI scanning, some SCSI devices might not be configured end_device-10:0:4: add: handle(0x0016), sas_addr(0x5001438022c73110) scsi_alloc_sdev: Allocation failure during SCSI scanning, some SCSI devices might not be configured end_device-10:0:5: add: handle(0x0017), sas_addr(0x5001438022c73111) scsi_alloc_sdev: Allocation failure during SCSI scanning, some SCSI devices might not be configured end_device-10:0:6: add: handle(0x0018), sas_addr(0x5001438022c73112) scsi_alloc_sdev: Allocation failure during SCSI scanning, some SCSI devices might not be configured end_device-10:0:7: add: handle(0x0019), sas_addr(0x5001438022c73113) scsi_alloc_sdev: Allocation failure during SCSI scanning, some SCSI devices might not be configured end_device-10:0:8: add: handle(0x001a), sas_addr(0x5001438022c73117) scsi_alloc_sdev: Allocation failure during SCSI scanning, some SCSI devices might not be configured end_device-10:0:9: add: handle(0x001b), sas_addr(0x5001438022c73125) mpt2sas_cm0: port enable: SUCCESS mpt2sas_cm1: CurrentHostPageSize is 0: Setting default host page size to 4k After some searching, I stumbled upon https://bugzilla.kernel.org/show_bug.cgi?id=209177. Notably it talks about a similar but not exactly the same error where mpt3sas fails to initialize due to a failed allocation.\nIn my own logs, I noticed the controlled queue depth was quite large just as in that bug report: Current Controller Queue Depth(29865),Max Controller Queue Depth(32455).\nThe Fix The fix was to add mpt3sas.max_queue_depth=10000 as a kernel param. As I am using GRUB, this meant editing GRUB_CMDLINE_LINUX in /etc/default/grub and then running sudo grub-mkconfig -o /boot/grub/grub.cfg.\nNow mpt3sas initializes correctly:\nmpt2sas_cm0: sense pool(0x(____ptrval____)) - dma(0x10aa00000): depth(10000), element_size(96), pool_size (937 kB) mpt2sas_cm0: sense pool(0x(____ptrval____))- dma(0x10aa00000): depth(10000),element_size(96), pool_size(0 kB) mpt2sas_cm0: reply pool(0x(____ptrval____)) - dma(0x109600000): depth(10323), frame_size(128), pool_size(1290 kB) mpt2sas_cm0: config page(0x(____ptrval____)) - dma(0x10a986000): size(512) mpt2sas_cm0: Allocated physical memory: size(22462 kB) mpt2sas_cm0: Current Controller Queue Depth(9997),Max Controller Queue Depth(32455) mpt2sas_cm0: Scatter Gather Elements per IO(128) random: fast init done mpt2sas_cm0: LSISAS2116: FWVersion(20.00.07.00), ChipRevision(0x02), BiosVersion(07.39.02.00) mpt2sas_cm0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ) scsi host10: Fusion MPT SAS Host mpt2sas_cm0: sending port enable !! mpt3sas 0000:02:00.0: can\u0026#39;t disable ASPM; OS doesn\u0026#39;t have ASPM control mpt2sas_cm1: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (12138696 kB) mpt2sas_cm0: hba_port entry: (____ptrval____), port: 255 is added to hba_port list mpt2sas_cm0: host_add: handle(0x0001), sas_addr(0x500062b20028efc0), phys(16) mpt2sas_cm0: expander_add: handle(0x0011), parent(0x0001), sas_addr(0x5001438022c73126), phys(37) expander-10:0: add: handle(0x0011), sas_addr(0x5001438022c73126) scsi 10:0:0:0: Direct-Access ATA WDC WD100EMAZ-00 0A83 PQ: 0 ANSI: 6 scsi 10:0:0:0: SATA: handle(0x0012), sas_addr(0x5001438022c7310c), phy(12), device_name(0x5000cca267c94f7f) scsi 10:0:0:0: enclosure logical id (0x5001438022c73125), slot(47) scsi 10:0:0:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y) scsi 10:0:0:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1) end_device-10:0:0: add: handle(0x0012), sas_addr(0x5001438022c7310c) ... mpt2sas_cm0: port enable: SUCCESS ","permalink":"https://branchmispredictor.github.io/post/fixing-lsi-sas-card-new-kernel/","summary":"After updating my Arch Linux server to kernel version 5.14.14, all of the drives attached through a LSI SAS Card (LSISAS2116) disappeared.\nTL;DR: The fix was to add mpt3sas.max_queue_depth=10000 as a kernel param.\nInvestigation The lines in question via dmesg:\nmpt2sas_cm0: sense pool(0x(____ptrval____)) - dma(0x10d000000): depth(29868), element_size(96), pool_size (2800 kB) mpt2sas_cm0: sense pool(0x(____ptrval____))- dma(0x10d000000): depth(29868),element_size(96), pool_size(0 kB) mpt2sas_cm0: reply pool(0x(____ptrval____)) - dma(0x10d400000): depth(30191), frame_size(128), pool_size(3773 kB) mpt2sas_cm0: config page(0x(____ptrval____)) - dma(0x10cfc9000): size(512) mpt2sas_cm0: Allocated physical memory: size(66932 kB) mpt2sas_cm0: Current Controller Queue Depth(29865),Max Controller Queue Depth(32455) mpt2sas_cm0: Scatter Gather Elements per IO(128) random: fast init done mpt2sas_cm0: LSISAS2116: FWVersion(20.","title":"Fixing LSI SAS Card After Kernel Upgrade"}]