Fulcrum process error starting [VirtualBox vboxsf fsync]

[zx9r]

First run. Ubuntu 20.04.3 LTS.

Here it is the output:

:~/Fulcrum-1.6.0-x86_64-linux$ ./Fulcrum fulcrum-config.conf 
[2022-02-01 20:45:53.335] Enabled JSON parser: simdjson
[2022-02-01 20:45:53.336] simdjson implementations:
[2022-02-01 20:45:53.336]     haswell: Intel/AMD AVX2  [not supported]
[2022-02-01 20:45:53.336]     westmere: Intel/AMD SSE4.2  [supported]
[2022-02-01 20:45:53.336]     fallback: Generic fallback implementation  [supported]
[2022-02-01 20:45:53.336] active implementation: westmere
[2022-02-01 20:45:53.338] jemalloc: version 5.2.1-0-gea6b3e9
[2022-02-01 20:45:53.338] Qt: version 5.15.2
[2022-02-01 20:45:53.338] rocksdb: version 6.14.6-ed43161
[2022-02-01 20:45:53.338] simdjson: version 0.6.0
[2022-02-01 20:45:53.338] ssl: OpenSSL 1.1.1f  31 Mar 2020
[2022-02-01 20:45:53.338] zmq: libzmq version: 4.3.3, cppzmq version: 4.7.1
[2022-02-01 20:45:53.338] Fulcrum 1.6.0 (Release 5e95c0f) - mar. feb. 1, 2022 20:45:53.338 CET - starting up ...
[2022-02-01 20:45:53.339] Max open files: 1048576 (increased from default: 1024)
[2022-02-01 20:45:53.339] Loading database ...
[2022-02-01 20:45:53.356] FATAL: Caught exception: Error opening meta database: IO error: While fsync: a directory: Invalid argument (path: /media/minodobitcoin/Fulcrum/meta)
[2022-02-01 20:45:53.356] Stopping Controller ... 
[2022-02-01 20:45:53.356] Closing storage ...
[2022-02-01 20:45:53.356] Shutdown complete

The meta directory is created fine with the following files:

CURRENT  IDENTITY  LOCK  LOG  MANIFEST-000001

and this is the content of the LOG file:

2022/02/01-20:45:53.341824 7fee5a546340 Git sha rocksdb_build_git_sha:ed4316166f67ec892603014634840d29f460f611
2022/02/01-20:45:53.341829 7fee5a546340 Compile date Jan 25 2022
2022/02/01-20:45:53.341880 7fee5a546340 DB SUMMARY
2022/02/01-20:45:53.341882 7fee5a546340 DB Session ID:  WGTZJQPJ1PCBOCL5U50X
2022/02/01-20:45:53.342360 7fee5a546340 SST files in /media/minodobitcoin/Fulcrum/meta dir, Total Num: 0, files: 
2022/02/01-20:45:53.342364 7fee5a546340 Write Ahead Log file in /media/minodobitcoin/Fulcrum/meta: 
2022/02/01-20:45:53.342367 7fee5a546340                         Options.error_if_exists: 0
2022/02/01-20:45:53.342369 7fee5a546340                       Options.create_if_missing: 1
2022/02/01-20:45:53.342371 7fee5a546340                         Options.paranoid_checks: 1
2022/02/01-20:45:53.342373 7fee5a546340                                     Options.env: 0x5595d9f3a6a0
2022/02/01-20:45:53.342375 7fee5a546340                                      Options.fs: Posix File System
2022/02/01-20:45:53.342377 7fee5a546340                                Options.info_log: 0x7fee5a10cf60
2022/02/01-20:45:53.342379 7fee5a546340                Options.max_file_opening_threads: 16
2022/02/01-20:45:53.342380 7fee5a546340                              Options.statistics: (nil)
2022/02/01-20:45:53.342382 7fee5a546340                               Options.use_fsync: 0
2022/02/01-20:45:53.342384 7fee5a546340                       Options.max_log_file_size: 0
2022/02/01-20:45:53.342386 7fee5a546340                  Options.max_manifest_file_size: 1073741824
2022/02/01-20:45:53.342388 7fee5a546340                   Options.log_file_time_to_roll: 0
2022/02/01-20:45:53.342390 7fee5a546340                       Options.keep_log_file_num: 5
2022/02/01-20:45:53.342392 7fee5a546340                    Options.recycle_log_file_num: 0
2022/02/01-20:45:53.342393 7fee5a546340                         Options.allow_fallocate: 1
2022/02/01-20:45:53.342395 7fee5a546340                        Options.allow_mmap_reads: 0
2022/02/01-20:45:53.342397 7fee5a546340                       Options.allow_mmap_writes: 0
2022/02/01-20:45:53.342399 7fee5a546340                        Options.use_direct_reads: 0
2022/02/01-20:45:53.342400 7fee5a546340                        Options.use_direct_io_for_flush_and_compaction: 0
2022/02/01-20:45:53.342402 7fee5a546340          Options.create_missing_column_families: 0
2022/02/01-20:45:53.342404 7fee5a546340                              Options.db_log_dir: 
2022/02/01-20:45:53.342406 7fee5a546340                                 Options.wal_dir: /media/minodobitcoin/Fulcrum/meta
2022/02/01-20:45:53.342408 7fee5a546340                Options.table_cache_numshardbits: 6
2022/02/01-20:45:53.342409 7fee5a546340                         Options.WAL_ttl_seconds: 0
2022/02/01-20:45:53.342411 7fee5a546340                       Options.WAL_size_limit_MB: 0
2022/02/01-20:45:53.342413 7fee5a546340                        Options.max_write_batch_group_size_bytes: 1048576
2022/02/01-20:45:53.342415 7fee5a546340             Options.manifest_preallocation_size: 4194304
2022/02/01-20:45:53.342417 7fee5a546340                     Options.is_fd_close_on_exec: 1
2022/02/01-20:45:53.342418 7fee5a546340                   Options.advise_random_on_open: 1
2022/02/01-20:45:53.342420 7fee5a546340                    Options.db_write_buffer_size: 0
2022/02/01-20:45:53.342422 7fee5a546340                    Options.write_buffer_manager: 0x7fee5a11e750
2022/02/01-20:45:53.342424 7fee5a546340         Options.access_hint_on_compaction_start: 1
2022/02/01-20:45:53.342425 7fee5a546340  Options.new_table_reader_for_compaction_inputs: 0
2022/02/01-20:45:53.342427 7fee5a546340           Options.random_access_max_buffer_size: 1048576
2022/02/01-20:45:53.342429 7fee5a546340                      Options.use_adaptive_mutex: 0
2022/02/01-20:45:53.342431 7fee5a546340                            Options.rate_limiter: (nil)
2022/02/01-20:45:53.342433 7fee5a546340     Options.sst_file_manager.rate_bytes_per_sec: 0
2022/02/01-20:45:53.342435 7fee5a546340                       Options.wal_recovery_mode: 2
2022/02/01-20:45:53.342436 7fee5a546340                  Options.enable_thread_tracking: 0
2022/02/01-20:45:53.342438 7fee5a546340                  Options.enable_pipelined_write: 0
2022/02/01-20:45:53.342440 7fee5a546340                  Options.unordered_write: 0
2022/02/01-20:45:53.342442 7fee5a546340         Options.allow_concurrent_memtable_write: 1
2022/02/01-20:45:53.342443 7fee5a546340      Options.enable_write_thread_adaptive_yield: 1
2022/02/01-20:45:53.342445 7fee5a546340             Options.write_thread_max_yield_usec: 100
2022/02/01-20:45:53.342447 7fee5a546340            Options.write_thread_slow_yield_usec: 3
2022/02/01-20:45:53.342449 7fee5a546340                               Options.row_cache: None
2022/02/01-20:45:53.342450 7fee5a546340                              Options.wal_filter: None
2022/02/01-20:45:53.342452 7fee5a546340             Options.avoid_flush_during_recovery: 0
2022/02/01-20:45:53.342454 7fee5a546340             Options.allow_ingest_behind: 0
2022/02/01-20:45:53.342456 7fee5a546340             Options.preserve_deletes: 0
2022/02/01-20:45:53.342458 7fee5a546340             Options.two_write_queues: 0
2022/02/01-20:45:53.342459 7fee5a546340             Options.manual_wal_flush: 0
2022/02/01-20:45:53.342461 7fee5a546340             Options.atomic_flush: 0
2022/02/01-20:45:53.342463 7fee5a546340             Options.avoid_unnecessary_blocking_io: 0
2022/02/01-20:45:53.342465 7fee5a546340                 Options.persist_stats_to_disk: 0
2022/02/01-20:45:53.342467 7fee5a546340                 Options.write_dbid_to_manifest: 0
2022/02/01-20:45:53.342468 7fee5a546340                 Options.log_readahead_size: 0
2022/02/01-20:45:53.342470 7fee5a546340                 Options.file_checksum_gen_factory: Unknown
2022/02/01-20:45:53.342472 7fee5a546340                 Options.best_efforts_recovery: 0
2022/02/01-20:45:53.342474 7fee5a546340                Options.max_bgerror_resume_count: 2147483647
2022/02/01-20:45:53.342475 7fee5a546340            Options.bgerror_resume_retry_interval: 1000000
2022/02/01-20:45:53.342477 7fee5a546340             Options.allow_data_in_errors: 0
2022/02/01-20:45:53.342479 7fee5a546340             Options.max_background_jobs: 1
2022/02/01-20:45:53.342481 7fee5a546340             Options.max_background_compactions: -1
2022/02/01-20:45:53.342483 7fee5a546340             Options.max_subcompactions: 1
2022/02/01-20:45:53.342484 7fee5a546340             Options.avoid_flush_during_shutdown: 0
2022/02/01-20:45:53.342486 7fee5a546340           Options.writable_file_max_buffer_size: 1048576
2022/02/01-20:45:53.342488 7fee5a546340             Options.delayed_write_rate : 16777216
2022/02/01-20:45:53.342490 7fee5a546340             Options.max_total_wal_size: 0
2022/02/01-20:45:53.342492 7fee5a546340             Options.delete_obsolete_files_period_micros: 21600000000
2022/02/01-20:45:53.342493 7fee5a546340                   Options.stats_dump_period_sec: 600
2022/02/01-20:45:53.342495 7fee5a546340                 Options.stats_persist_period_sec: 600
2022/02/01-20:45:53.342497 7fee5a546340                 Options.stats_history_buffer_size: 1048576
2022/02/01-20:45:53.342499 7fee5a546340                          Options.max_open_files: 40
2022/02/01-20:45:53.342501 7fee5a546340                          Options.bytes_per_sync: 0
2022/02/01-20:45:53.342502 7fee5a546340                      Options.wal_bytes_per_sync: 0
2022/02/01-20:45:53.342504 7fee5a546340                   Options.strict_bytes_per_sync: 0
2022/02/01-20:45:53.342506 7fee5a546340       Options.compaction_readahead_size: 0
2022/02/01-20:45:53.342508 7fee5a546340                  Options.max_background_flushes: -1
2022/02/01-20:45:53.342509 7fee5a546340 Compression algorithms supported:
2022/02/01-20:45:53.342511 7fee5a546340 	kZSTDNotFinalCompression supported: 0
2022/02/01-20:45:53.342513 7fee5a546340 	kZSTD supported: 0
2022/02/01-20:45:53.342515 7fee5a546340 	kXpressCompression supported: 0
2022/02/01-20:45:53.342517 7fee5a546340 	kLZ4HCCompression supported: 0
2022/02/01-20:45:53.342829 7fee5a546340 	kLZ4Compression supported: 0
2022/02/01-20:45:53.342834 7fee5a546340 	kBZip2Compression supported: 1
2022/02/01-20:45:53.342836 7fee5a546340 	kZlibCompression supported: 1
2022/02/01-20:45:53.342838 7fee5a546340 	kSnappyCompression supported: 0
2022/02/01-20:45:53.342841 7fee5a546340 Fast CRC32 supported: Not supported on x86
2022/02/01-20:45:53.344249 7fee5a546340 [/db_impl/db_impl_open.cc:290] Creating manifest 1 
2022/02/01-20:45:53.346493 7fee5a546340 [WARN] [/db_impl/db_impl_open.cc:1805] Persisting Option File error: OK
2022/02/01-20:45:53.346603 7fee5a546340 [/db_impl/db_impl.cc:447] Shutdown: canceling all background work
2022/02/01-20:45:53.346692 7fee5a546340 [/db_impl/db_impl.cc:623] Shutdown complete

[cculianu]

Let me ask you -- did you copy the database from someplace like from another install, or?

I have never seen that error. What filesystem is it on (ext4, etc)?

[zx9r]

It is a new installation from zero, never used Fulcrum before, so no pre existing database. Just an empty data dir.

I am using binary release from https://github.com/cculianu/Fulcrum/releases/download/v1.6.0/Fulcrum-1.6.0-x86_64-linux.tar.gz

filesystem is NTFS, it is a WD Blue SSD NVMe. Ubuntu is running in a VirtualBox, however I see that the mounted directory says type vboxsf

minodobitcoin on /media/*username*/minodobitcoin type vboxsf (rw,nodev,relatime,iocharset=utf8,uid=0,gid=999,dmode=0770,fmode=0770,tag=VBoxAutomounter)

Let me explain in more detail:

• Host Machine: Windows 10
  Here SSD is accesible as unit E: but I never use this disk from Windows

• Virtual Box: Ubuntu 20.04.3 LTS
  Here the E: drive is mounted in /media/*username*/minodobitcoin (I deleted username in the logs posted before)

In the Ubuntu VM I am running a bitcoind with datadir in /media/*username*/minodobitcoin/bitcoin
and also a lnd with datadir in /media/*username*/minodobitcoin/lnd

They are running from about a year ago, without any disk problem at all.

[EchterAgo]

vboxsf does not support directory fsync, which is required for rocksdb. You could give the whole volume to the VM, it would work with ntfs, but you probably need it shared. You could disable that in rocksdb (see facebook/rocksdb#2932) but it is not a good idea (see facebook/rocksdb#2929 (comment)). Running with the aptly named libeatmydata should also work.

[zx9r]

Thanks for the response. Unfortunately I cannot give the whole volume to the VM because the VM itself (.vmi) is installed in another directory of this volume.

[cculianu]

Then you need to create another filesystem.. or you can run it with libeatmydata...

[cculianu]

Hey @zx9r -- would you mind renaming the issue to something that includes "VirtualBox" and/or vboxsf and/or fsync error in the title? I feel like this little landmine might hit others with similar setups (as rare as the setup may be, it's not going to be 0% probability someone else is doing similar).

May be good for others to be able to quickly find this issue...

[zx9r]

I finally installed an Ubuntu 20.04 into a USB drive and mounted the SSD from there to get rid of VMs and run Linux natively.

Fulcrum fully synced in about 14h which is pretty awesome

[cculianu]

Awesome!! Just curious: Did you use the --fast-sync option or not?

[zx9r]

Yes, --fast-sync=6000 (6GB)

[cculianu]

Nice nice nice! :D

[cculianu]

I'm going to go ahead and close this issue but I will pin it first so others may see it in case they also are using VirtualBox's vboxsf.

[cadayton]

The workaround I used was to create a second harddrive for the VirtualBox VM rather than trying to use a directory on VirtualBox shared drive which only supports basic file copy operations. So far 35.7% sync'ed in about 45 minutes. The Bitcoin Core blockchain directory is located on the shared drive and it works just fine.