You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I've been experimenting with enabling ZSTD compression and I beehive ZSTD should be offered as an option for those who might benefit from it.
Benefits:
DB size is 15% smaller (as of Jan 2024 fully synced ZSTD compressed DB is 112GB while uncompressed one ~134GB)
Reduced I/O which might help with slower SSDs
Depending on host, sync speed shouldn't increase much if at all (on i5-6200u with 8GB RAM, sync took 29H)
I am not asking to enable it by default (as I did in the patch below) but having a config option for people to utilize would be really nice.
diff --git a/src/Storage.cpp b/src/Storage.cpp
index 8ee8a52..1001eb2 100644
--- a/src/Storage.cpp
+++ b/src/Storage.cpp
@@ -1806,7 +1806,7 @@ void Storage::startup()
opts.error_if_exists = false;
opts.max_open_files = options->db.maxOpenFiles <= 0 ? -1 : options->db.maxOpenFiles; ///< this affects memory usage see: https://github.com/facebook/rocksdb/issues/4112
opts.keep_log_file_num = options->db.keepLogFileNum;
- opts.compression = rocksdb::CompressionType::kNoCompression; // for now we test without compression. TODO: characterize what is fastest and best..
+ opts.compression = rocksdb::CompressionType::kZSTD; // for now we test without compression. TODO: characterize what is fastest and best..
opts.use_fsync = options->db.useFsync; // the false default is perfectly safe, but Jt asked for this as an option, so here it is.
shistOpts = opts; // copy what we just did (will implicitly copy over the shared table_factory and write_buffer_manager)
@@ -1834,7 +1834,7 @@ void Storage::startup()
Debug() << "DB \"" << name << "\" mem: " << QString::number(mem / 1024. / 1024., 'f', 2) << " MiB";
opts.OptimizeLevelStyleCompaction(mem);
for (auto & comp : opts.compression_per_level)
- comp = rocksdb::CompressionType::kNoCompression; // paranoia -- enforce no compression since our data compresses so poorly
+ comp = rocksdb::CompressionType::kZSTD; // paranoia -- enforce no compression since our data compresses so poorly
memTotal += mem;
rocksdb::Status s;
// try and open database
The text was updated successfully, but these errors were encountered:
I've been experimenting with enabling ZSTD compression and I beehive ZSTD should be offered as an option for those who might benefit from it.
Benefits:
I am not asking to enable it by default (as I did in the patch below) but having a config option for people to utilize would be really nice.
The text was updated successfully, but these errors were encountered: