Surprisingly slow write throughput

[dbbnrl]

I'm playing around with ReDB and Fjall. Based on what I've read, I'd expect Fjall to have slightly better write throughput, generally speaking. I'm getting dramatically slower results in Fjall and I've boiled my test code down to the two snippets below. The ReDB version completes (on my 16-core machine) in 27s. The Fjall version takes 700s.

I feel like I must be doing something fundamentally wrong or misunderstanding the intended use-cases of the crate. Any thoughts?

(The commented-out lines are various tuning parameters I tried, none of which changed things dramatically).

use glam::*;
use rayon::prelude::*;
use std::time::{Duration, Instant};
use redb::{Database, Error, ReadableTable, TableDefinition};

pub const TABLE: TableDefinition<[u8; 16], [u8; 256]> = TableDefinition::new("test_data");

fn main() {
    let db = Database::create("test.redb").unwrap();
    let start = Instant::now();
    (0..100).into_par_iter().for_each(|i| {
        println!("Item {i}");
        let mut write_txn = db.begin_write().unwrap();
        // write_txn.set_durability(redb::Durability::Eventual);
        let mut table = write_txn.open_table(TABLE).unwrap();
        for j in 0..100000 {
            let loc = uvec4(i, j, j, j);
            let v:[u8;256] = [100; 256];
            let k : &[u8; 16] = bytemuck::cast_ref(loc.as_ref());
            table.insert(k, v).unwrap();
        }
        drop(table);
        write_txn.commit().unwrap();
    });
    drop(db);
    let duration = start.elapsed();
    println!("load time={duration:?}");
}

use glam::*;
use rayon::prelude::*;
use std::time::{Duration, Instant};
use fjall::{Config, BlockCache, PartitionCreateOptions};

fn main() {
    let keyspace = Config::new("./test_db")
        // .block_cache(BlockCache::with_capacity_bytes(1024 * 1024 * 1024).into())
        // .max_write_buffer_size(1024 * 1024 * 1024)
        // .max_journaling_size(1024 * 1024 * 1024)
        .open().unwrap();
    let db = keyspace.open_partition("test", PartitionCreateOptions::default()).unwrap();
    let start = Instant::now();
    (0..100).into_par_iter().for_each(|i| {
        println!("Item {i}");
        for j in 0..100000 {
            let loc = uvec4(i, j, j, j);
            let v:[u8;256] = [100; 256];
            let k : &[u8; 16] = bytemuck::cast_ref(loc.as_ref());
            db.insert(&k, &v).unwrap();
        }
    });
    drop(db);
    drop(keyspace);
    let duration = start.elapsed();
    println!("load time={duration:?}");
}

[marvin-j97]

I feel like I must be doing something fundamentally wrong or misunderstanding the intended use-cases of the crate. Any thoughts?

First off, reproduction I got, just to confirm:

ReDB: 28s, ~2GB max memory, ~5GB disk space
Fjall: 686s, ~700 MB max memory, 206 MB disk space (your values are all the same so compression is absurdly effective here)

First off, ReDB is written to using a single write transaction per writer. From what I can gather (and looking at the memory usage), it buffers the entire transaction in memory, then writes it out to disk, all sorted. That's possible because it is limited to a single writer, so nothing tampers with the disk file. Because we're just bulk loading data, that's generally fine.

So there are two different things at play here:

(1) Fjall's default configuration is rather memory-conservative. The partition's memtable (= write buffer) is just 8 MB, which is not a lot.

The commented-out lines are various tuning parameters I tried, none of which changed things dramatically

• The journal size limit is not hit I think, so it doesn't need to change
• the block cache only affects read performance
• Your hunch with the write buffer size was pretty correct, however, that is just the global write buffer pool limit. Each partition has its own (as mentioned above) smaller write buffer. So using a single partition, the increased global limit (1G as you put it) does nothing, because it will get evicted after ~8MB anyway. Increasing it to 64MB (which is the default in RocksDB I think), yields:

Fjall Mod1: 32s, ~700 MB max memory, 203 MB disk space

So that almost already closes the gap, but we are using less temporary memory and disk space.

(2) Data is not written in order. Removing the rayon par_iter (for both ReDB and Fjall) yields (I reduced the memtable size to 32MB):

ReDB: 27s, 1 GB max memory, 4.8 GB disk space
Fjall Mod2: 24s, ~430 MB max memory, 203 MB disk space

When writing in order we should be able to decrease the memtable size without too much impact on performance. I've noticed, there's a bit of an undesired inefficiency when setting it back to 8~16 MB, but I think that is fixable. May be related to #63.

Result code:

use fjall::{Config, PartitionCreateOptions};

let keyspace = Config::new("./test_db")
    .open()
    .unwrap();
let db = keyspace
    .open_partition("test", PartitionCreateOptions::default())
    .unwrap();

// Not part of CreateOptions because it can be set during runtime
db.set_max_memtable_size(32 * 1_024 * 1_024);

let start = Instant::now();
(0..100).for_each(|i| {
    println!("Item {i}");
    for j in 0..100_000 {
        let loc = uvec4(i, j, j, j);
        let v: [u8; 256] = [100; 256];
        let k: &[u8; 16] = bytemuck::cast_ref(loc.as_ref());
        db.insert(&k, &v).unwrap();
    }
});
drop(db);
drop(keyspace);
let duration = start.elapsed();
println!("load time={duration:?}");

---

The smaller the transactions get, the more ReDB loses out (can be alleviated by setting the Durability to None, but that bloats disk space). Changing the loop to

(0..1_000).for_each(|i| {
    for j in 0..100 {
        let loc = uvec4(i, j, j, j);
        let v: [u8; 256] = [100; 256];
        let k: &[u8; 16] = bytemuck::cast_ref(loc.as_ref());
        db.insert(&k, &v).unwrap();
    }
});

yields:

ReDB: 4.2s
Fjall: 0.13s (or 4.0s when each batch is sync'd to disk individually)

[dbbnrl]

Aha! Thank you so much, I knew it must relate to write buffering in some way.

Thinking back on my process, I can tell you exactly why I missed this tuning knob: Because PartitionCreateOptions exists, I assumed it contained all of the things I could tune. I totally missed the existence of the set_* functions on PartitionHandle. I'm guessing you split them out this way because some settings are permanently baked into the Partition at creation, while others are run-time adjustable.

I would suggest doing one of two things. Either:

1. Add a brief documentation note. I.e. in PartitionCreateOptions, mention that there are additional run-time adjustable parameters documented in PartitionHandle.
2. Add compaction strategy and memtable size to the builder methods for PartitionCreateOptions, perhaps with a note that they can also be adjusted later.

At a higher level, a guide to proper tuning would be very helpful -- there are quite a few knobs that can be adjusted! This is more of a long-term suggestion; I recognize this is a very new library.

Thanks for creating Fjall and for your great response!

[marvin-j97]

Add compaction strategy and memtable size to the builder methods for PartitionCreateOptions, perhaps with a note that they can also be adjusted later.

This couldn't be added in V1 because there was no persisted configuration file for partitions.

In V2 this will change, the max_memtable_size and set_compaction_strategy are moved into the PartitionCreateOptions.

[marvin-j97]

In V2, max_memtable_size and compaction_strategy are both now part of the config object: https://docs.rs/fjall/latest/fjall/struct.PartitionCreateOptions.html#method.max_memtable_size