Home

Non-default settings

On Linux we set swappiness to 0 and disable transparent huge pages (THP). ext4 with no tuning is the most common file system.

We make no changes to Windows settings apart from disabling scheduled disk defragmentation (also called "Disk optimization").

We also enable parallel bucket and views compaction. Fragmentation threshold differs from default settings and depends on test case and workload.

Bucket compaction

0. Disable auto-compaction.
1. Create initial dataset, wait for persistence and TAP replication.
2. Mutate (update) all items in dataset. Wait for persistence and TAP replication.
3. Trigger bucket compaction, report total compaction throughput (MBytes/sec) measured as:

(data_disk_size_before_compaction - data_disk_size_after_compaction) / total_compaction_time

Index compaction

0. Disable auto-compaction, disable automatic index updates.
1. Create initial dataset, wait for persistence and TAP replication.
2. Define design documents.
3. Trigger index build, wait for indexing to finish.
4. Mutate all items in dataset. Wait for persistence and TAP replication.
5. Trigger index build, wait for indexing to finish.
6. Trigger index compaction, report total compaction throughput (MBytes/sec) measured as:

(views_disk_size_before_compaction - views_disk_size_after_compaction) / total_compaction_time

Initial indexing

0. Disable auto-compaction.
1. Create initial dataset, wait for persistence and TAP replication.
2. Compact bucket.
3. Define design documents.
4. Trigger index build, report total indexing time in minutes.

Incremental indexing

0. Disable auto-compaction, disable automatic index updates.
1. Create initial dataset, wait for persistence and TAP replication.
2. Compact bucket.
3. Define design documents.
4. Trigger index build, wait for indexing to finish.
5. Double dataset. Wait for persistence and TAP replication.
6. Compact bucket.
7. Trigger index build, report total indexing time in minutes.

BgFetcher disk reads/sec

0. Disable auto-compaction.
1. Create initial DGM dataset, wait for persistence and TAP replication.
2. Create working set via update or read operations, wait for persistence and TAP replication.
3. Compact bucket.
4. Start read-heavy (70/30) front-end workload with high cache miss ratio (~40%).
5. Run workload for predefined time (e.g., 1 hour), report average ep_bg_fetched per node.

Flusher drain rate

1. Create initial DGM dataset, wait for persistence and TAP replication.
2. Create working set via update or read operations, wait for persistence and TAP replication.
3. Compact bucket.
4. Start write-heavy (90/10) front-end workload.
5. Run workload for predefined time (e.g., 30 minutes), report average ep_diskqueue_drain per node.

OBSERVE latency

1. Create initial DGM dataset, wait for persistence and TAP replication.
2. Create working set via update or read operations, wait for persistence and TAP replication.
3. Compact bucket.
4. Start write-heavy (80/20) front-end workload.
5. Run workload for predefined time (e.g., 1 hour), report 95th percentile of OBSERVE latency.

SET/GET latency (in-memory)

1. Create initial non-DGM dataset, wait for persistence and TAP replication.
2. Compact bucket.
3. Start mixed (50/50) front-end workload.
4. Run workload for predefined time (e.g., 1 hour), report 95th percentile of SET and GET latency.

GET latency (disk fetches)

1. Create initial DGM dataset, wait for persistence and TAP replication.
2. Create working set via update or read operations, wait for persistence and TAP replication.
3. Compact bucket.
4. Start mixed (50/50) front-end workload with high cache miss ratio (~30%).
5. Run workload for predefined time (e.g., 1 hour), report 95th percentile of GET latency.

Warm up

1. Create initial dataset, wait for persistence and TAP replication.
2. Create working set via update or read operations, wait for persistence and TAP replication.
3. Compact bucket.
4. Start mixed (50/50) front-end workload.
5. Run workload for predefined time (e.g., 1 hour).
6. Restart all nodes, report master's ep_warmup_time in minutes.

TAP/URP throughput

0. Single node, single bucket.
1. Create initial dataset, wait for persistence and TAP replication.
2. Read all data via TAP or UPR protocol, report average throughput (items/sec) measured as:

total_items / total_time

Query latency

1. Create initial non-DGM or DGM dataset, wait for persistence and TAP replication.
2. Create working set via update or read operations, wait for persistence and TAP replication.
3. Compact bucket.
4. Define design documents.
5. Trigger index build, wait for indexing to finish.
6. Start read-heavy (80/20) front-end workload.
7. Run workload for predefined time (e.g., 1 hour), report 80th of query latency ("stale=update_after" or "stale=false").

Rebalance

1. Create initial dataset, wait for persistence and TAP replication.
2. Create working set via update or read operations, wait for persistence and TAP replication.
3. Compact bucket.
4. Start front-end workload.
5. Trigger cluster rebalance (IN/OUT/SWAP), report total rebalance time. Notice that there is a delay before and after rebalance.

Rebalance with Queries

1. Create initial dataset, wait for persistence and TAP replication.
2. Create working set via update or read operations, wait for persistence and TAP replication.
3. Compact bucket.
4. Define design documents.
5. Trigger index build, wait for indexing to finish.
6. Start front-end workload with view queries.
7. Trigger cluster rebalance (IN/OUT), report total rebalance time. Notice that there is a delay before and after rebalance.

Initial XDC replication

0. Disable auto-compaction.
1. Create initial dataset (source cluster), wait for persistence and TAP replication.
2. Compact bucket.
3. Initialize remote replication, report average replication rate.

XDCR

1. Create initial dataset (source cluster), wait for persistence and TAP replication.
2. Initialize remote replication, wait for initial replication, wait for persistence and TAP replication.
3. Compact bucket.
4. Run front-end workload, report maximum XDCR lag and maximum XDCR queue length.