Home

Non-default settings

On Linux we set swappiness to 0 and disable transparent huge pages (THP). ext4 with no tuning is the most common file system.

We make no changes to Windows settings apart from disabling scheduled disk defragmentation (also called "Disk optimization").

We also enable parallel bucket and views compaction. Fragmentation threshold differs from default settings and depends on test case and workload.

Hacks

Changing internal settings:

    curl -XPOST -u user:password http://host:port/internalSettings -d 'maxBucketCount=30'

Passing alternative "swt":

    ERL_AFLAGS="+swt low" /etc/init.d/couchbase-server restart

Changing number of vbuckets per bucket:

    COUCHBASE_NUM_VBUCKETS=64 /etc/init.d/couchbase-server restart

Changing memcached settings in 3.0:

    curl -XPOST -u user:password http://host:port/diag/eval -d 'ns_config:set({node, node(), {memcached, extra_args}}, ["-t12", "-c5000"]).'

Bucket compaction

0. Disable auto-compaction.

1. Create initial dataset, wait for persistence and TAP replication.

2. Mutate (update) all items in dataset. Wait for persistence and TAP replication.

3. Trigger bucket compaction, report total compaction throughput (MBytes/sec) measured as:

    (data_disk_size_before_compaction - data_disk_size_after_compaction) / total_compaction_time
   

Index compaction

0. Disable auto-compaction, disable automatic index updates.

1. Create initial dataset, wait for persistence and TAP replaication.

2. Define design documents.

3. Trigger index build, wait for indexing to finish.

4. Mutate all items in dataset. Wait for persistence and TAP replication.

5. Trigger index build, wait for indexing to finish.

6. Trigger index compaction, report total compaction throughput (MBytes/sec) measured as:

    (views_disk_actual_size_before_compaction - views_disk_size_actual_after_compaction) / total_compaction_time
   

Initial indexing

0. Disable auto-compaction.
1. Create initial dataset, wait for persistence and TAP replication.
2. Compact bucket.
3. Define design documents.
4. Trigger index build, report total indexing time in minutes.

Incremental indexing

0. Disable auto-compaction, disable automatic index updates.
1. Create initial dataset, wait for persistence and TAP replication.
2. Compact bucket.
3. Define design documents.
4. Trigger index build, wait for indexing to finish.
5. Double dataset. Wait for persistence and TAP replication.
6. Compact bucket.
7. Trigger index build, report total indexing time in minutes.

BgFetcher disk reads/sec

0. Disable auto-compaction.
1. Create initial DGM dataset, wait for persistence and TAP replication.
2. Create working set via update or read operations, wait for persistence and TAP replication.
3. Compact bucket.
4. Start read-heavy (70/30) front-end workload with high cache miss ratio (~40%).
5. Run workload for predefined time (e.g., 1 hour), report average ep_bg_fetched per node.

Flusher drain rate

1. Create initial DGM dataset, wait for persistence and TAP replication.
2. Create working set via update or read operations, wait for persistence and TAP replication.
3. Compact bucket.
4. Start write-heavy (90/10) front-end workload.
5. Run workload for predefined time (e.g., 30 minutes), report average ep_diskqueue_drain per node.

OBSERVE latency

1. Create initial DGM dataset, wait for persistence and TAP replication.
2. Create working set via update or read operations, wait for persistence and TAP replication.
3. Compact bucket.
4. Start write-heavy (80/20) front-end workload.
5. Run workload for predefined time (e.g., 1 hour), report 95th percentile of OBSERVE latency.

SET/GET latency (in-memory)

1. Create initial non-DGM dataset, wait for persistence and TAP replication.
2. Compact bucket.
3. Start mixed (50/50) front-end workload.
4. Run workload for predefined time (e.g., 1 hour), report 95th percentile of SET and GET latency.

GET latency (disk fetches)

1. Create initial DGM dataset, wait for persistence and TAP replication.
2. Create working set via update or read operations, wait for persistence and TAP replication.
3. Compact bucket.
4. Start mixed (50/50) front-end workload with high cache miss ratio (~30%).
5. Run workload for predefined time (e.g., 1 hour), report 95th percentile of GET latency.

Warm up

1. Create initial dataset, wait for persistence and TAP replication.
2. Create working set via update or read operations, wait for persistence and TAP replication.
3. Compact bucket.
4. Start mixed (50/50) front-end workload.
5. Run workload for predefined time (e.g., 1 hour).
6. Restart all nodes, report master's ep_warmup_time in minutes.

TAP/URP throughput

0. Single node, single bucket.
1. Create initial dataset, wait for persistence and TAP replication.
2. Read all data via TAP or UPR protocol, report average throughput (items/sec) measured as:

total_items / total_time

Query latency

1. Create initial non-DGM or DGM dataset, wait for persistence and TAP replication.
2. Create working set via update or read operations, wait for persistence and TAP replication.
3. Compact bucket.
4. Define design documents.
5. Trigger index build, wait for indexing to finish.
6. Start read-heavy (80/20) front-end workload with bounded view queries.
7. Run workload for predefined time (e.g., 1 hour), report 80th of query latency (stale=update_after or stale=false).

Query throughput

1. Create initial non-DGM dataset, wait for persistence and TAP replication.
2. Create working set via update or read operations, wait for persistence and TAP replication.
3. Compact bucket.
4. Define design documents.
5. Trigger index build, wait for indexing to finish.
6. Start read-heavy (80/20) front-end workload with unbounded view queries.
7. Run workload for predefined time (e.g., 1 hour), report average couch_view_ops.

Initial XDC replication

0. Disable auto-compaction.
1. Create initial dataset (source cluster), wait for persistence and TAP replication.
2. Compact bucket.
3. Initialize remote replication, report average replication rate (items/sec).

Ongoing XDCR

1. Create initial dataset (source cluster), wait for persistence and TAP replication.
2. Initialize remote replication, wait for initial replication, wait for persistence and TAP replication.
3. Compact buckets.
4. Start mixed (50/50) front-end workload
5. Run workload for predefined time (e.g., 3 hours), report 90th percentile of XDCR lag.

The way lag is measured is based on following timeline:

t0 - client performs SET operation for key X on source cluster.

t1 - client receives response from source cluster.

t2 - client starts repeating GET requests for key X (with progressive polling interval) on destination.

t3 - client receives successful response from destination cluster.

The lag is calculated as t3 - t2

Rebalance

1. Create initial dataset, wait for persistence and TAP replication.
2. Create working set via update or read operations, wait for persistence and TAP replication.
3. Compact bucket.
4. Start mixed (50/50) front-end workload.
5. Wait for predefined time (e.g., 20 minutes).
6. Add/remove/swap nodes and trigger cluster rebalance.
7. Terminate front-end workload, report total rebalance time in minutes.

Rebalance with view queries

1. Create initial dataset, wait for persistence and TAP replication.
2. Create working set via update or read operations, wait for persistence and TAP replication.
3. Compact bucket.
4. Define design documents.
5. Trigger index build, wait for indexing to finish.
6. Start mixed (50/50) front-end workload with bounded view queries.
7. Wait for predefined time (e.g., 20 minutes).
8. Add/remove/swap nodes and trigger cluster rebalance.
9. Terminate front-end workload, report total rebalance time in minutes.

Rebalance with XDCR

1. Create initial dataset, wait for persistence and TAP replication.
2. Initialize remote replication, wait for initial replication, wait for persistence and TAP replication.
3. Create working set via update or read operations, wait for persistence and TAP replication.
4. Compact buckets.
5. Start mixed (50/50) front-end workload.
6. Wait for predefined time (e.g., 20 minutes).
7. Add/remove nodes on source on destination sides, trigger cluster rebalance.
8. Terminate front-end workload, report total rebalance time in minutes.