Configuring DVID

Before running DVID, you have to specify the various ports, storage backends, email notification address and mail server in case of crashes, and other run-time options.

DVID configuration is specified in a TOML file. It's an easily readable configuration format that maps unambiguously to a hash table. There is an example configuration file that comes with DVID source code.

A filled-in example might look like this:

[store]
    [store.mutable]
    engine = "basholeveldb"
    path = "/data/dbs/emdata1"

    [store.immutable]
    engine = "kvautobus"
    path = "http://tem-dvid.int.janelia.org:9000"

[server]
httpAddress = ":6000"
rpcAddress = ":6001"
webClient = "/path/to/dvid-console"

# Who to send email in case of panic
[email]
notify = ["[email protected]"]
username = "[email protected]"
password = ""
server = "10.1.50.90"
port = 25

[logging]
logfile = "/path/to/dvid.log"
max_log_size = 500 # MB
max_log_age = 360  # days

The only required configuration is the evolving [store] section. In particular, you must specify a mutable store via the [store.mutable] setting and can optionally specify immutable and metadata stores.

Immutable stores are for write-once data and can achieve higher performance for less money because the store doesn't have to worry about mutating data. For example, caching becomes easier and the store doesn't have to worry about distributed transactions. There is a natural match between mutable and immutable datastores and how DVID thinks of versioned data as a directed acyclic graph (DAG):

For connectomics research at Janelia (and the FlyEM research group in particular), the majority of data exists near the top of the DAG since most of our workflows involve ingesting very large image volumes and pre-generating segmentation for every voxel. So the bulk of our data can persist in immutable stores, allowing us to use smaller, faster storage solutions for the mutable portion of the DAG, namely the leaf nodes where manual editing tends to dominate.

The [server] section allows you to set the RPC and HTTP ports as well as the path to a web app that comes up when you point a web browser to the root HTTP address. The httpAddress option allows for binding to unique IP addresses. This is crucial for hosting on cloud services such as AWS.

The webClient property is a path to static website code for DVID administration, etc. We suggest you download the dvid console project and set the webClient path to its location. The stock dvid console lets you browse data repos and provides nice graphical representations of the data versions.

If you've configured the [email] section and there is a catchable panic, the DVID server will try to email an admin and continue on. Here's an example email you'll hopefully never receive but could be useful during debugging new features:

Subject: DVID panic report
Date: Friday, July 17, 2015 at 3:19:06 PM Eastern Daylight Time From: [email protected]
To: DVID admin

Panic detected on request c06u27.int.janelia.org/67bmYZ78V0-10888375: runtime error: index out of range
IP: 10.8.1.45:64608, URL: /api/node/ee7dc/annotations/keyrange/0
Stack trace:
goroutine 17099711 [running]: github.com/janelia-flyem/dvid/server.func·005()
/groups/flyem/home/katzw/work/go/src/github.com/janelia-flyem/dvid/server/web.go:399 +0xd6 github.com/janelia-flyem/dvid/datatype/keyvalue.(*Data).ServeHTTP(0xc2081dc648, 0xc2081a1760, 0x20, 0xc215091788, 0x7f2504847080, 0xc2082904b0, 0xc20bd41110)
/groups/flyem/home/katzw/work/go/src/github.com/janelia- flyem/dvid/datatype/keyvalue/keyvalue.go:430 +0x2143 github.com/janelia-flyem/dvid/server.func·009(0x7f2504847080, 0xc2082904b0, 0xc20bd41110)
/groups/flyem/home/katzw/work/go/src/github.com/janelia-flyem/dvid/server/web.go:536 +0x649 net/http.HandlerFunc.ServeHTTP(0xc2243fda00, 0x7f2504847080, 0xc2082904b0, 0xc20bd41110)
...
created by net/http.(*Server).Serve
/opt/buildem/src/golang-1.4.2/src/net/http/server.go:1751 +0x35e

Sincerely,
DVID at c06u27.int.janelia.org