Add configuration system

[yarikoptic]

ATM we only rely on some env vars defined in DEVELOPMENT.md

❯ grep DANDI_ DEVELOPMENT.md
`DANDI_DEVEL` environment variable, which is needed in order to expose the
- `DANDI_DEVEL` -- enables otherwise hidden command line options, such as
- `DANDI_API_KEY` -- avoids using keyrings, thus making it possible to
- `DANDI_LOG_LEVEL` -- set log level. By default `INFO`, should be an int (`10` - `DEBUG`).
- `DANDI_CACHE` -- clear persistent cache handling. Known values
- `DANDI_INSTANCEHOST` -- defaults to `localhost`. Point to host/IP which hosts
- `DANDI_TESTS_PERSIST_DOCKER_COMPOSE` -- When set, the tests will reuse the
- `DANDI_TESTS_PULL_DOCKER_COMPOSE` -- When set to an empty string or `0`, the
- `DANDI_TESTS_NO_VCR` — When set, the use of vcrpy to playback captured HTTP
- `DANDI_DEVEL_INSTRUMENT_REQUESTS_SUPERLEN` -- When set, the `upload()`
- `DANDI_DOWNLOAD_AGGRESSIVE_RETRY` -- When set, would make `download()` retry

and a set of options to individual commands which in principle could have their defaults overwritten "account wide", e.g. to modify default instance to use etc. Some additional considerations

• Should likely follow similar to git and datalad logic of going through ways to overload: system config, user config, local dandiset config, env variable, CLI option.
• to accommodate more of instances (e.g. as now for LINC), we might allow to define those, so configuration should allow (likely similar to how git does it in git config) to define new instances.

[aaronkanzer]

@yarikoptic Could you provide an example of configs you've used in datalad that you mention could be similar here?

My one mini-opinion here is that there is a lot of setup/institutional knowledge required to run these unit tests (I was stumbling down the hill quite a bit 😄 ) Perhaps I'd suggest clear defaults, or even (alluded in #1522) that certain values for just running test suites should also be included here. Will think about this more....

[yarikoptic]

re "howto" config -- it is not even "datalad" but git per se, e.g. have

[remote "{name}"]
  url = 
  ...

as for how overloads etc implemented within datalad -- checkout https://github.com/datalad/datalad/blob/maint/datalad/config.py#L256 .

there is a lot of setup/institutional knowledge required to run these unit tests

what specific unittests you have in mind? in dandi-cli we primarily (if not exclusively) run tests against docker-compose up'ed instance , so should not interact with other instances potentially be present in the universe.