Merge branch 'enh-organize' which did more than that

* enh-organize: (81 commits)
  DOC: adjust README.md to the current state of affairs
  BF(WIN): try/except rmtree in a fixture
  BF(PY36): need to import dateutil.parser not just dateutil
  BF: tqdm was forgotten from dependecies
  ENH: upload also nwb's object_id if present
  ENH: doc/demos/basic-workflow1.sh - a basic demo of most of the commands
  RF: when ls -r is used, consider only .mwb files
  BF: special treatment for dandiset.yaml which goes into folder metadata
  BF(TST,WIN): organize - do not run symlink-relative on windows
  RF+ENH: register - guard against and act on a dandiset, --dandi-instance not --girder-instance
  RF: remove useless .dat from a named cache directory
  BF: minor - forgotten f""
  RF: Drop Python 3.5 (3.6 should still be ok) -- too many f""strings
  Revert "RF/BF: require python >= 3.7"
  BF(PY3.6,TST): just ask for stdout from subprocess.run
  BF: fixed typo (evil catch all!!!)
  BF(PY3.6): use dateutil to parse isodate in older pythons
  BF(TST,WIN): skip testing if requested symlink is not a symlink
  BF(WIN): use op.join
  RF+ENH: upload - only mtime of use, refresh now only updates older
  ...

DEVELOPMENT.md

@@ -0,0 +1,53 @@
+# DANDI Client Development
+
+## Development environment
+
+Assuming that you have `python3` (and virtualenv) installed, the fastest
+way to establish yourself a development environment (or a sample deployment),
+is via virtualenv:
+
+    git clone https://github.com/dandi/dandi-cli \
+        && cd dandi-cli \
+        &&  virtualenv --system-site-packages --python=python3 venvs/dev3 \
+        && source venvs/dev3/bin/activate \
+        && pip install -e .[test]
+
+### Install and activate precommit
+
+Install pre-commit dependency with `pip install pre-commit`
+
+In the source directory
+```
+pre-commit install
+```
+
+### dandiarchive instance
+
+[dandiarchive](https://github.com/dandi/dandiarchive) repository provides
+docker-compose recipe to establish local instance of the minimally provisioned
+dandiarchive (both with our web frontend, and girder backend).
+See [README.md:Docker](https://github.com/dandi/dandiarchive#docker) for the
+instructions.  In a new instance you would need to generate a new API key to be
+used by `dandi` client for upload etc.
+
+Relevant `dandi` client commands are aware of such an instance (such as `upload`)
+as `local-docker` (as opposed from `local` for a plain girder instance).  See note
+below on `DANDI_DEVEL` environment variable which would be needed to expose
+development command line options.
+
+## Environment variables
+
+- `DANDI_DEVEL` -- enables otherwise hidden command line options,
+  such as explicit specification of the girder instance, collection, etc.
+  All those options would otherwise be hidden from the user visible (`--help`)
+  interface, unless this env variable is set to non-empty value
+
+- `DANDI_API_KEY` -- avoids using keyrings, thus making it possible to
+  "temporarily" use another account etc.
+
+- `DANDI_LOG_LEVEL` -- set log level. By default `INFO`, should be an int (`10` - `DEBUG`).
+
+- `DANDI_CACHE_CLEAR` -- clear persistent cache before re-using.  E.g., for
+  metadata cache we use only released portion of `dandi.__version__` as a token.
+  If handling of metadata has changed while developing, set this env var to
+  have cache `clear()`ed before use.

README.md

@@ -32,15 +32,17 @@ Usage: dandi [OPTIONS] COMMAND [ARGS]...
 
 Options:
   --version
-  -l, --log-level INTEGER RANGE  Log level (TODO non-numeric values)
-  --pdb                          Fall into pdb if errors out
-  --help                         Show this message and exit.
+  -l, --log-level [DEBUG|INFO|WARNING|ERROR|CRITICAL]
+                                  Log level name  [default: INFO]
+  --pdb                           Fall into pdb if errors out
+  --help                          Show this message and exit.
 
 Commands:
   download  Download a file or entire folder from DANDI
-  ls        List .nwb files metadata
+  ls        List .nwb files and dandisets metadata.
   organize  (Re)organize files according to the metadata.
-  upload    Upload files to DANDI archive
+  register  Register a new dandiset in the DANDI archive
+  upload    Upload dandiset (files) to DANDI archive.
   validate  Validate files for NWB (and DANDI) compliance.
 ```
 
@@ -96,41 +98,40 @@ overall tentative target workflow will be:
    Result of the reorganization is a dandiset -- a dataset with consistent layout,
    and dataset level metadata.
 
-4. Rerun `dandi validate` on the `dadaset.yaml` (WiP) or the entire dandiset to assure
-   that everything is correct.
-5. Use `dandi register` (WiP) to register a new dataset ID on DANDI archive.
-6. Use `dandi upload` to upload your dandiset to the archive [staging area](TODO).
+4. Rerun `dandi validate` on the entire dandiset to assure that everything is
+   correct.
+5. Use `dandi register` to register a new dataset ID on DANDI archive.  If you
+   run it within a dandiset, its `dandiset.yaml` will be automatically updated
+   to contain new dandiset identifier.
+6. Use `dandi upload` to upload your dandiset to the archive
+   ["drafts" collection](https://gui.dandiarchive.org/#/collection/5e59bb0af19e820ab6ea6c62).
 
 If you change anything in your dandiset and decide to update its state in the
 archive, just use `dandi upload` again.
 
+You could also visit [doc/demos/basic-workflow1.sh]() for an example script
+which does all above actions (assuming no changes to files are necessary).
+
 
 ## Downloading dandiset from the archive
 
-Web has UI elements to download invidual files from the archiv.  For bulk
-download, or repeated re-syncing of the dandiset(s) from the archive, use
-`dandi download` command (WiP).
+`dandi download` can be used to download full dandisets or individual files or
+folders from the archive.
 
-## Development/contributing
+Using `--existing refresh` option available for
+`dandi upload` and `dandi download` it is possible to avoid transfer if files
+are identical locally and in the archive.
 
-Assuming that you have `python3` (and virtualenv) installed, the fastest
-way to establish yourself a development environment (or a sample deployment),
-is via virtualenv:
+**Warning:**  There is no version control tracking beyond checking correspondence
+of file size and modification time.  So in collaborative setting it is possible
+to "refresh" a file which was modified locally with a version from the archive,
+or vise versa.
 
-    git clone https://github.com/dandi/dandi-cli \
-        && cd dandi-cli \
-        &&  virtualenv --system-site-packages --python=python3 venvs/dev3 \
-        && source venvs/dev3/bin/activate \
-        && pip install -e .[test]
 
-### Install and activate precommit
+## Development/contributing
 
-Install pre-commit dependency with `pip install pre-commit`
+Please see [DEVELOPMENT.md]() file.
 
-In the source directory
-```
-pre-commit install
-```
 ## 3rd party components included
 
 ### dandi/support/generatorify.py

dandi/cli/cmd_download.py

@@ -0,0 +1,53 @@
+import datetime
+import os
+import sys
+import time
+
+import click
+from .command import devel_option, main, map_to_click_exceptions
+
+
+@main.command()
+@click.option(
+    "-o",
+    "--output-dir",
+    help="Directory where to download to (directory must exist). "
+    "Files will be downloaded with paths relative to that directory. ",
+    type=click.Path(exists=True, dir_okay=True, file_okay=False),
+    default=os.curdir,
+)
+@click.option(
+    "-e",
+    "--existing",
+    type=click.Choice(
+        ["error", "skip", "overwrite", "refresh"]
+    ),  # TODO: verify-reupload (to become default)
+    help="What to do if a file found existing locally. 'refresh': verify "
+    "that according to the size and mtime, it is the same file, if not - "
+    "download and overwrite.",
+    default="refresh",
+    show_default=True,
+)
+# Might be a cool feature, not unlike verifying a checksum, we verify that
+# downloaded file passes the validator, and if not -- alert
+# @click.option(
+#     "--validation",
+#     "validation",
+#     type=click.Choice(["require", "skip", "ignore"]),
+#     default="require",
+# )
+@devel_option(
+    "--develop-debug",
+    help="For development: do not use pyout callbacks, do not swallow exception",
+    default=False,
+    is_flag=True,
+)
+@click.argument("url", nargs=-1)
+@map_to_click_exceptions
+def download(url, output_dir, existing, develop_debug=False):
+    """Download a file or entire folder from DANDI"""
+    # First boring attempt at click commands being merely an interface to
+    # Python function
+    from ..download import download
+
+    return download(url, output_dir, existing, develop_debug)