Allow SM archive-dir to be specified by setting ARCHIVE_DIR.

Update the (commented) te2 definition.
Update README to explain the working of IMPORT_LOCAL and IMPORT_REMOTE.

README.md

@@ -1,13 +1,21 @@
 # nuodb-compose #
-Docker compose files for starting a nuodb database on the local host.
+Docker compose files for starting a nuodb database on the local host;
+  - and - optionally - initialising the database state from an existing NuoDB backup.
+
+## Use Cases ##
+* Create a local NuoDB database in docker on a developer's laptop;
+* Create a local copy of a running database for diagnostic purposes;
+* Create a local copy of an existing database for UAT or testing purposes;
+* create a simple multi-engine database on a single cloud node (VM);
 
 These docker compose files will create:
 * a new docker network specifically for this database;
 * separate AP (admin), TE, and SM containers - one for each NuoDB process;
+  - With changes to the file, a second TE can be supported;
 * a separate CD (collector) container for each engine container - to enable NuoDB Insights;
 * influxdb and grafana containers to host the NuoDB Insights dashboards.
 
-Note that the container names will have the `project` name embedded - which is the name of the directory (`nuodb`), or set with the `-p` option to `docker-compose`.
+Note that the container names will have the `project` name embedded - which is the name of the directory (`nuodb`) or set with the `-p` option to `docker-compose`.
 
 # Instructions #
 ## Creating the database ##
@@ -25,13 +33,12 @@ Note that the container names will have the `project` name embedded - which is t
      - on some platforms, setting `EXTERNAL_ADDRESS` to `127.0.0.1` also works;
    - if you want to import initial state from a database backup into the new database, set `IMPORT_LOCAL` and/or `IMPORT_REMOTE` (see `Notes` below for details of `IMPORT_LOCAL` and `IMPORT_REMOTE`);
       - the `import` operation is only performed when the archive dir is _empty_ - so the SM container can be stopped and restarted without being reinitialised each time.
-     - if you have set `IMPORT_LOCAL` or `IMPORT_REMOTE` _and_ it is a large file that takes multiple minutes to extract, you _will_ need to
-        set `STARTUP_TIMEOUT` to a larger value, to stop the startup from timing out before the IMPORT has completed.
+      - if you have set `IMPORT_LOCAL` or `IMPORT_REMOTE` _and_ it is a large archive that takes multiple minutes to import, you _will_ need to
+        set `STARTUP_TIMEOUT` to a value larger than the time taken to import, to stop the DB startup from timing out before the IMPORT has completed.
 
 4. create and start the nuodb database with `docker-compose up -d`.
 
 _*NOTE:*_ The `docker-compose` command may suggest to you to use `docker compose` instead.
-
 *Don't - it doesn't work.*
 
 
@@ -55,23 +62,37 @@ _*NOTE:*_ The `docker-compose` command may suggest to you to use `docker compose
 
 2. the initial state of the database can be imported using `IMPORT_LOCAL` and/or `IMPORT_REMOTE`, as follows:
 - set `IMPORT_LOCAL` to a path on the _local_ machine.
-  The SM container will mount this file as a volume and extract (`untar`) it into the
-  archive dir prior to starting the SM process;
-- set `IMPORT_REMOTE` to a URL of a remote file hosted on a server - typically accessed through `http(s)` or `(s)ftp`.
+  The SM container will mount this path as a volume and import it into the
+  archive dir prior to starting the SM process (presuming the archive is empty);
+
+  The path that `IMPORT_LOCAL` points to can be one of:
+  - a `tar.gzip` file of a `nuodb backup`;
+  - a directory cotaining a `nuodb backup`.
+
+  *Note*: that a `nuodb backup` can come in 1 of 2 formats:
+  - a nuodb `backup set` - which is the result of a `hotcopy --type full` backup;
+  - a nuodb `archive` - which is the result of a `hotcopy --type simple`, or just a copy of an SM archive and journal copied while the SM is _NOT_ running.
+
+  *Note*: a `backupset` can only be imported from a directory.
+
+- set `IMPORT_REMOTE` to a URL of a remote `file` hosted on a server - typically accessed through `http(s)` or `(s)ftp`.
   - Ex: `https://some.server.io/backup-4-3.tz`
   - Ex: `sftp://sftp.some.domain.com/backup-4-3.tz`
 
-  The SM container will download the file via the URL and extract it into
-  the archive dir prior to starting the SM process.
-- if you set _both_ `IMPORT_LOCAL` _and_ `IMPORT_REMOTE`, then `IMPORT_REMOTE` is treated as the remote copy, and `IMPORT_LOCAL` is treated as a locally cached copy - hence the behaviour is as follows:
-  - if `IMPORT_LOCAL` is a `file` _and_ is non-empty, then it is used directly, and `IMPORT_REMOTE` is ignored;
-  - if `IMPORT_LOCAL` is a file _and_ is `empty`, then `IMPORT_REMOTE` is downloaded into `IMPORT_LOCAL`, and the `import` is then performed from `IMPORT_LOCAL`;
-  - if `IMPORT_LOCAL` does _not_ exist - or is _not_ a file (ex a `directory`), then it is ignored, and the archive is imported directly from `IMPORT_REMOTE`.
-
-  _*NOTE:*_ To cause the initial download from `IMPORT_REMOTE` to be cached in `IMPORT_LOCAL`, `IMPORT_LOCAL` _must_ exist _and_ be empty.
+  *Note* that:
+  The SM container will download the remote file via the URL and extract it into the archive dir prior to starting the SM process.
+- if you set _both_ `IMPORT_LOCAL` _and_ `IMPORT_REMOTE`, then `IMPORT_REMOTE` is treated as the remote source, and `IMPORT_LOCAL` is treated as a locally cached copy - hence the behaviour is as follows:
+  - if `IMPORT_LOCAL` is a _non-empty_ `file` or `directory`, then it is used directly, and `IMPORT_REMOTE` is ignored.
+  - if `IMPORT_LOCAL` is an _empty_ `file` then `IMPORT_REMOTE` is downloaded into `IMPORT_LOCAL`, and the `import` is then performed by `extracting` from `IMPORT_LOCAL` into the `archive`;
+    - note this _only_ works for a `tar.gzip` file of an `archive` (see above).
+  - if `IMPORT_LOCAL` is an _empty_ `directory` then `IMPORT_REMOTE` is downloaded and extracted into `IMPORT_LOCAL`, and the `import` is then performed from `IMPORT_LOCAL` into the `archive`;
+    - note this works for _both_ forms of `nuodb backup` (see above);
+    - *Note* importing from a `directory` can be significantly _slower_ than imorting (`extracting directly`) from a `tar.gzip` file.
+
+  _Hence:*_ To cause the initial download from `IMPORT_REMOTE` to be cached in `IMPORT_LOCAL`, `IMPORT_LOCAL` _must_ exist _and_ be empty.
   To ensure this, you can do something like the following:
     - `$ rm -rf a/b/c`
-    - `$ touch a/b/c`
+    - `$ touch a/b/c` or `mkdir -p /a/b/c`
 
     Now you can set `IMPORT_REMOTE` as needed, and set `IMPORT_LOCAL` to `a/b/c`.
 
@@ -103,18 +124,28 @@ then check the logs of the `sm_1` container to see why it has failed to start.
 To check the logs of any container, run `docker logs <name-of-container>`
 Ex: `docker logs nuodb_sm_1`
 
-4. If you get an error of the form:
-```
-host:/a/a/c is not a valid, non-empty file - import failed...
-```
-then you have probably set `IMPORT_LOCAL` to point to a non-existent file, or to a directory.
-
-5. If you get an error in the form:
+4. If you get an error in the form:
 ```
 IMPORT_REMOTE is not a valid URL: ... - import aborted
 ```
 then you have not set `IMPORT_REMOTE` to a valid URL.
 A URL is in the form: <protocol>://<host>/<path>
+Ex: `sftp://myhost.com/my-archives/backup-xyz.tar.gzip`
 
-6. If an error causes only part of the database to be deployed, you can start the remaining containers - after fixing the error - by simply running `docker-compose up -d` again. The `up` command only starts those containers that are not currently running.
+5. If you get an error in the form:
+```
+This database has <n> archives with no running SM.
+No new SMs can start while the database is in this state.
+```
+then you have somehow restarted the database with existing archives but too few running SMs.
+This could happen if an import has somehow failed after the initial import started, and you restart with `IMPORT_X` set.
+This could also happen if an SM has shut down, and you try to restart it with `docker-compose up`, but have accidentally set `IMPORT_X`.
+(You cannot attempt to import the database state if there is existing state in some archive - even if the SM for that archive is not currently running.)
+Follow the instructions following the error message to resolve the problem(s), and then continue stating with:
+`... docker-compose up -d`
+
+6. If an error causes only part of the database to be deployed, you can start the remaining containers - after fixing the error - by simply running `... docker-compose up -d` again. The `up` command only starts those containers that are not currently running.
+When running `... docker-compose up` a subsequent time, you need to decide if you still need to set `IMPORT_X` variable(s):
+  - you _DON'T_ need to if the database state has already been successfully imported;
+  - you probably _DO_ need to if you had them set for the original `docker up` command, and the `import` has not yet succeeded.
 

nuodb/docker-compose.yaml

@@ -42,6 +42,8 @@ services:
         msg=$$(nuocmd check servers --timeout ${STARTUP_TIMEOUT:-60} --check-converged --check-active)
         [ $$? -ne 0 ] && echo "ERROR: SM timed out waiting for admin layer to be ready: $$msg" && exit 98
         echo "ARCHIVE_DIR=$$ARCHIVE_DIR"
+        mkdir -p $$ARCHIVE_DIR
+        [ $$? -ne 0 ] && echo "Error creating $$ARCHIVE_DIR: $$?" && exit 98
         nuocmd show archives --db-name $DB_NAME
         runningArchives=$$(nuocmd get archives --db-name "$DB_NAME" | grep 'state=RUNNING' | wc -l)
         notRunningArchives=$$(nuocmd get archives --db-name "DB_NAME" | grep -v "state=RUNNING" | wc -l)
@@ -111,12 +113,12 @@ services:
             nuodocker restore archive --origin-dir "$$ARCHIVE_DIR" --restore-dir "$$ARCHIVE_DIR" --db-name "$DB_NAME" --clean-metadata || exit 99
           fi
         fi
-        nuodocker start sm --db-name '$DB_NAME' --server-id '${PEER_ADDRESS:-nuoadmin1}' --dba-user '$DB_USER' --dba-password '$DB_PASSWORD' --options 'alt-address sm1' --database-options "$$DB_OPTIONS"
+        nuodocker start sm --db-name '$DB_NAME' --server-id '${PEER_ADDRESS:-nuoadmin1}' --archive-dir $$ARCHIVE_DIR --dba-user '$DB_USER' --dba-password '$DB_PASSWORD' --options 'alt-address sm1' --database-options "$$DB_OPTIONS"
 
   te1:
     image: $NUODB_IMAGE
     environment:
-      # push the current resolved value of the VAR
+      # push the current resolved value of the VARs
       NUOCMD_API_SERVER: ${PEER_ADDRESS:-nuoadmin1}:8888
       STARTUP_TIMEOUT: ${STARTUP_TIMEOUT:-90}
     hostname: te1
@@ -149,16 +151,36 @@ services:
   # te2:
   #   image: $NUODB_IMAGE
   #   environment:
-  #     # push the current resolved value of the VAR
-  #     NUOCMD_API_SERVER:
+  #     # push the current resolved value of the VARs
+  #     NUOCMD_API_SERVER: ${PEER_ADDRESS:-nuoadmin1}:8888
+  #     STARTUP_TIMEOUT: ${STARTUP_TIMEOUT:-90}
   #   hostname: te2
   #   scale: 0
   #   depends_on:
   #     - nuoadmin1
   #     - sm
   #   ports:
   #     - 48007:48006
-  #   command: [ "nuodocker", "start", "te", "--db-name", "$DB_NAME", "--server-id", "${PEER_ADDRESS:-nuoadmin1}", "--options", "alt-address ${EXTERNAL_ADDRESS:-te2}" ]
+  #
+  #   ## NOTE: use '$$' for any variable that is to be evaluated at runtime IN THE SHELL.
+  #   # Any variable with a single '$' is resolved by docker-compose and written literally into the command string.
+  #   command:
+  #     - "sh"
+  #     - "-c"
+  #     - |
+  #       sleepTime=0
+  #       sleepQuantum=10
+  #       while [ -z "$$(nuocmd get processes --db-name $DB_NAME | grep 'state=RUNNING')" -o $$? -ne 0 ] ; do
+  #         if [ $$sleepTime -ge $$STARTUP_TIMEOUT ]; then
+  #           echo "Timed out waiting for database startup ($$sleepTime sec)..."
+  #           exit 97
+  #         fi
+  #         echo "Waiting for confirmation that database $DB_NAME is started..."
+  #         sleep $$sleepQuantum
+  #         sleepTime=$$(( sleepTime + sleepQuantum ))
+  #         [ $$sleepQuantum -lt $$((STARTUP_TIMEOUT / 2)) ] && sleepQuantum=$$(( sleepQuantum + 30 ))
+  #       done
+  #       nuodocker start te --db-name '$DB_NAME' --server-id '${PEER_ADDRESS:-nuoadmin1}' --options 'alt-address ${EXTERNAL_ADDRESS:-te2}'
 
   influxdb:
     image: influxdb:1.8
@@ -219,8 +241,9 @@ services:
   #   depends_on:
   #     - te1
   #   environment:
-  #     PEER_ADDRESS: ${PEER_ADDRESS:-nuoadmin1}
+  #     # a VAR with no value pushes the currently resolved value of that VAR
   #     DB_NAME:
   #     DB_USER:
   #     DB_PASSWORD:
+  #     PEER_ADDRESS: ${PEER_ADDRESS:-nuoadmin1}
   #   command: ["/driver/startup.sh"]