From 9d109fa59a916225e1fdf4cbd6ae9744c6614e89 Mon Sep 17 00:00:00 2001 From: NikTJ777 Date: Tue, 14 Mar 2023 08:51:57 -0400 Subject: [PATCH] Ntj/refactor and allinone (#1) * Refactor scripts into separate script files. Mount script file or dir into each container. Incorporate Aaron's db-in-a-container. Add profiles to control which services (containers) are started. * Changes after first test runs * Rename various services and profiles. Remove profiles for the default "distributed" deployment. Move waitfor-nuoadmin code back into nuosm. Add log capture and better wait semantics to start-monolith. Add stop-nuodb script for internally triggered graceful shutdown of processes. * Improve arg parsing in stop-nuodb. * Add explicit docker-network config to all services. * Only enable nuocd-te2 in the insights profile. * Enable monolith to be scaled to multiple instances. Map ports to dynamic "ephemeral" prts on host. Remove hostname seting to allow dynamic hostname generation by docker compose. * Revert monolith to statically mapped ports. Set ENTRYPOINT and API_SERVER to localhst in monolith. Add instadb service that is db-in-a-container with dynamically-mapped ports, so multiple instances can run simultaneously. * Set LOGDIR default value in nuote. * Add --timeout option to delete server-processes command to force immediate cleanup of stranded processes after a service restart. Bump default NuoDB version to 5.0.1-2 Improve diagnostics and logging. * Reimplement all timeouts and gates with the nuodocker timeouts. Add new IMPORT_TIMEOUT variable. Add and improve logging and diagnostics. Refactor raftstate cleanup into new 'remove-zombie' script. * Inject remove-zombies into every container. * Add separate compose files for the different deployment styles. This allows commands such as: docker compose -f instadb.yaml up -d and: docker compose -f instadb.yaml down * Improve how errors are logged to console and file. Update the README with the latest options. * Update README file. * Fixed typos, formatting. Some rewording/clarifications. * Force TE internal port to be 48006, and force SM internal port to be 48007. Add clarifications to README. Fix typos in README. * More tidying * Fix a typo in the README. --------- Co-authored-by: acabrele Co-authored-by: Paul Chapman Co-authored-by: Paul Chapman --- README.md | 309 +++++++++++++++++++++++---------- nuodb/docker-compose.yaml | 328 ++++++++++++++++++----------------- nuodb/env-default | 24 ++- nuodb/instadb.yaml | 47 +++++ nuodb/monolith.yaml | 48 +++++ nuodb/scripts/import-archive | 81 +++++++++ nuodb/scripts/remove-zombie | 25 +++ nuodb/scripts/start-monolith | 63 +++++++ nuodb/scripts/start-nuoadmin | 13 ++ nuodb/scripts/start-nuosm | 69 ++++++++ nuodb/scripts/start-nuote | 37 ++++ nuodb/scripts/stop-nuodb | 30 ++++ 12 files changed, 814 insertions(+), 260 deletions(-) create mode 100644 nuodb/instadb.yaml create mode 100644 nuodb/monolith.yaml create mode 100755 nuodb/scripts/import-archive create mode 100644 nuodb/scripts/remove-zombie create mode 100755 nuodb/scripts/start-monolith create mode 100755 nuodb/scripts/start-nuoadmin create mode 100755 nuodb/scripts/start-nuosm create mode 100755 nuodb/scripts/start-nuote create mode 100755 nuodb/scripts/stop-nuodb diff --git a/README.md b/README.md index 3a2c8db..c785ca4 100644 --- a/README.md +++ b/README.md @@ -1,151 +1,270 @@ # nuodb-compose # -Docker compose files for starting a nuodb database on the local host; +Docker compose files for starting a NuoDB database on your local host; - and - optionally - initialising the database state from an existing NuoDB backup. +_NOTE:_ `docker compose` is intentionally a very simple tool - so the commands are more wordy than would be ideal, some desirable automation is just not possible, and some benign error messages are occasionally emitted. + ## Use Cases ## -* Create a local NuoDB database in docker on a developer's laptop; -* Create a local copy of a running database for diagnostic purposes; +* Create a local NuoDB database in Docker on a developer's laptop; +* Create a local copy of an existing database for diagnostic purposes; + * see the `IMPORT_LOCAL` and `IMPORT_REMOTE` options; * Create a local copy of an existing database for UAT or testing purposes; -* create a simple multi-engine database on a single cloud node (VM); + * see the `IMPORT_LOCAL` and `IMPORT_REMOTE` options; +* Create a local database for testing versions, SQL, ... +* Create a simple multi-engine database on a single cloud node (VM); +* Create multiple local databases for testing different software versions of either NuoDB or an application. + +## Configurations Supported ## +* `distributed` (default) + * This is a database comprising a _separate_ container for each database process: + * Separate AP (admin), TE, and SM containers - one for each NuoDB process; + * This is the same topology used when NuoDB is deployed in production and under Kubernetes, but may be overkill for a local machine - especially a laptop. +* `distributed + insights` + * This is a database comprising 3 separate containers for the database processes (as above), plus additional containers including `influxdb` and `grafana` to enable metrics publishing and display. + * This is currently the _only_ configuration that supports NuoDB `Insights` monitoring. +* `monolith` + * This is a database in a single container (monolith). All 3 NuoDB processes are running inside the same container. + * This is _not_ how NuoDB is deployed in production, but is an easily managed, resource optimised, option on a local machine. +* `instadb` + * This is a database in a single container (as per `monolith`) - but with _dynamic port mapping_. + * This allows multiple `instadb` databases to run on the same host simultaneously. + * The downside is that because of the dynamically-mapped ports, the mapped public port cannot be predicted, and so external connections (such as non-containerized applications running on the same computer) will need to use the `direct=true` connection property; + * To ensure container name uniqueness, each `instadb` instance must be running in a different `project` from any other `insadb` instance. + +These Docker compose files will create: +* A new Docker network specifically for the project; +* A database in one of 4 possible configurations (see above). -These docker compose files will create: -* a new docker network specifically for this database; -* separate AP (admin), TE, and SM containers - one for each NuoDB process; - - With changes to the file, a second TE can be supported; -* a separate CD (collector) container for each engine container - to enable NuoDB Insights; -* influxdb and grafana containers to host the NuoDB Insights dashboards. +### scaling a database ## +* A second TE can be added to a `distributed` database + * See the `scale-te2` profile. -Note that the container names will have the `project` name embedded - which is the name of the directory (`nuodb`) or set with the `-p` option to `docker-compose`. +Note that all container names will have the `project` name embedded - which is the name of the directory (`nuodb`), but you can override the project name using the `-p` option to `docker compose`. +This default is fine for all configurations except `instadb` databases (you can use the default for your first `instadb` database, but all subsequent databases must be named explicity using `-p` to ensure unique contaienr names). # Instructions # -## Creating the database ## -0. clone the repo. +## Getting Started ## +0. Clone this repo. + +1. `cd` to the `nuodb` directory. + +2. Copy the `env-default` file to `.env` (this step is _NOT_ optional). + +3. Edit the `.env` file: + - `ENGINE_MEM` + - Sets the memory cache size for each TE and SM. + - `SQL_ENGINE`: + - if you want to use a specific SQL engine, such as `scalar` (`vector` is the default), you will also need an image that supports that engine; + - `EXTERNAL_ADDRESS`: + - if you want to access the database from outside the `docker network` - for example from an app running direcly on the local host - then set `EXTERNAL_ADDRESS`; + - either in the `.env` file, _or_ by setting `EXTERNAL_ADDRESS` on the `docker compose up` command-line; + - set to the address of the local host machine (Ex `192.168.0.123`); + - on some platforms, setting `EXTERNAL_ADDRESS` to `127.0.0.1` also works; + - `IMPORT_LOCAL`, `IMPORT_REMOTE`, `IMPORT_TIMEOUT`, `IMPORT_AUTH`, `IMPORT_LEVEL` + - if you want to import initial state from a database backup into the new database, set `IMPORT_LOCAL` and/or `IMPORT_REMOTE` (see `Notes` below for details of `IMPORT_LOCAL` and `IMPORT_REMOTE`); + - the `import` operation is only performed when the archive dir is _empty_ - so the SM container can be stopped and restarted without being reinitialised each time. + - if you have set `IMPORT_LOCAL` or `IMPORT_REMOTE` _and_ it is a large archive that takes multiple minutes to import, you _will_ need to + set `IMPORT_TIMEOUT` to a value larger than the time taken to import - in order to stop the DB startup from timing out before the IMPORT has completed. + + +_*NOTE:*_ In earlier versions of `docker`, the `docker-compose` command was the only form that worked with nuodb-compose. +However, with newer versions of `docker` both `docker-compose` _and_ `docker compose` work. `docker-compose` is slated to be removed from the `docker` product. + +## Overview: Managing a database ## + +* A database is created using `docker compose ... up ...`; +* A database can be `stopped` _WITHOUT_ deleting its storage: + * `docker compose ... stop ...`; +* A stopped database can be `restarted` and will _CONTINUE_ using its previous existing storage: + * `docker compose ... start ...`; +* A database is `deleted` _WITH_ its storage using: + * `docker compose ... down`; +* A client app connects to a database by configuring a port on the host network - set with the variable `EXTERNAL_ADDRESS` - into its connection string/params; + * example: `EXTERNAL_ADDRESS=192.167.0.123` if the local host's network address is `192.168.0.123` + * example: `EXTERNAL_ADDRESS=localhost` if `docker` can resolve +* Because `docker compose` does not scope the Docker networks it creates to a particular file or profile, `docker compose ... down` may attempt to delete a network that is still in use by a different database. +The error looks like the following, and can be ignored: -1. cd to the `nuodb` directory. + ``` + ⠿ Network nuodb_net Error 0.0s + failed to remove network df0df85905b1702fea9c1a20a1142b9f4ff85f07844087b520f072c8a6af5e68: Error response from daemon: error while removing network: network nuodb_net id df0df85905b1702fea9c1a20a1142b9f4ff85f07844087b520f072c8a6af5e68 has active endpoints + ``` -2. copy the `env-default` file to `.env` (this step is _NOT_ optional). +**WARNING:** In the following do not confuse `start` and `stop` with `up` and `down`. +* `docker up` is used to create the database container and start it running. +* You can stop and restart the container using `docker stop` and `docker start` - no database data will be lost. +* `docker down` _destroys_ the container and your database will be lost. -3. edit the `.env` file: - - if you want to use a specific SQL engine - for example `scalar` - you will need an image that supports that engine; - - if you want to access the database from outside the `docker network` - for example from an app running direcly on the local host - then set `EXTERNAL_ADDRESS`; - - either in the `.env` file, _or_ by setting `EXTERNAL_ADDRESS` on the `docker-compose up` command-line; - - set to the address of the local host machine (Ex `192.168.0.123`); - - on some platforms, setting `EXTERNAL_ADDRESS` to `127.0.0.1` also works; - - if you want to import initial state from a database backup into the new database, set `IMPORT_LOCAL` and/or `IMPORT_REMOTE` (see `Notes` below for details of `IMPORT_LOCAL` and `IMPORT_REMOTE`); - - the `import` operation is only performed when the archive dir is _empty_ - so the SM container can be stopped and restarted without being reinitialised each time. - - if you have set `IMPORT_LOCAL` or `IMPORT_REMOTE` _and_ it is a large archive that takes multiple minutes to import, you _will_ need to - set `STARTUP_TIMEOUT` to a value larger than the time taken to import, to stop the DB startup from timing out before the IMPORT has completed. +### Managing a `distributed` Database ### -4. create and start the nuodb database with `docker-compose up -d`. +*NOTE:* the `distributed` database is the default configuration. -_*NOTE:*_ The `docker-compose` command may suggest to you to use `docker compose` instead. -*Don't - it doesn't work.* +* `create` with: `docker compose up -d`; +* `stop` (temporarily) all containers with: `docker compose stop` + * this will _*NOT*_ delete the database storage; +* `restart` a stopped database with: `docker compose start`; +* `delete` - including storage - with `docker compose down`; +* `connect` to a `distributed` database using the value of `EXTERNAL_ADDRESS` in the connection string. +#### Scaling Out a `distributed` Database #### -## Stopping the database ## -1. To stop all containers, but retain all stored state including the database contents: - - cd to the `nuodb` directory; - - execute `docker-compose stop` +* To scale out a `distributed` database with a _second_ TE: + * `docker compose --profile scale-te2 up -d`; +* to scale in a `te2` on a `distributed` database: + * `docker compose --profile scale-te2 stop`; +* to delete a `distributed` _plus_ its scaled-out `te2` in a single command: + * `docker compose --profile scale-te2 down`; -2. To restart a stopped database - complete with the database contents: - - cd to the `nuodb` directory; - - execute `docker-compose start`. +### Managing a `monolith` Database ### -## Deleting the database and all its storage ## -1. To stop all containers, and delete all resources - including the stored state and database contents: - - cd to the `nuodb` directory; - - execute `docker-compose down`. +*NOTE:* the `monolith` topology must be specified explicitly by using `-f monolith.yaml`. -## Notes ## -1. You can specify env vars on the command-line in linux or MacOS, by setting them _before_ the `docker-compose` command. -- Ex: `$ IMPORT_SOURCE=./mydb.bak.tgz STARTUP_TIMEOUT=300 docker-compose up -d` +* `create` with: `docker compose -f monolith.yaml up -d`; +* `stop` (temporarily) all containers with: `docker compose -f monolith.yaml stop`; + * this will _*NOT*_ delete the database storage; +* `restart` a stopped database with: `docker compose -f monolith.yaml start`; +* `delete` - including storage - with `docker compose -f monolith.yaml down`; +* `connect` to a `monolith` database using the value of `EXTERNAL_ADDRESS` in the connection string; + * example connection string: `jdbc:com.nuodb://192/.168.0.123/demo` + * example connection string: `jdbc:com.nuodb://localhost/demo` + +### Managing an `instadb` Database ### + +*NOTE:* the `instadb` topology must be specified explicitly by using `-f instadb.yaml`. + +* `create` with: `docker compose -f instadb.yaml up -d`; +* `stop` (temporarily) all containers with: `docker compose -f instadb.yaml stop`; + * this will _*NOT*_ delete the database storage; +* `restart` a stopped database with: `docker compose -f intadb.yaml start`; +* `delete` - including storage - with `docker compose -f instadb.yaml down`; +* `connect` to an `instadb` by setting `direct=true` in the connection properties; + * and setting the value of `EXTERNAL_ADDRESS:` in the connection string; + * where `` is the public port mapped to port `48006` for that container. + +**Port mapping example:** + +Note that the TE is always on container (internal) port 48006 + +``` +$ docker ps +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +458760270ee7 nuodb/nuodb:5.0.0.2 "docker-entrypoint.s…" 6 minutes ago Up 6 minutes 0.0.0.0:54587->8888/tcp, 0.0.0.0:54584->48004/tcp, 0.0.0.0:54585->48005/tcp, 0.0.0.0:54586->48006/tcp nuodb-instadb-1 + +$ docker compose port instadb 48006 +0.0.0.0:54586 +``` -2. the initial state of the database can be imported using `IMPORT_LOCAL` and/or `IMPORT_REMOTE`, as follows: -- set `IMPORT_LOCAL` to a path on the _local_ machine. - The SM container will mount this path as a volume and import it into the - archive dir prior to starting the SM process (presuming the archive is empty); +In this example, to connect to the database (actually to its TE) use the value of `EXTERNAL_ADDRESS:54586` in the connection string (along with `direct=true`). + * Example connection string: `jdbc:com.nuodb://192.168.0.123:54586/demo?direct=true` + * Example connection string: `jdbc:com.nuodb://localhost:54586/demo?direct=true` + +### Managing _Multiple_ `instadb` Databases ### + +* `create` an additional `instadb` with: `docker compose -p -f instadb.yaml up -d`; +* `stop` (temporarily) all containers of a specific `instadb` with: `docker compose -p -f instadb.yaml stop`; + * this will _*NOT*_ delete the database storage; +* `restart` a stopped database with: `docker compose -p -f intadb.yaml start`; +* `delete` - including storage - with `docker compose -p -f instadb.yaml down`; + +## Importing Existing Data ## + +The initial state of the database can be imported using `IMPORT_LOCAL` and/or `IMPORT_REMOTE`. + +*NOTE:* You can specify environment variables on the command-line on Linux or MacOS, by setting them _before_ the `docker compose` command. + +- *Example:* `$ IMPORT_LOCAL=./mydb.bak.tgz STARTUP_TIMEOUT=300 docker compose up -d` + +Steps required: + +- Set `IMPORT_LOCAL` to a path on the _local_ machine. + The SM container will mount this path as a volume and import its contents into the archive dir prior to starting the SM process (presuming the archive is empty); The path that `IMPORT_LOCAL` points to can be one of: - a `tar.gzip` file of a `nuodb backup`; - - a directory cotaining a `nuodb backup`. + - a directory containing a `nuodb backup`. - *Note*: that a `nuodb backup` can come in 1 of 2 formats: + *Note*: that a `nuodb backup` can come in one of two formats: - a nuodb `backup set` - which is the result of a `hotcopy --type full` backup; - a nuodb `archive` - which is the result of a `hotcopy --type simple`, or just a copy of an SM archive and journal copied while the SM is _NOT_ running. *Note*: a `backupset` can only be imported from a directory. -- set `IMPORT_REMOTE` to a URL of a remote `file` hosted on a server - typically accessed through `http(s)` or `(s)ftp`. - - Ex: `https://some.server.io/backup-4-3.tz` - - Ex: `sftp://sftp.some.domain.com/backup-4-3.tz` +- Set `IMPORT_REMOTE` to the URL of a remote `file` hosted on a server - typically accessed through `http(s)` or `(s)ftp`. + - Example: `https://some.server.io/backup-4-3.tz` + - Example: `sftp://sftp.some.domain.com/backup-4-3.tz` *Note* that: - The SM container will download the remote file via the URL and extract it into the archive dir prior to starting the SM process. -- if you set _both_ `IMPORT_LOCAL` _and_ `IMPORT_REMOTE`, then `IMPORT_REMOTE` is treated as the remote source, and `IMPORT_LOCAL` is treated as a locally cached copy - hence the behaviour is as follows: - - if `IMPORT_LOCAL` is a _non-empty_ `file` or `directory`, then it is used directly, and `IMPORT_REMOTE` is ignored. - - if `IMPORT_LOCAL` is an _empty_ `file` then `IMPORT_REMOTE` is downloaded into `IMPORT_LOCAL`, and the `import` is then performed by `extracting` from `IMPORT_LOCAL` into the `archive`; + The SM container will download a `remote` file via the URL and extract it into the archive directory prior to starting the SM process. +- If you set _both_ `IMPORT_LOCAL` _and_ `IMPORT_REMOTE`, then `IMPORT_REMOTE` is treated as the remote source, and `IMPORT_LOCAL` is treated as a locally cached copy. - Hence the behaviour is as follows: + - If `IMPORT_LOCAL` is a _non-empty_ `file` or `directory`, then it is used directly, and `IMPORT_REMOTE` is ignored. + - If `IMPORT_LOCAL` is an _empty_ `file` then `IMPORT_REMOTE` is downloaded into `IMPORT_LOCAL`, and the `import` is then performed by `extracting` from `IMPORT_LOCAL` into the `archive`; - note this _only_ works for a `tar.gzip` file of an `archive` (see above). - - if `IMPORT_LOCAL` is an _empty_ `directory` then `IMPORT_REMOTE` is downloaded and extracted into `IMPORT_LOCAL`, and the `import` is then performed from `IMPORT_LOCAL` into the `archive`; + - If `IMPORT_LOCAL` is an _empty_ `directory` then `IMPORT_REMOTE` is downloaded and extracted into `IMPORT_LOCAL`, and the `import` is then performed from `IMPORT_LOCAL` into the `archive`; - note this works for _both_ forms of `nuodb backup` (see above); - - *Note* importing from a `directory` can be significantly _slower_ than imorting (`extracting directly`) from a `tar.gzip` file. + - *Note:* importing from a `directory` can be significantly _slower_ than importing (`extracting directly`) from a `tar.gzip` file. - _Hence:*_ To cause the initial download from `IMPORT_REMOTE` to be cached in `IMPORT_LOCAL`, `IMPORT_LOCAL` _must_ exist _and_ be empty. + _*Hence:*_ To cause the initial download from `IMPORT_REMOTE` to be cached in `IMPORT_LOCAL`, `IMPORT_LOCAL` _must_ exist _and_ be empty. To ensure this, you can do something like the following: - `$ rm -rf a/b/c` - `$ touch a/b/c` or `mkdir -p /a/b/c` Now you can set `IMPORT_REMOTE` as needed, and set `IMPORT_LOCAL` to `a/b/c`. -## What could possibly go wrong?? ## +## What Could Possibly Go Wrong?? ## 1. If you get an error in the form: -``` + ``` WARNING: The NUODB_IMAGE variable is not set. Defaulting to a blank string. WARNING: The PEER_ADDRESS variable is not set. Defaulting to a blank string. WARNING: The IMPORT_SOURCE variable is not set. Defaulting to a blank string. ... ERROR: Couldn't find env file: /a/b/c/nuodb/.env - ``` -- then you have most likely forgotten to copy the `env-default` file to `.env`; -- or (less popular) not created your own `.env` file from scratch. + ``` + - then you have most likely forgotten to copy the `env-default` file to `.env`; + - or (less popular) not created your own `.env` file from scratch. 2. If you get one or more warnings of the form: -``` + ``` WARNING: The variable is not set. Defaulting to a blank string. -``` - - but NO ERROR, then you probably need to update your `.env` file. - - Do a `diff` of `.env` vs `env-default`, and look for new VARs defined in the latest `env-default`. + ``` + - but NO ERROR, then you probably need to update your `.env` file. + - Do a `diff` of `.env` vs `env-default`, and look for new VARs defined in the latest `env-default`. -3. if you get an error of the form: -``` -ERROR: for nuodb_nuocd-sm_1 Cannot start service nuocd-sm: container ... is not running -``` -then check the logs of the `sm_1` container to see why it has failed to start. -To check the logs of any container, run `docker logs ` -Ex: `docker logs nuodb_sm_1` +3. If you get an error of the form: + ``` + ERROR: for nuodb_nuocd-sm_1 Cannot start service nuocd-sm: container ... is not running + ``` + check the logs of the `sm_1` container to see why it has failed to start. + - To check the logs of any container, run `docker logs ` + - Example: `docker logs nuodb_sm_1` 4. If you get an error in the form: -``` -IMPORT_REMOTE is not a valid URL: ... - import aborted -``` -then you have not set `IMPORT_REMOTE` to a valid URL. -A URL is in the form: :/// -Ex: `sftp://myhost.com/my-archives/backup-xyz.tar.gzip` + ``` + IMPORT_REMOTE is not a valid URL: ... - import aborted + ``` + then you have not set `IMPORT_REMOTE` to a valid URL. + - URL is expected to be in the form: `:///` + - Example: `sftp://myhost.com/my-archives/backup-xyz.tar.gzip` 5. If you get an error in the form: -``` -This database has archives with no running SM. -No new SMs can start while the database is in this state. -``` -then you have somehow restarted the database with existing archives but too few running SMs. -This could happen if an import has somehow failed after the initial import started, and you restart with `IMPORT_X` set. -This could also happen if an SM has shut down, and you try to restart it with `docker-compose up`, but have accidentally set `IMPORT_X`. -(You cannot attempt to import the database state if there is existing state in some archive - even if the SM for that archive is not currently running.) -Follow the instructions following the error message to resolve the problem(s), and then continue stating with: -`... docker-compose up -d` - -6. If an error causes only part of the database to be deployed, you can start the remaining containers - after fixing the error - by simply running `... docker-compose up -d` again. The `up` command only starts those containers that are not currently running. -When running `... docker-compose up` a subsequent time, you need to decide if you still need to set `IMPORT_X` variable(s): - - you _DON'T_ need to if the database state has already been successfully imported; - - you probably _DO_ need to if you had them set for the original `docker up` command, and the `import` has not yet succeeded. + ``` + This database has archives with no running SM. + No new SMs can start while the database is in this state. + ``` + then you have somehow restarted the database with existing archives but too few running SMs. + - This could happen if an import has somehow failed after the initial import started, and you restart with `IMPORT_X` set. + - This could also happen if an SM has shut down, and you try to restart it with `docker compose up`, but have accidentally set `IMPORT_X`. + (You cannot attempt to import the database state if there is existing state in some archive - even if the SM for that archive is not currently running.) + - Follow the instructions following the error message to resolve the problem(s), and then continue stating with: + `... docker compose up -d` + +6. If an error causes only part of a `distributed` database to be deployed, you can start the remaining containers - after fixing the error - by simply running `... docker compose up -d` again. The `up` command only starts those containers that are not currently running. + - When running `... docker compose up` a subsequent time, you need to decide if you still need to set `IMPORT_X` variable(s): + - you _DON'T_ need to if the database state has already been successfully imported; + - you probably _DO_ need to if you had them set for the original `docker compose up` command, and the `import` has not yet succeeded. +7. If you get an error about being unable to delete a network because it has active end-points, you can normally safely ignore this. + ``` + ⠿ Network nuodb_net Error 0.0s + failed to remove network df0df85905b1702fea9c1a20a1142b9f4ff85f07844087b520f072c8a6af5e68: Error response from daemon: error while removing network: network nuodb_net id df0df85905b1702fea9c1a20a1142b9f4ff85f07844087b520f072c8a6af5e68 has active endpoints + ``` diff --git a/nuodb/docker-compose.yaml b/nuodb/docker-compose.yaml index 5004dd9..70f4773 100644 --- a/nuodb/docker-compose.yaml +++ b/nuodb/docker-compose.yaml @@ -1,7 +1,16 @@ version: '3' + +networks: + net: + services: nuoadmin1: image: $NUODB_IMAGE + # profiles: [ "distributed" ] + restart: ${RESTART_POLICY:-unless-stopped} + + networks: + net: # Do NOT remove this env_file value!! env_file: .env @@ -12,178 +21,159 @@ services: ports: - 48004-48005:48004-48005 - 8888:8888 - command: - - "nuoadmin" - - "--" - - "pendingProcessTimeout=${STARTUP_TIMEOUT:-60}000" - - "pendingReconnectTimeout=90000" - - "thrift.message.max=1073741824" - - "processLivenessCheckSec=30" + volumes: + - ./scripts:/usr/local/scripts + - ./scripts/stop-nuodb:/usr/local/bin/stop-nuodb + + command: [ "/usr/local/scripts/start-nuoadmin" ] sm: image: $NUODB_IMAGE + # profiles: [ "distributed" ] + restart: ${RESTART_POLICY:-unless-stopped} + + networks: + net: + + # Do NOT remove this env_file value!! + env_file: .env + environment: - # push the current resolved value of the VAR... + PEER_ADDRESS: ${PEER_ADDRESS:-nuoadmin1} NUOCMD_API_SERVER: ${PEER_ADDRESS:-nuoadmin1}:8888 ARCHIVE_DIR: ${ARCHIVE_PATH:-/var/opt/nuodb/archive} DB_OPTIONS: "mem ${ENGINE_MEM:-1Gi} execution-engine ${SQL_ENGINE:-vee} ${ENGINE_OPTIONS:-}" + STARTUP_TIMEOUT: ${STARTUP_TIMEOUT:-90} hostname: sm1 depends_on: - nuoadmin1 volumes: + - ./scripts:/usr/local/scripts + - ./scripts/stop-nuodb:/usr/local/bin/stop-nuodb - ${IMPORT_LOCAL:-./empty-file}:${IMPORT_MOUNT:-/var/tmp/env} - ## NOTE: use '$$' for any variable that is to be evaluated at runtime IN THE SHELL. - # Any variable with a single '$' is resolved by docker-compose and written literally into the command string. - command: - - "sh" - - "-c" - - | - msg=$$(nuocmd check servers --timeout ${STARTUP_TIMEOUT:-60} --check-converged --check-active) - [ $$? -ne 0 ] && echo "ERROR: SM timed out waiting for admin layer to be ready: $$msg" && exit 98 - echo "ARCHIVE_DIR=$$ARCHIVE_DIR" - mkdir -p $$ARCHIVE_DIR - [ $$? -ne 0 ] && echo "Error creating $$ARCHIVE_DIR: $$?" && exit 98 - nuocmd show archives --db-name $DB_NAME - runningArchives=$$(nuocmd get archives --db-name "$DB_NAME" | grep 'state=RUNNING' | wc -l) - notRunningArchives=$$(nuocmd get archives --db-name "DB_NAME" | grep -v "state=RUNNING" | wc -l) - myArchive=$$( nuocmd show archives --db-name $DB_NAME --archive-format "archive-id: {id}" | sed -En "/^archive-id: / {N; /$$HOSTNAME/ s/^archive-id: ([0-9]+).*$$/\1/; T; p}" | head -n 1 ) - [ -z "$$myArchive" ] && myArchive=$$( nuocmd show archives --db-name $DB_NAME --removed --removed-archive-format "archive-id: {id}" | sed -En "/^archive-id: / {N; /$$HOSTNAME/ s/^archive-id: ([0-9]+).*$$/\1/; T; p}" | head -n 1 ) - if [ -n "$$myArchive" -a $$notRunningArchives -gt 1 ] || [ -z "$$myArchive" -a $$notRunningArchives -gt 0 ]; then - echo "This database has $$notRunningArchives archives with no running SM." - echo "No new SMs can start while the database is in this state." - echo "To fix this, you can:" - echo " 1. start SMs running for some or all non-running archives;" - echo " 2. remove all remaining non-running archives with 'nuocmd delete archive (--purge)';" - echo " 3. or delete the entire database with 'compose down'." - echo "After fixing the issue(s) per above, you can run 'compose up' again." - exit 98 - fi - if [ -n '$IMPORT_LOCAL$IMPORT_REMOTE' -a ! -f "$$ARCHIVE_DIR/1.atm" -a "$$runningArchives" -eq 0 ]; then - echo "Importing into empty archive..." - [[ -n '$IMPORT_REMOTE' && '$IMPORT_REMOTE' != ?*://?* ]] && echo "ERROR: IMPORT_REMOTE is not a valid URL: $IMPORT_REMOTE - import aborted" && exit 98 - if [ -n "$$myArchive" ]; then - echo "Cleaning up archive tombstone for $$HOSTNAME: $$myArchive..." - [ $$(nuocmd get archives --db-name $DB_NAME | wc -l) -eq 1 ] && echo "Cleaning up database first..." && nuocmd delete database --db-name $DB_NAME 2>&1 || exit 98 - nuocmd delete archive --archive-id $$myArchive --purge 2>&1 || exit 98 - fi - importFromCache='false' - if [ -n '$IMPORT_REMOTE' ]; then - [ -n '$IMPORT_AUTH' -a '$IMPORT_AUTH' != ':' ] && curlAuth='--user $IMPORT_AUTH' - if [ -n '$IMPORT_LOCAL' ]; then - if [ -d '$IMPORT_MOUNT' -a $$(ls -1 '$IMPORT_MOUNT' | wc -l) -eq 0 ]; then - echo "Extracting and caching $IMPORT_REMOTE into directory host:$IMPORT_LOCAL..." - time curl -k $${curlAuth:-} '$IMPORT_REMOTE' | tar xzf - --strip-components ${IMPORT_LEVEL:-1} -C $IMPORT_MOUNT || exit 98 - importFromCache='true' - elif [ ! -s '$IMPORT_MOUNT' ]; then - echo "Caching $IMPORT_REMOTE into file host:$IMPORT_LOCAL..." - time curl -k $${curlAuth:-} '$IMPORT_REMOTE' > '$IMPORT_MOUNT' || exit 98 - importFromCache='true' - else - echo "host:$IMPORT_LOCAL is not empty - assuming it contains a cached copy of $IMPORT_REMOTE." - importFromCache='true' - fi - else - echo "IMPORT_LOCAL is not set - caching disabled." - echo "Importing from $IMPORT_REMOTE into $$ARCHIVE_DIR..." - time curl -k $${curlAuth:-} '$IMPORT_REMOTE' | tar xzf - --strip-components ${IMPORT_LEVEL:-1} -C $$ARCHIVE_DIR || exit 98 - fi - else - [ -f '$IMPORT_MOUNT' -a ! -s '$IMPORT_MOUNT' ] && echo "ERROR: IMPORT_LOCAL file host:$IMPORT_LOCAL is empty." && exit 98 - [ -d '$IMPORT_MOUNT' -a $$(ls -1 '$IMPORT_MOUNT' | wc -l) -eq 0 ] && echo "ERROR: IMPORT_LOCAL directory host:$IMPORT_LOCAL is empty." && exit 98 - importFromCache='true' - fi - if [ -n '$IMPORT_LOCAL' ]; then - [ -n '$IMPORT_REMOTE' -a "$$importFromCache" = 'true' -a -s '$IMPORT_MOUNT' ] && echo "Using host:$IMPORT_LOCAL as a cached copy of $IMPORT_REMOTE..." - if [ -d '$IMPORT_MOUNT' ]; then - echo "Importing directory host:$IMPORT_LOCAL into $$ARCHIVE_DIR..." - time nuodocker restore archive --origin-dir $IMPORT_MOUNT --restore-dir $$ARCHIVE_DIR --db-name "$DB_NAME" --clean-metadata || exit 98 - elif [ "$$importFromCache" = 'true' -a -s '$IMPORT_MOUNT' ]; then - echo "Importing file host:$IMPORT_LOCAL into $$ARCHIVE_DIR..." - time tar xf '$IMPORT_MOUNT' --strip-components ${IMPORT_LEVEL:-1} -C "$$ARCHIVE_DIR" || exit 98 - else - echo "ERROR: IMPORT_LOCAL has been specified, but host:$IMPORT_LOCAL is not a valid import source - IMPORT_LOCAL must be a directory, an initially empty file, or a cached copy of IMPORT_REMOTE - import aborted..." - exit 98 - fi - fi - [ -d "$$ARCHIVE_DIR/full" ] && echo "ERROR: Imported data looks like a BACKUPSET (in which case IMPORT_LOCAL must be a DIRECTORY): $$(ls -l $$ARCHIVE_DIR | head -n 10)" && exit 98 - [ ! -f "$$ARCHIVE_DIR/1.atm" ] && echo "ERROR: Imported archive does not seem to contain valid data: $$(ls -l $$ARCHIVE_DIR | head -n 10)" && exit 98 - echo "Imported data looks good: $$(ls -l $$ARCHIVE_DIR | head -n 5)" - if [ ! -d '$IMPORT_MOUNT' ]; then - nuodocker restore archive --origin-dir "$$ARCHIVE_DIR" --restore-dir "$$ARCHIVE_DIR" --db-name "$DB_NAME" --clean-metadata || exit 99 - fi - fi - nuodocker start sm --db-name '$DB_NAME' --server-id '${PEER_ADDRESS:-nuoadmin1}' --archive-dir $$ARCHIVE_DIR --dba-user '$DB_USER' --dba-password '$DB_PASSWORD' --options 'alt-address sm1' --database-options "$$DB_OPTIONS" + command: [ "/usr/local/scripts/start-nuosm" ] te1: image: $NUODB_IMAGE + # profiles: [ "distributed" ] + restart: ${RESTART_POLICY:-unless-stopped} + + networks: + net: + + # Do NOT remove this env_file value!! + env_file: .env + environment: - # push the current resolved value of the VARs + PEER_ADDRESS: ${PEER_ADDRESS:-nuoadmin1} NUOCMD_API_SERVER: ${PEER_ADDRESS:-nuoadmin1}:8888 STARTUP_TIMEOUT: ${STARTUP_TIMEOUT:-90} + EXTERNAL_ADDRESS: ${EXTERNAL_ADDRESS:-te1} hostname: te1 depends_on: - nuoadmin1 - sm ports: - 48006:48006 + volumes: + - ./scripts:/usr/local/scripts + - ./scripts/stop-nuodb:/usr/local/bin/stop-nuodb + + command: [ "/usr/local/scripts/start-nuote" ] + + te2: + image: $NUODB_IMAGE + profiles: [ "scale-te2" ] + restart: ${RESTART_POLICY:-unless-stopped} + + networks: + net: + + # Do NOT remove this env_file value!! + env_file: .env + + environment: + PEER_ADDRESS: ${PEER_ADDRESS:-nuoadmin1} + NUOCMD_API_SERVER: ${PEER_ADDRESS:-nuoadmin1}:8888 + STARTUP_TIMEOUT: ${STARTUP_TIMEOUT:-90} + EXTERNAL_ADDRESS: ${EXTERNAL_ADDRESS:-te2} + hostname: te2 + + ports: + - 48007:48006 + volumes: + - ./scripts:/usr/local/scripts + - ./scripts/stop-nuodb:/usr/local/bin/stop-nuodb + + command: [ "/usr/local/scripts/start-nuote" ] + + monolith: + image: $NUODB_IMAGE + profiles: [ "monolith" ] + restart: ${RESTART_POLICY:-unless-stopped} + networks: + net: + + # Do NOT remove this env_file value!! + env_file: .env + + environment: + PEER_ADDRESS: ${PEER_ADDRESS:-db} + NUODB_DOMAIN_ENTRYPOINT: ${PEER_ADDRESS:-db} + NUOCMD_API_SERVER: localhost:8888 + STARTUP_TIMEOUT: ${STARTUP_TIMEOUT:-90} + EXTERNAL_ADDRESS: ${EXTERNAL_ADDRESS:-localhost} + ARCHIVE_DIR: ${ARCHIVE_PATH:-/var/opt/nuodb/archive} + DB_OPTIONS: "mem ${ENGINE_MEM:-1Gi} execution-engine ${SQL_ENGINE:-vee} ${ENGINE_OPTIONS:-}" + hostname: ${PEER_ADDRESS:-db} + ports: + - 48004-48006:48004-48006 + - 8888:8888 + volumes: + - ./scripts:/usr/local/scripts + - ./scripts/stop-nuodb:/usr/local/bin/stop-nuodb + - ${IMPORT_LOCAL:-./empty-file}:${IMPORT_MOUNT:-/var/tmp/env} + + command: [ "/usr/local/scripts/start-monolith" ] + + instadb: + image: $NUODB_IMAGE + profiles: [ "instadb" ] + restart: ${RESTART_POLICY:-unless-stopped} + networks: + net: + + # Do NOT remove this env_file value!! + env_file: .env + + environment: + PEER_ADDRESS: localhost + NUODB_DOMAIN_ENTRYPOINT: localhost + NUOCMD_API_SERVER: localhost:8888 + STARTUP_TIMEOUT: ${STARTUP_TIMEOUT:-90} + EXTERNAL_ADDRESS: ${EXTERNAL_ADDRESS:-localhost} + ARCHIVE_DIR: ${ARCHIVE_PATH:-/var/opt/nuodb/archive} + DB_OPTIONS: "mem ${ENGINE_MEM:-1Gi} execution-engine ${SQL_ENGINE:-vee} ${ENGINE_OPTIONS:-}" + ports: + - :48004-48006 + - :8888 + volumes: + - ./scripts:/usr/local/scripts + - ./scripts/stop-nuodb:/usr/local/bin/stop-nuodb + - ${IMPORT_LOCAL:-./empty-file}:${IMPORT_MOUNT:-/var/tmp/env} + + command: [ "/usr/local/scripts/start-monolith" ] - ## NOTE: use '$$' for any variable that is to be evaluated at runtime IN THE SHELL. - # Any variable with a single '$' is resolved by docker-compose and written literally into the command string. - command: - - "sh" - - "-c" - - | - sleepTime=0 - sleepQuantum=10 - while [ -z "$$(nuocmd get processes --db-name $DB_NAME | grep 'state=RUNNING')" -o $$? -ne 0 ] ; do - if [ $$sleepTime -ge $$STARTUP_TIMEOUT ]; then - echo "Timed out waiting for database startup ($$sleepTime sec)..." - exit 97 - fi - echo "Waiting for confirmation that database $DB_NAME is started..." - sleep $$sleepQuantum - sleepTime=$$(( sleepTime + sleepQuantum )) - [ $$sleepQuantum -lt $$((STARTUP_TIMEOUT / 2)) ] && sleepQuantum=$$(( sleepQuantum + 30 )) - done - nuodocker start te --db-name '$DB_NAME' --server-id '${PEER_ADDRESS:-nuoadmin1}' --options 'alt-address ${EXTERNAL_ADDRESS:-te1}' - - # te2: - # image: $NUODB_IMAGE - # environment: - # # push the current resolved value of the VARs - # NUOCMD_API_SERVER: ${PEER_ADDRESS:-nuoadmin1}:8888 - # STARTUP_TIMEOUT: ${STARTUP_TIMEOUT:-90} - # hostname: te2 - # scale: 0 - # depends_on: - # - nuoadmin1 - # - sm - # ports: - # - 48007:48006 - # - # ## NOTE: use '$$' for any variable that is to be evaluated at runtime IN THE SHELL. - # # Any variable with a single '$' is resolved by docker-compose and written literally into the command string. - # command: - # - "sh" - # - "-c" - # - | - # sleepTime=0 - # sleepQuantum=10 - # while [ -z "$$(nuocmd get processes --db-name $DB_NAME | grep 'state=RUNNING')" -o $$? -ne 0 ] ; do - # if [ $$sleepTime -ge $$STARTUP_TIMEOUT ]; then - # echo "Timed out waiting for database startup ($$sleepTime sec)..." - # exit 97 - # fi - # echo "Waiting for confirmation that database $DB_NAME is started..." - # sleep $$sleepQuantum - # sleepTime=$$(( sleepTime + sleepQuantum )) - # [ $$sleepQuantum -lt $$((STARTUP_TIMEOUT / 2)) ] && sleepQuantum=$$(( sleepQuantum + 30 )) - # done - # nuodocker start te --db-name '$DB_NAME' --server-id '${PEER_ADDRESS:-nuoadmin1}' --options 'alt-address ${EXTERNAL_ADDRESS:-te2}' influxdb: image: influxdb:1.8 + profiles: [ "insights" ] + restart: ${RESTART_POLICY:-unless-stopped} + + networks: + net: + # ports: # The API for InfluxDB is served on port 8086 # - "8086:8086" @@ -193,17 +183,29 @@ services: nuocd-sm: image: nuodb/nuodb-collector:latest + profiles: [ "insights" ] + restart: ${RESTART_POLICY:-unless-stopped} + + networks: + net: + depends_on: - nuoadmin1 - sm - influxdb environment: INFLUXURL: http://influxdb:8086 - NUOCD_HOSTNAME: sm + NUOCD_HOSTNAME: sm1 pid: 'service:sm' nuocd-te1: image: nuodb/nuodb-collector:latest + profiles: [ "insights" ] + restart: ${RESTART_POLICY:-unless-stopped} + + networks: + net: + depends_on: - nuoadmin1 - te1 @@ -213,20 +215,31 @@ services: NUOCD_HOSTNAME: te1 pid: 'service:te1' - # nuocd-te2: - # image: nuodb/nuodb-collector:latest - # depends_on: - # - nuoadmin1 - # - te2 - # - influxdb - # environment: - # INFLUXURL: http://influxdb:8086 - # NUOCMD_API_SERVER: - # NUOCD_HOSTNAME: te2 - # pid: 'service:te2' + nuocd-te2: + image: nuodb/nuodb-collector:latest + profiles: [ "scale-te2" ] + restart: ${RESTART_POLICY:-unless-stopped} + + networks: + net: + + depends_on: + - nuoadmin1 + - te2 + - influxdb + environment: + INFLUXURL: http://influxdb:8086 + NUOCD_HOSTNAME: te2 + pid: 'service:te2' grafana: image: grafana/grafana:7.5.4 + profiles: [ "insights" ] + restart: ${RESTART_POLICY:-unless-stopped} + + networks: + net: + depends_on: - influxdb environment: @@ -238,12 +251,13 @@ services: # ycsb-demo: # image: nuodb/ycsb:latest + # networks: + # net: # depends_on: # - te1 # environment: - # # a VAR with no value pushes the currently resolved value of that VAR + # PEER_ADDRESS: ${PEER_ADDRESS:-nuoadmin1} # DB_NAME: # DB_USER: # DB_PASSWORD: - # PEER_ADDRESS: ${PEER_ADDRESS:-nuoadmin1} # command: ["/driver/startup.sh"] diff --git a/nuodb/env-default b/nuodb/env-default index 58e0ad3..ed444fa 100644 --- a/nuodb/env-default +++ b/nuodb/env-default @@ -2,18 +2,25 @@ # default ENV VAR values # -NUODB_IMAGE=nuodb/nuodb-ce:4.1.2.vee-4 +NUODB_IMAGE=nuodb/nuodb-ce:5.0.1-2 DB_NAME=demo DB_USER=dba DB_PASSWORD=dba ENGINE_MEM=1Gi SQL_ENGINE=vee +LOGDIR=/var/log/nuodb -# Set to a larger value if SM startup takes unusually long -# - for example if IMPORT_LOCAL or IMPORT_REMOTE (see below) is a large file that takes multiple minutes to extract. -# Value is in seconds. -STARTUP_TIMEOUT=60 +# docker compose restart policy. +# set to one of: +# - "no" +# - always +# - on-failure +# - unless-stopped +RESTART_POLICY=unless-stopped + +# Set to a larger value if database startup takes unusually long +STARTUP_TIMEOUT=90 # Uncomment and set, or set on the docker-compose command-line to add further engine options # ENGINE_OPTIONS= @@ -21,6 +28,10 @@ STARTUP_TIMEOUT=60 # normally this is left unset, causing the default to be used. ARCHIVE_PATH= +# Set to a larger value if IMPORT_x is set to a large file or dir that takes multiple minutes to restore. +# Value is in seconds. +IMPORT_TIMEOUT= + # set IMPORT_LOCAL to the path of a LOCAL tar file on the host where docker-compose is being run. # The SM container will mount the file, extract (untar) it and use the contents as the initial state of the database. IMPORT_LOCAL= @@ -39,9 +50,6 @@ IMPORT_AUTH= # can advise on any non-standard value required for IMPORT_LEVEL. IMPORT_LEVEL=1 -# Set this to 'true' if the content to be imported is a backupset output from a hotcopy --full WITHOUT the --simple option. -IMPORT_IS_BACKUPSET=false - # This value is not normally changed. IMPORT_MOUNT=/var/opt/nuodb/import diff --git a/nuodb/instadb.yaml b/nuodb/instadb.yaml new file mode 100644 index 0000000..ceeb414 --- /dev/null +++ b/nuodb/instadb.yaml @@ -0,0 +1,47 @@ +version: '3' + +networks: + instadb: + +services: + instadb: + image: $NUODB_IMAGE + # profiles: [ "instadb" ] + restart: ${RESTART_POLICY:-unless-stopped} + networks: + instadb: + + # Do NOT remove this env_file value!! + env_file: .env + + environment: + PEER_ADDRESS: localhost + NUODB_DOMAIN_ENTRYPOINT: localhost + NUOCMD_API_SERVER: localhost:8888 + STARTUP_TIMEOUT: ${STARTUP_TIMEOUT:-90} + EXTERNAL_ADDRESS: ${EXTERNAL_ADDRESS:-localhost} + ARCHIVE_DIR: ${ARCHIVE_PATH:-/var/opt/nuodb/archive} + DB_OPTIONS: "mem ${ENGINE_MEM:-1Gi} execution-engine ${SQL_ENGINE:-vee} ${ENGINE_OPTIONS:-}" + ports: + - :48004-48006 + - :8888 + volumes: + - ./scripts:/usr/local/scripts + - ./scripts/stop-nuodb:/usr/local/bin/stop-nuodb + - ${IMPORT_LOCAL:-./empty-file}:${IMPORT_MOUNT:-/var/tmp/env} + + command: [ "/usr/local/scripts/start-monolith" ] + + + # ycsb-demo: + # image: nuodb/ycsb:latest + # networks: + # net: + # depends_on: + # - te1 + # environment: + # PEER_ADDRESS: ${PEER_ADDRESS:-nuoadmin1} + # DB_NAME: + # DB_USER: + # DB_PASSWORD: + # command: ["/driver/startup.sh"] diff --git a/nuodb/monolith.yaml b/nuodb/monolith.yaml new file mode 100644 index 0000000..d3c4d53 --- /dev/null +++ b/nuodb/monolith.yaml @@ -0,0 +1,48 @@ +version: '3' + +networks: + net: + +services: + monolith: + image: $NUODB_IMAGE + # profiles: [ "monolith" ] + restart: ${RESTART_POLICY:-unless-stopped} + networks: + net: + + # Do NOT remove this env_file value!! + env_file: .env + + environment: + PEER_ADDRESS: ${PEER_ADDRESS:-db} + NUODB_DOMAIN_ENTRYPOINT: ${PEER_ADDRESS:-db} + NUOCMD_API_SERVER: localhost:8888 + STARTUP_TIMEOUT: ${STARTUP_TIMEOUT:-90} + EXTERNAL_ADDRESS: ${EXTERNAL_ADDRESS:-localhost} + ARCHIVE_DIR: ${ARCHIVE_PATH:-/var/opt/nuodb/archive} + DB_OPTIONS: "mem ${ENGINE_MEM:-1Gi} execution-engine ${SQL_ENGINE:-vee} ${ENGINE_OPTIONS:-}" + hostname: ${PEER_ADDRESS:-db} + ports: + - 48004-48006:48004-48006 + - 8888:8888 + volumes: + - ./scripts:/usr/local/scripts + - ./scripts/stop-nuodb:/usr/local/bin/stop-nuodb + - ${IMPORT_LOCAL:-./empty-file}:${IMPORT_MOUNT:-/var/tmp/env} + + command: [ "/usr/local/scripts/start-monolith" ] + + + # ycsb-demo: + # image: nuodb/ycsb:latest + # networks: + # net: + # depends_on: + # - te1 + # environment: + # PEER_ADDRESS: ${PEER_ADDRESS:-nuoadmin1} + # DB_NAME: + # DB_USER: + # DB_PASSWORD: + # command: ["/driver/startup.sh"] diff --git a/nuodb/scripts/import-archive b/nuodb/scripts/import-archive new file mode 100755 index 0000000..8c9baca --- /dev/null +++ b/nuodb/scripts/import-archive @@ -0,0 +1,81 @@ +#!/bin/sh + +# import the contents of the database archive + +: ${IMPORT_LEVEL:=1} + +# If archive IMPORT has been defined, and there is no existing archive, then perform the import +if [ -n "$IMPORT_LOCAL$IMPORT_REMOTE" -a ! -f "$ARCHIVE_DIR/1.atm" -a "$runningArchives" -eq 0 ]; then + echo "Importing into empty archive..." + [[ -n "$IMPORT_REMOTE" && "$IMPORT_REMOTE" != ?*://?* ]] && echo "ERROR: IMPORT_REMOTE is not a valid URL: $IMPORT_REMOTE - import aborted" && exit 98 + + # clean up any tombstone of the archive for this SM + if [ -n "$myArchive" ]; then + echo "Cleaning up archive tombstone for $HOSTNAME: $myArchive..." + [ $(nuocmd get archives --db-name $DB_NAME | wc -l) -eq 1 ] && echo "Cleaning up database first..." && nuocmd delete database --db-name $DB_NAME 2>&1 || exit 98 + nuocmd delete archive --archive-id $myArchive --purge 2>&1 || exit 98 + fi + + # if IMPORT_REMOTE is set - work out whether to import from existing (IMPORT_LOCAL) cache + importFromCache='false' + if [ -n "$IMPORT_REMOTE" ]; then + [ -n "$IMPORT_AUTH" -a "$IMPORT_AUTH" != ':' ] && curlAuth="--user $IMPORT_AUTH" + if [ -n "$IMPORT_LOCAL" ]; then + + # IMPORT_LOCAL is an empty dir + if [ -d "$IMPORT_MOUNT" -a $(ls -1 "$IMPORT_MOUNT" | wc -l) -eq 0 ]; then + echo "Extracting and caching $IMPORT_REMOTE into directory host:$IMPORT_LOCAL..." + time curl -k ${curlAuth:-} "$IMPORT_REMOTE" | tar xzf - --strip-components ${IMPORT_LEVEL} -C $IMPORT_MOUNT || exit 98 + importFromCache='true' + + # IMPORT_LOCAL is an empty file + elif [ ! -s "$IMPORT_MOUNT" ]; then + echo "Caching $IMPORT_REMOTE into file host:$IMPORT_LOCAL..." + time curl -k ${curlAuth:-} "$IMPORT_REMOTE" > "$IMPORT_MOUNT" || exit 98 + importFromCache='true' + + # IMPORT_LOCAL is not empty - assume it is a valid cache + else + echo "host:$IMPORT_LOCAL is not empty - assuming it contains a cached copy of $IMPORT_REMOTE." + importFromCache='true' + fi + + # IMPORT_LOCAL is not set - so there is no local cache + else + echo "IMPORT_LOCAL is not set - caching disabled." + echo "Importing from $IMPORT_REMOTE into $ARCHIVE_DIR..." + time curl -k ${curlAuth:-} "$IMPORT_REMOTE" | tar xzf - --strip-components ${IMPORT_LEVEL} -C $ARCHIVE_DIR || exit 98 + fi + + # IMPORT_REMOTE is not set, so check that IMPORT_LOCAL is not empty + else + [ -f "$IMPORT_MOUNT" -a ! -s "$IMPORT_MOUNT" ] && echo "ERROR: IMPORT_LOCAL file host:$IMPORT_LOCAL is empty." && exit 98 + [ -d "$IMPORT_MOUNT" -a $(ls -1 "$IMPORT_MOUNT" | wc -l) -eq 0 ] && echo "ERROR: IMPORT_LOCAL directory host:$IMPORT_LOCAL is empty." && exit 98 + importFromCache='true' + fi + + # IMPORT_LOCAL should now have the correct content - import it into the archive + if [ -n "$IMPORT_LOCAL" ]; then + [ -n "$IMPORT_REMOTE" -a "$importFromCache" = 'true' -a -s "$IMPORT_MOUNT" ] && echo "Using host:$IMPORT_LOCAL as a cached copy of $IMPORT_REMOTE..." + if [ -d "$IMPORT_MOUNT" ]; then + echo "Importing directory host:$IMPORT_LOCAL into $ARCHIVE_DIR..." + time nuodocker restore archive --origin-dir $IMPORT_MOUNT --restore-dir $ARCHIVE_DIR --db-name "$DB_NAME" --clean-metadata || exit 98 + elif [ "$importFromCache" = 'true' -a -s "$IMPORT_MOUNT" ]; then + echo "Importing file host:$IMPORT_LOCAL into $ARCHIVE_DIR..." + time tar xf "$IMPORT_MOUNT" --strip-components ${IMPORT_LEVEL} -C "$ARCHIVE_DIR" || exit 98 + else + echo "ERROR: IMPORT_LOCAL has been specified, but host:$IMPORT_LOCAL is not a valid import source - IMPORT_LOCAL must be a directory, an initially empty file, or a cached copy of IMPORT_REMOTE - import aborted..." + exit 98 + fi + fi + + # sanity check the imported content in the archive + [ -d "$ARCHIVE_DIR/full" ] && echo "ERROR: Imported data looks like a BACKUPSET (in which case IMPORT_LOCAL must be a DIRECTORY): $(ls -l $ARCHIVE_DIR | head -n 10)" && exit 98 + [ ! -f "$ARCHIVE_DIR/1.atm" ] && echo "ERROR: Imported archive does not seem to contain valid data: $(ls -l $ARCHIVE_DIR | head -n 10)" && exit 98 + echo "Imported data looks good: $(ls -l $ARCHIVE_DIR | head -n 5)" + + # if the archive was not imported from a dir, then clean the meta-data in the archive + if [ ! -d "$IMPORT_MOUNT" ]; then + nuodocker restore archive --origin-dir "$ARCHIVE_DIR" --restore-dir "$ARCHIVE_DIR" --db-name "$DB_NAME" --clean-metadata || exit 99 + fi +fi diff --git a/nuodb/scripts/remove-zombie b/nuodb/scripts/remove-zombie new file mode 100644 index 0000000..385d5b3 --- /dev/null +++ b/nuodb/scripts/remove-zombie @@ -0,0 +1,25 @@ +#!/bin/sh +# +# remove a zombie of the engine that is trying to start + +# the caller mut specify the hostname +hostType=$1 +hostName=$2 + +# wait until the admin layer has become ready +msg=$(nuocmd check servers --timeout ${STARTUP_TIMEOUT} --check-converged --check-active) +if [ $? -ne 0 ]; then + echo "$me: ERROR: Timed out waiting for admin layer to be ready: $msg" + exit 98 +fi + +myStartIds="$(nuocmd get processes --db-name $DB_NAME | grep 'type=$hostType' | grep 'address=$hostName/' | grep -o 'start-id: [0-9]*' | sed 's/start-id: //' )" + +count=$(echo $myStartIds | wc -l) +echo "$(basename $0): Found $((count - 1)) matching start-ids: $myStartIds" + +for id in $myStartIds ; do + # delete any matching engine processes still in the Raft state + msg="$(nuocmd shutdown process --server-id --start-id $id --evict --timeout 0)" + [ $? -ne 0 ] && echo "ERROR: Unable to remove engine with start-id $id: $msg" +done diff --git a/nuodb/scripts/start-monolith b/nuodb/scripts/start-monolith new file mode 100755 index 0000000..f923651 --- /dev/null +++ b/nuodb/scripts/start-monolith @@ -0,0 +1,63 @@ +#!/bin/sh + +# Start all 3 processes needed for a database in this (single) container. + +PATH=$PATH:/usr/local/scripts + +PEER_ADDRESS=$HOSTNAME + +me="$(basename $0)" + +echo "==================================" + +# start a background nuoadmin process +start-nuoadmin & + +# wait until the admin layer has become ready +msg=$(nuocmd check servers --timeout ${STARTUP_TIMEOUT} --check-converged --check-active) +if [ $? -ne 0 ]; then + echo "$me: ERROR: Timed out waiting for admin layer to be ready: $msg" + exit 98 +fi + +# delete any engine processes still in the Raft state +nuocmd shutdown server-processes --server-id "${PEER_ADDRESS}" --db-name "$DB_NAME" --evict --timeout 0 + +echo "$me: AP is ready - starting SM and TE" + +# start a background nuosm process +start-nuosm & + +# start a background nuote process +start-nuote & + +echo "$me: Waiting for DB $DB_NAME to become RUNNING..." +nuocmd check database --db-name $DB_NAME --check-running --wait-for-acks --timeout "${STARTUP_TIMEOUT}" # wait for RUNNING SM +nuocmd check database --db-name $DB_NAME --check-running --wait-for-acks --timeout 10 # wait for RUNNING SM + all other engines are alive +if [ -n "$(nuocmd get processes --db-name $DB_NAME | grep 'type=TE' | grep 'state=RUNNING')" -a $? = 0 ]; then + echo "$me: Database is RUNNING..." +else + echo "$me: Database check timed out after $STARTUP_TIMEOUT sec" + + echo "$me: $(nuocmd show database --db-name "$DB_NAME" --all-incarnations)" + + if [ -n "$NUODB_DEBUG" ]; then + echo "$me: SM logs" + cat /var/log/nuodb/SM.log + + echo + echo "$me: TE logs" + cat /var/log/nuodb/TE.log + + echo + echo "$me: AP logs" + cat /var/log/nuodb/AP.log + fi +fi + +echo "$me: $(nuocmd show domain)" + +# wait for all child processes to stop +wait + +echo "$me: Database $DB_NAME has been stopped. Exiting." diff --git a/nuodb/scripts/start-nuoadmin b/nuodb/scripts/start-nuoadmin new file mode 100755 index 0000000..eec5a0d --- /dev/null +++ b/nuodb/scripts/start-nuoadmin @@ -0,0 +1,13 @@ +#!/bin/bash +# +# Start a nuoadmin AP process + +: ${LOGDIR:=/var/log/nuodb} +echo "Starting AP..." + +nuoadmin -- \ + pendingProcessTimeout=${STARTUP_TIMEOUT}000 \ + pendingReconnectTimeout=90000 \ + thrift.message.max=1073741824 \ + processLivenessCheckSec=30 \ + 1>/dev/null | tee $LOGDIR/AP.log diff --git a/nuodb/scripts/start-nuosm b/nuodb/scripts/start-nuosm new file mode 100755 index 0000000..57f9a6c --- /dev/null +++ b/nuodb/scripts/start-nuosm @@ -0,0 +1,69 @@ +#!/bin/sh +# +# Start an SM +: ${LOGDIR:=/var/log/nuodb} +PATH=$PATH:/usr/local/scripts + +me="$(basename $0)" + +echo "starting SM..." + +# do not proceed if the admin layer has failed to become ready +# msg=$(nuocmd check servers --timeout ${STARTUP_TIMEOUT} --check-converged --check-active) +# if [ $? -ne 0 ]; then +# echo "$me: ERROR: Timed out waiting for admin layer to be ready: $msg" +# exit 98 +# fi + +# echo "$me: AP is ready..." +# echo "$me: $(nuocmd show domain)" +# echo "$me: PEER_ADDRESS=$PEER_ADDRESS" +# echo "$me: $(nuocmd get servers)" + +# shutdown and delete any zombie engine processes (TE or SM) +# nuocmd shutdown server-processes --server-id "${PEER_ADDRESS}" --db-name "$DB_NAME" --evict --timeout 0 + +# remove any zombie process matching the engine that is about to be started +remove-zombie SM $HOSTNAME + +# set up the archive dir +echo "$me: ARCHIVE_DIR=$ARCHIVE_DIR" + +mkdir -p $ARCHIVE_DIR +[ $? -ne 0 ] && echo "$0: Error creating $ARCHIVE_DIR: $?" && exit 98 + +# check for stranded archive definitions - eg from a previous failed startup +nuocmd show archives --db-name $DB_NAME +runningArchives=$(nuocmd get archives --db-name "$DB_NAME" | grep 'state=RUNNING' | wc -l) +notRunningArchives=$(nuocmd get archives --db-name "DB_NAME" | grep -v "state=RUNNING" | wc -l) +myArchive=$( nuocmd show archives --db-name $DB_NAME --archive-format "archive-id: {id}" | sed -En "/^archive-id: / {N; /$HOSTNAME/ s/^archive-id: ([0-9]+).*$/\1/; T; p}" | head -n 1 ) +[ -z "$myArchive" ] && myArchive=$( nuocmd show archives --db-name $DB_NAME --removed --removed-archive-format "archive-id: {id}" | sed -En "/^archive-id: / {N; /$HOSTNAME/ s/^archive-id: ([0-9]+).*$/\1/; T; p}" | head -n 1 ) +if [ -n "$myArchive" -a $notRunningArchives -gt 1 ] || [ -z "$myArchive" -a $notRunningArchives -gt 0 ]; then + echo "This database has $notRunningArchives archives with no running SM." + echo "No new SMs can start while the database is in this state." + echo "To fix this, you can:" + echo " 1. start SMs running for some or all non-running archives;" + echo " 2. remove all remaining non-running archives with 'nuocmd delete archive (--purge)';" + echo " 3. or delete the entire database with 'compose down'." + echo "After fixing the issue(s) per above, you can run 'compose up' again." + exit 98 +fi + +export myArchive runningArchives notRunningArchives + +# import archive content, if specified +import-archive || exit $? + +# start the SM +nuodocker start sm \ + --servers-ready-timeout $((STARTUP_TIMEOUT / 2)) \ + --database-restore-timeout ${RESTORE_TIMEOUT:-$STARTUP_TIMEOUT} \ + --processes-ready-timeout $((STARTUP_TIMEOUT / 2)) \ + --db-name "$DB_NAME" \ + --server-id "${PEER_ADDRESS}" \ + --dba-user "$DB_USER" \ + --dba-password "$DB_PASSWORD" \ + --database-options "$DB_OPTIONS" \ + --options "node-port 48007" \ + 2>&1 | tee $LOGDIR/SM.log + diff --git a/nuodb/scripts/start-nuote b/nuodb/scripts/start-nuote new file mode 100755 index 0000000..2d06dc5 --- /dev/null +++ b/nuodb/scripts/start-nuote @@ -0,0 +1,37 @@ +#!/bin/sh +# +# start a nuote +: ${LOGDIR:=/var/log/nuodb} +PATH=$PATH:/usr/local/scripts + +me="$(basename $0)" + +echo "Starting TE..." + +# do not proceed if the admin layer has failed to become ready +# msg=$(nuocmd check servers --timeout ${STARTUP_TIMEOUT} --check-converged --check-active) +# if [ $? -ne 0 ]; then +# echo "$me: ERROR: Timed out waiting for admin layer to be ready: $msg" +# exit 98 +# fi + +# echo "$me: AP ready..." + +# remove any zombie process matching the engine that is about to be started +remove-zombie TE $HOSTNAME + +# wait for at least 1 SM to be running +# if [ -n "$(nuocmd check database --db-name $DB_NAME --check-running --wait-for-acks --timeout ${STARTUP_TIMEOUT})" -o $? -ne 0 ]; then +# echo "$me: ERROR: Timed out waiting for database startup ($STARTUP_TIMEOUT sec)..." +# exit 97 +# fi + +nuodocker start te \ + --servers-ready-timeout $((STARTUP_TIMEOUT / 2)) \ + --database-restore-timeout ${RESTORE_TIMEOUT:-$STARTUP_TIMEOUT} \ + --database-created-timeout $((STARTUP_TIMEOUT / 2)) \ + --processes-ready-timeout $((STARTUP_TIMEOUT / 2)) \ + --db-name "$DB_NAME" \ + --server-id "${PEER_ADDRESS}" \ + --options "alt-address ${EXTERNAL_ADDRESS} node-port 48006" \ + 2>&1 | tee $LOGDIR/TE.log diff --git a/nuodb/scripts/stop-nuodb b/nuodb/scripts/stop-nuodb new file mode 100755 index 0000000..5e9f929 --- /dev/null +++ b/nuodb/scripts/stop-nuodb @@ -0,0 +1,30 @@ +#!/bin/sh + +while [ $# -gt 0 ]; +do + opt="$1"; + shift; + case "$opt" in + "--help") + echo "Shutdown the database." + echo "Add the '+domain' flag to shut down the nuoadmin AP as well." + exit 0 + ;; + + "+domain") + echo "Shutting down Domain as well as database." + SHUTDOWN_DOMAIN="true" + ;; + + *) + DBNAME=$opt + ;; + esac +done + +# gracefully shutdown the database +echo "Shutting down DB $DB_NAME..." +nuocmd shutdown database --db-name $DB_NAME || exit $? + +# optionally shut dhown the domain as well +[ -n "$SHUTDOWN_DOMAIN" ] && nuocmd shutdown server --server-id "$PEER_ADDRESS" || exit $?