Add execution and wipe scripts to sandboxes

* Add script 'wipe_and_restart' to single sandboxes
* Add scripts 'exec_all', 'exec_all_masters', 'exec_all_slaves', 'wipe_and_restart_all' to replication sandboxes
* Add subcommand 'exec' to command 'global'.

.build/COMPATIBLE_VERSION

@@ -1 +1 @@
-1.54.0
+1.58.0

.build/VERSION

@@ -1 +1 @@
-1.57.0
+1.58.0

CHANGELOG.md

@@ -1,3 +1,11 @@
+## 1.58.0 12-Dec-2020
+
+### NEW FEATURES
+
+* Add script `wipe_and_restart` to single sandboxes
+* Add scripts `exec_all`, `exec_all_masters`, `exec_all_slaves`, `wipe_and_restart_all` to replication sandboxes
+* Add subcommand `exec` to command `global`.
+
 ## 1.57.0 09-Dec-2020
 
 ### NEW FEATURES

README.md

@@ -2,17 +2,29 @@
 [DBdeployer](https://github.com/datacharmer/dbdeployer) is a tool that deploys MySQL database servers easily.
 This is a port of [MySQL-Sandbox](https://github.com/datacharmer/mysql-sandbox), originally written in Perl, and re-designed from the ground up in [Go](https://golang.org). See the [features comparison](https://github.com/datacharmer/dbdeployer/blob/master/docs/features.md) for more detail.
 
-Documentation updated for version 1.57.0 (09-Dec-2020 18:28 UTC)
+Documentation updated for version 1.58.0 (13-Dec-2020 16:14 UTC)
 
 ![Build Status](https://github.com/datacharmer/dbdeployer/workflows/.github/workflows/all_tests.yml/badge.svg)
 
 - [Installation](https://github.com/datacharmer/dbdeployer/wiki/installation)
 - [Initializing the environment](https://github.com/datacharmer/dbdeployer/wiki/initializing-the-environment)
 - [Updating dbdeployer](https://github.com/datacharmer/dbdeployer/wiki/updating-dbdeployer)
 - [Main operations](https://github.com/datacharmer/dbdeployer/wiki/main-operations)
+ - [Overview ](https://github.com/datacharmer/dbdeployer/wiki/main-operations#overview-)
+ - [Unpack](https://github.com/datacharmer/dbdeployer/wiki/main-operations#unpack)
+ - [Deploy single](https://github.com/datacharmer/dbdeployer/wiki/main-operations#deploy-single)
+ - [Deploy multiple](https://github.com/datacharmer/dbdeployer/wiki/main-operations#deploy-multiple)
+ - [Deploy replication](https://github.com/datacharmer/dbdeployer/wiki/main-operations#deploy-replication)
+ - [Re-deploy a sandbox](https://github.com/datacharmer/dbdeployer/wiki/main-operations#re-deploy-a-sandbox)
 - [Database users](https://github.com/datacharmer/dbdeployer/wiki/database-users)
 - [Database server flavors](https://github.com/datacharmer/dbdeployer/wiki/database-server-flavors)
 - [Getting remote tarballs](https://github.com/datacharmer/dbdeployer/wiki/getting-remote-tarballs)
+ - [Looking at the available tarballs](https://github.com/datacharmer/dbdeployer/wiki/getting-remote-tarballs#looking-at-the-available-tarballs)
+ - [Getting a tarball](https://github.com/datacharmer/dbdeployer/wiki/getting-remote-tarballs#getting-a-tarball)
+ - [Customizing the tarball list](https://github.com/datacharmer/dbdeployer/wiki/getting-remote-tarballs#customizing-the-tarball-list)
+ - [Changing the tarball list permanently](https://github.com/datacharmer/dbdeployer/wiki/getting-remote-tarballs#changing-the-tarball-list-permanently)
+ - [From remote tarball to ready to use in one step](https://github.com/datacharmer/dbdeployer/wiki/getting-remote-tarballs#from-remote-tarball-to-ready-to-use-in-one-step)
+ - [Guessing the latest MySQL version](https://github.com/datacharmer/dbdeployer/wiki/getting-remote-tarballs#guessing-the-latest-mysql-version)
 - [Practical examples](https://github.com/datacharmer/dbdeployer/wiki/practical-examples)
 - [Standard and non-standard basedir names](https://github.com/datacharmer/dbdeployer/wiki/standard-and-non-standard-basedir-names)
 - [Using short version numbers](https://github.com/datacharmer/dbdeployer/wiki/using-short-version-numbers)
@@ -29,6 +41,9 @@ Documentation updated for version 1.57.0 (09-Dec-2020 18:28 UTC)
 - [Sandbox customization](https://github.com/datacharmer/dbdeployer/wiki/sandbox-customization)
 - [Sandbox management](https://github.com/datacharmer/dbdeployer/wiki/sandbox-management)
 - [Sandbox macro operations](https://github.com/datacharmer/dbdeployer/wiki/sandbox-macro-operations)
+ - [dbdeployer global exec](https://github.com/datacharmer/dbdeployer/wiki/sandbox-macro-operations#dbdeployer-global-exec)
+ - [dbdeployer global use](https://github.com/datacharmer/dbdeployer/wiki/sandbox-macro-operations#dbdeployer-global-use)
+- [Sandbox deletion](https://github.com/datacharmer/dbdeployer/wiki/sandbox-deletion)
 - [Default sandbox](https://github.com/datacharmer/dbdeployer/wiki/default-sandbox)
 - [Using the latest sandbox](https://github.com/datacharmer/dbdeployer/wiki/using-the-latest-sandbox)
 - [Sandbox upgrade](https://github.com/datacharmer/dbdeployer/wiki/sandbox-upgrade)
@@ -37,6 +52,11 @@ Documentation updated for version 1.57.0 (09-Dec-2020 18:28 UTC)
 - [Running sysbench](https://github.com/datacharmer/dbdeployer/wiki/running-sysbench)
 - [Obtaining sandbox metadata](https://github.com/datacharmer/dbdeployer/wiki/obtaining-sandbox-metadata)
 - [Replication between sandboxes](https://github.com/datacharmer/dbdeployer/wiki/replication-between-sandboxes)
+ - [a. NDB to NDB](https://github.com/datacharmer/dbdeployer/wiki/replication-between-sandboxes#a.-ndb-to-ndb)
+ - [b. Group replication to group replication. ](https://github.com/datacharmer/dbdeployer/wiki/replication-between-sandboxes#b.-group-replication-to-group-replication.-)
+ - [c. Master/slave to master/slave.](https://github.com/datacharmer/dbdeployer/wiki/replication-between-sandboxes#c.-master/slave-to-master/slave.)
+ - [d. Hibrid replication](https://github.com/datacharmer/dbdeployer/wiki/replication-between-sandboxes#d.-hibrid-replication)
+ - [e. Cloning](https://github.com/datacharmer/dbdeployer/wiki/replication-between-sandboxes#e.-cloning)
 - [Using dbdeployer in scripts](https://github.com/datacharmer/dbdeployer/wiki/using-dbdeployer-in-scripts)
 - [Importing databases into sandboxes](https://github.com/datacharmer/dbdeployer/wiki/importing-databases-into-sandboxes)
 - [Cloning databases](https://github.com/datacharmer/dbdeployer/wiki/cloning-databases)

cmd/export_test.go

@@ -229,7 +229,7 @@ func TestExportImport(t *testing.T) {
  subCommandName: "",
  expectedName: "global",
  expectedAncestors: 2,
- expectedSubCommands: 8,
+ expectedSubCommands: 9,
  expectedArgument: "",
  },
  {

cmd/global.go

@@ -17,6 +17,7 @@ package cmd
 
 import (
  "fmt"
+ "os"
  "path"
  "regexp"
  "strings"
@@ -165,6 +166,34 @@ func globalRunCommand(cmd *cobra.Command, executable string, args []string, requ
  }
  }
 
+ if executable == "exec" {
+ if dryRun {
+ fmt.Printf("%v\n", args)
+ } else {
+ currentDir := os.Getenv("PWD")
+ err = os.Chdir(fullDirPath)
+ if err != nil {
+ fmt.Printf("error changing directory to %s:%s\n", fullDirPath, err)
+ }
+ cmd := args[0]
+ var cmdArgs []string
+ //if (cmd == "bash" || cmd == "sh") && len(args) >1 {
+ // cmdArgs = append(cmdArgs, "-c")
+ //}
+ for N := 1; N < len(args); N++ {
+ cmdArgs = append(cmdArgs, args[N])
+ }
+ fmt.Printf("# %s\n", fullDirPath)
+ if len(cmdArgs) > 0 {
+ _, err = common.RunCmdWithArgs(cmd, cmdArgs)
+ } else {
+ _, err = common.RunCmd(cmd)
+ }
+ common.ErrCheckExitf(err, 1, "error while running %s\n", cmd)
+ _ = os.Chdir(currentDir)
+ }
+ continue
+ }
  cmdFile := path.Join(fullDirPath, executable)
  realExecutable := executable
  if !common.ExecExists(cmdFile) {
@@ -235,6 +264,10 @@ func useAllSandboxes(cmd *cobra.Command, args []string) {
  globalRunCommand(cmd, globals.ScriptUse, args, true, false)
 }
 
+func execAllSandboxes(cmd *cobra.Command, args []string) {
+ globalRunCommand(cmd, "exec", args, true, false)
+}
+
 func metadataAllSandboxes(cmd *cobra.Command, args []string) {
  globalRunCommand(cmd, globals.ScriptMetadata, args, true, false)
 }
@@ -312,6 +345,31 @@ For example, a query using @@port won't run in MySQL 5.0.x`,
  Annotations: map[string]string{"export": ExportAnnotationToJson(StringExport)},
  }
 
+ globalExecCmd = &cobra.Command{
+ Use: "exec {command}",
+ Short: "Runs a command in all sandboxes",
+ Long: `Runs a command in all sandboxes.
+The command will be executed inside the sandbox. Thus, you can reference a file that you know should be there.
+There is no check to ensure that a give command is doable.
+For example: the command "cat filename" will result in an error if filename is not present in the sandbox directory.
+You can run complex shell commands by prepending them with either "bash -- -c" or "sh -- -c". Such command must be quoted. 
+Only one command can be passed, although you can use a shell command as described above to overcome this limitation.
+IMPORTANT: if your command argument contains flags, you must use a double dash (--) before any of the flags.
+
+You may combine the command passed to "exec" with other drill-down commands in the sandbox directory, 
+such as "./exec_all". In this case, you need to make sure that all your sandbox contain the command, or use "--type"
+to only run "exec" in the specific topologies.
+`,
+ Example: `$ dbdeployer global exec grep 'version\|type' sbdescription.json
+ $ dbdeployer global exec grep -- -w basedir sbdescription.json
+ $ dbdeployer global exec pwd
+ $ dbdeployer global exec shell -- -c "if [ -f filename ] ; then echo 'found filename'; fi"
+ $ dbdeployer global exec --type=group-multi-primary ./exec_all ls -- -l data/ibdata1
+`,
+ Run: execAllSandboxes,
+ Annotations: map[string]string{"export": ExportAnnotationToJson(StringExport)},
+ }
+
  globalMetadataCmd = &cobra.Command{
  Use: "metadata {keyword}",
  Short: "Runs a metadata query in all sandboxes",
@@ -332,6 +390,7 @@ func init() {
  globalCmd.AddCommand(globalTestCmd)
  globalCmd.AddCommand(globalTestReplicationCmd)
  globalCmd.AddCommand(globalUseCmd)
+ globalCmd.AddCommand(globalExecCmd)
  globalCmd.AddCommand(globalMetadataCmd)
 
  setPflag(globalCmd, globals.VersionLabel, "", "", "", "Runs command only in sandboxes of the given version", false)

common/version.go

@@ -16,11 +16,11 @@
 package common
 
 // This file was generated during build. Do not edit.
-// Build time: 2020-12-08 19:44
+// Build time: 2020-12-11 20:55
 
-var VersionDef string = "1.57.0" // 2020-12-08
+var VersionDef string = "1.58.0" // 2020-12-11
 
 // Compatible version is the version used to mark compatible archives (templates, configuration).
 // It is usually major.minor.0, except when we are at version 0.x, when
 // every revision may bring incompatibility
-var CompatibleVersion string = "1.54.0" // 2020-09-13
+var CompatibleVersion string = "1.58.0" // 2020-12-11