diff --git a/.build/COMPATIBLE_VERSION b/.build/COMPATIBLE_VERSION index 5e57fb89..034552a8 100644 --- a/.build/COMPATIBLE_VERSION +++ b/.build/COMPATIBLE_VERSION @@ -1 +1 @@ -1.29.0 +1.30.0 diff --git a/.build/VERSION b/.build/VERSION index 5e57fb89..034552a8 100644 --- a/.build/VERSION +++ b/.build/VERSION @@ -1 +1 @@ -1.29.0 +1.30.0 diff --git a/Changelog b/Changelog index 00aadd27..71658798 100644 --- a/Changelog +++ b/Changelog @@ -1,3 +1,17 @@ +1.30.0 05-May-2019 + BUGS FIXED + - Fix Issue #72 Catalog operations can clash with multiple dbdeployer runs + - Fix wrong description for "defaults update" + - Fix wrong description for "cookbook create" + - Fix option --sort-by incorrectly assigned to command "cookbook" instead + of subcommand "cookbook list" + NEW FEATURES + - Add command "dbdeployer info" to improve dbdeployer usage in scripts + TESTING + - Add test for parallel usage + - Reinstate parallel deployment in functional test + - Improve replication between sandboxes test, by checking + for existing versions better. 1.29.0 30-Apr-2019 ADJUSTMENTS - Change upgrade procedure for MySQL 8.0.16+ diff --git a/README.md b/README.md index e3bbacc7..158a094e 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ [DBdeployer](https://github.com/datacharmer/dbdeployer) is a tool that deploys MySQL database servers easily. This is a port of [MySQL-Sandbox](https://github.com/datacharmer/mysql-sandbox), originally written in Perl, and re-designed from the ground up in [Go](https://golang.org). See the [features comparison](https://github.com/datacharmer/dbdeployer/blob/master/docs/features.md) for more detail. -Documentation updated for version 1.29.0 (30-Apr-2019 17:41 UTC) +Documentation updated for version 1.30.0 (05-May-2019 07:21 UTC) [![Build Status](https://travis-ci.org/datacharmer/dbdeployer.svg "Travis CI status")](https://travis-ci.org/datacharmer/dbdeployer) @@ -11,7 +11,7 @@ Documentation updated for version 1.29.0 (30-Apr-2019 17:41 UTC) - [Main operations](#Main-operations) - [Database server flavors](#Database-server-flavors) - [Getting remote tarballs](#Getting-remote-tarballs) -- [Practical examples](#Practical-examples) +- [Practical examples (cookbook)](#Practical-examples) - [Standard and non-standard basedir names](#Standard-and-non-standard-basedir-names) - [Using short version numbers](#Using-short-version-numbers) - [Multiple sandboxes, same version and type](#Multiple-sandboxes-same-version-and-type) @@ -31,6 +31,7 @@ Documentation updated for version 1.29.0 (30-Apr-2019 17:41 UTC) - [Dedicated admin address](#Dedicated-admin-address) - [Obtaining sandbox metadata](#Obtaining-sandbox-metadata) - [Replication between sandboxes](#Replication-between-sandboxes) +- [Using dbdeployer in scripts](#Using-dbdeployer-in-scripts) - [Compiling dbdeployer](#Compiling-dbdeployer) - [Generating additional documentation](#Generating-additional-documentation) - [Command line completion](#Command-line-completion) @@ -48,7 +49,7 @@ Get the one for your O.S. from [dbdeployer releases](https://github.com/datachar For example: - $ VERSION=1.29.0 + $ VERSION=1.30.0 $ OS=linux $ origin=https://github.com/datacharmer/dbdeployer/releases/download/v$VERSION $ wget $origin/dbdeployer-$VERSION.$OS.tar.gz @@ -83,7 +84,7 @@ For example: The program doesn't have any dependencies. Everything is included in the binary. Calling *dbdeployer* without arguments or with ``--help`` will show the main help screen. $ dbdeployer --version - dbdeployer version 1.29.0 + dbdeployer version 1.30.0 $ dbdeployer -h @@ -103,6 +104,7 @@ The program doesn't have any dependencies. Everything is included in the binary. export Exports the command structure in JSON format global Runs a given command in every sandbox help Help about any command + info Shows information about dbdeployer environment samples remote Manages remote tarballs sandboxes List installed sandboxes unpack unpack a tarball into the binary directory @@ -497,9 +499,8 @@ dbdeployer will detect the latest versions available in you system. If you don't show Shows the contents of a given recipe Flags: - --flavor string For which flavor this recipe is - -h, --help help for cookbook - --sort-by string Sort order for the list (name, flavor, script) (default "name") + --flavor string For which flavor this recipe is + -h, --help help for cookbook @@ -1371,6 +1372,102 @@ Examples: ~/sandboxes/group_8_0_15_2/replicate_from ms_8_0_15_1 ``` +# Using dbdeployer in scripts + +dbdeployer has been designed to simplify automated operations. Using it in scripts is easy, as shown in the [cookbook examples](#Practical-examples). +In addition to run operations on sandboxes, dbdeployer can also provide information about the environment in a way that is suitable for scripting. + +For example, if you want to deploy a sandbox using the most recent 5.7 binaries, you may run `dbdeployer versions`, look which versions are available, and pick the most recent one. But dbdeployer 1.30.0 can aytomate this procedure using `dbdeployer info version 5.7`. This command will print the latest 5.7 binaries to the standard output, allowing us to create dynamic scripts such as: + +```bash +# the absolute latest version +latest=$(dbdeployer info version) +latest57=$(dbdeployer info version 5.7) +latest80=$(dbdeployer info version 8.0) + +if [ -z "$latest" ] +then + echo "No versions found" + exit 1 +fi + +echo "The latest version is $latest" + +if [ -n "$latest57" ] +then + echo "# latest for 5.7 : $latest57" + dbdeployer deploy single $latest57 +fi + +if [ -n "$latest80" ] +then + echo "# latest for 8.0 : $latest80" + dbdeployer deploy single $latest80 +fi +``` + + $ dbdeployer info version -h + Displays the latest version available for deployment. + If a short version is indicated (such as 5.7, or 8.0), only the versions belonging to that short + version are searched. + If "all" is indicated after the short version, displays all versions belonging to that short version. + + Usage: + dbdeployer info version [short-version|all] [all] [flags] + + Examples: + + # Shows the latest version available + $ dbdeployer info version + 8.0.16 + + # shows the latest version belonging to 5.7 + $ dbdeployer info version 5.7 + 5.7.26 + + # shows the latest version for every short version + $ dbdeployer info version all + 5.0.96 5.1.73 5.5.53 5.6.41 5.7.26 8.0.16 + + # shows all the versions for a given short version + $ dbdeployer info version 8.0 all + 8.0.11 8.0.12 8.0.13 8.0.14 8.0.15 8.0.16 + + + Flags: + -h, --help help for version + + + +Similarly to `versions`, the `defaults` subcommand allows us to get dbdeployer metadata in a way that can be used in scripts + + $ dbdeployer info defaults -h + Displays one field of the defaults. + + Usage: + dbdeployer info defaults field-name [flags] + + Examples: + + $ dbdeployer info defaults master-slave-base-port + + + Flags: + -h, --help help for defaults + + + +For example + +``` +$ dbdeployer info defaults sandbox-prefix +msb_ + +$ dbdeployer info defaults master-slave-ptrefix +rsandbox_ +``` +You can ask for any fields from the defaults (see `dbdeployer defaults list` for the field names). + # Compiling dbdeployer Should you need to compile your own binaries for dbdeployer, follow these steps: @@ -1378,18 +1475,18 @@ Should you need to compile your own binaries for dbdeployer, follow these steps: 1. Make sure you have go 1.10+ installed in your system, and that the ``$GOPATH`` variable is set. 2. Run ``go get -u github.com/datacharmer/dbdeployer``. This will import all the code that is needed to build dbdeployer. 3. Change directory to ``$GOPATH/src/github.com/datacharmer/dbdeployer``. -4. Run ``./scripts/build.sh {linux|OSX} 1.29.0`` -5. If you need the docs enabled binaries (see the section "Generating additional documentation") run ``MKDOCS=1 ./scripts/build.sh {linux|OSX} 1.29.0`` +4. Run ``./scripts/build.sh {linux|OSX} 1.30.0`` +5. If you need the docs enabled binaries (see the section "Generating additional documentation") run ``MKDOCS=1 ./scripts/build.sh {linux|OSX} 1.30.0`` # Generating additional documentation Between this file and [the API API list](https://github.com/datacharmer/dbdeployer/blob/master/docs/API/API-1.1.md), you have all the existing documentation for dbdeployer. Should you need additional formats, though, dbdeployer is able to generate them on-the-fly. Tou will need the docs-enabled binaries: in the distribution list, you will find: -* dbdeployer-1.29.0-docs.linux.tar.gz -* dbdeployer-1.29.0-docs.osx.tar.gz -* dbdeployer-1.29.0.linux.tar.gz -* dbdeployer-1.29.0.osx.tar.gz +* dbdeployer-1.30.0-docs.linux.tar.gz +* dbdeployer-1.30.0-docs.osx.tar.gz +* dbdeployer-1.30.0.linux.tar.gz +* dbdeployer-1.30.0.osx.tar.gz The executables containing ``-docs`` in their name have the same capabilities of the regular ones, but in addition they can run the *hidden* command ``tree``, with alias ``docs``. diff --git a/cmd/delete_binaries.go b/cmd/delete_binaries.go index c75bcc1a..9dacb805 100644 --- a/cmd/delete_binaries.go +++ b/cmd/delete_binaries.go @@ -29,8 +29,9 @@ import ( func sandboxesUsingBinariesDir(basedir, binariesDir string) []string { var sandboxes []string - - sandboxList, err := defaults.ReadCatalog() + var sandboxList defaults.SandboxCatalog + var err error + sandboxList, err = defaults.ReadCatalog() common.ErrCheckExitf(err, 1, "error getting sandboxes from catalog: %s", err) fullPath := path.Join(basedir, binariesDir) for _, sb := range sandboxList { diff --git a/cmd/info.go b/cmd/info.go new file mode 100644 index 00000000..5e85f368 --- /dev/null +++ b/cmd/info.go @@ -0,0 +1,167 @@ +// DBDeployer - The MySQL Sandbox +// Copyright © 2006-2018 Giuseppe Maxia +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package cmd + +import ( + "fmt" + "regexp" + "strings" + + "github.com/spf13/cobra" + + "github.com/datacharmer/dbdeployer/common" + "github.com/datacharmer/dbdeployer/cookbook" + "github.com/datacharmer/dbdeployer/defaults" + "github.com/datacharmer/dbdeployer/globals" +) + +func displayDefaults(cmd *cobra.Command, args []string) { + if len(args) < 1 { + common.Exit(1, + "'defaults' requires a label", + "Example: dbdeployer info defaults master-slave-base-port") + } + label := args[0] + defaultsMap := defaults.DefaultsToMap() + value, ok := defaultsMap[label] + if ok { + fmt.Println(value) + } else { + fmt.Printf("# ERROR: field %s not found in defaults\n", label) + } +} + +func displayAllVersions(basedir, wantedVersion, flavor string) { + result := "" + var versionInfoList []common.VersionInfo = common.GetVersionInfoFromDir(basedir) + for _, verInfo := range versionInfoList { + versionList, err := common.VersionToList(verInfo.Version) + if err != nil { + common.Exitf(1, "error retrieving version list from %s", verInfo.Version) + } + shortVersion := fmt.Sprintf("%d.%d", versionList[0], versionList[1]) + if wantedVersion == shortVersion || strings.ToLower(wantedVersion) == "all" { + if verInfo.Flavor == flavor { + if result != "" { + result += " " + } + result += verInfo.Version + } + } + } + if result != "" { + fmt.Println(result) + } +} + +func displayVersion(cmd *cobra.Command, args []string) { + wantedVersion := "" + allVersions := "" + if len(args) > 0 { + wantedVersion = args[0] + } + + reNotFound := regexp.MustCompile(cookbook.VersionNotFound) + if len(args) > 1 { + allVersions = args[1] + } + flavor, _ := cmd.Flags().GetString(globals.FlavorLabel) + if flavor == "" { + flavor = common.MySQLFlavor + } + if strings.ToLower(allVersions) == "all" { + basedir, err := getAbsolutePathFromFlag(cmd, "sandbox-binary") + common.ErrCheckExitf(err, 1, "error getting absolute path for 'sandbox-binary'") + displayAllVersions(basedir, wantedVersion, flavor) + } else { + if strings.ToLower(wantedVersion) == "all" { + result := "" + for _, v := range globals.SupportedAllVersions { + latest := cookbook.GetLatestVersion(v, flavor) + if !reNotFound.MatchString(latest) { + if result != "" { + result += " " + } + result += latest + } + } + if result != "" { + fmt.Println(result) + } + } else { + latest := cookbook.GetLatestVersion(wantedVersion, flavor) + if !reNotFound.MatchString(latest) { + fmt.Println(latest) + + } + } + } +} + +var infoCmd = &cobra.Command{ + Use: "info", + Short: "Shows information about dbdeployer environment samples", + Long: `Shows current information about defaults and environment.`, +} + +var infoDefaultsCmd = &cobra.Command{ + Use: "defaults field-name", + + Short: "displays a defaults value", + Example: ` + $ dbdeployer info defaults master-slave-base-port +`, + Long: `Displays one field of the defaults.`, + Run: displayDefaults, + Annotations: map[string]string{"export": ExportAnnotationToJson(StringExport)}, +} + +var infoVersionCmd = &cobra.Command{ + Use: "version [short-version|all] [all]", + + Short: "displays the latest version available", + Example: ` + # Shows the latest version available + $ dbdeployer info version + 8.0.16 + + # shows the latest version belonging to 5.7 + $ dbdeployer info version 5.7 + 5.7.26 + + # shows the latest version for every short version + $ dbdeployer info version all + 5.0.96 5.1.73 5.5.53 5.6.41 5.7.26 8.0.16 + + # shows all the versions for a given short version + $ dbdeployer info version 8.0 all + 8.0.11 8.0.12 8.0.13 8.0.14 8.0.15 8.0.16 +`, + Long: `Displays the latest version available for deployment. +If a short version is indicated (such as 5.7, or 8.0), only the versions belonging to that short +version are searched. +If "all" is indicated after the short version, displays all versions belonging to that short version. +`, + Run: displayVersion, + Annotations: map[string]string{"export": ExportAnnotationToJson(StringExport)}, +} + +func init() { + rootCmd.AddCommand(infoCmd) + infoCmd.AddCommand(infoDefaultsCmd) + infoCmd.AddCommand(infoVersionCmd) + setPflag(infoCmd, globals.FlavorLabel, "", "", "", "For which flavor this info is", false) +} diff --git a/cmd/sandboxes.go b/cmd/sandboxes.go index 8ef99e58..ead200e9 100644 --- a/cmd/sandboxes.go +++ b/cmd/sandboxes.go @@ -20,15 +20,18 @@ import ( "path" "github.com/alexeyco/simpletable" + "github.com/dustin/go-humanize/english" + "github.com/spf13/cobra" + "github.com/datacharmer/dbdeployer/common" "github.com/datacharmer/dbdeployer/defaults" "github.com/datacharmer/dbdeployer/globals" - "github.com/dustin/go-humanize/english" - "github.com/spf13/cobra" ) func showSandboxesFromCatalog(currentSandboxHome string, useFlavor, useHeader, useTable bool) { - sandboxList, err := defaults.ReadCatalog() + var sandboxList defaults.SandboxCatalog + var err error + sandboxList, err = defaults.ReadCatalog() common.ErrCheckExitf(err, 1, "error getting sandboxes from catalog: %s", err) if len(sandboxList) == 0 { diff --git a/common/version.go b/common/version.go index 2cc45d82..9a582f43 100644 --- a/common/version.go +++ b/common/version.go @@ -16,11 +16,11 @@ package common // This file was generated during build. Do not edit. -// Build time: 2019-04-30 19:38 +// Build time: 2019-05-05 05:18 -var VersionDef string = "1.29.0" // 2019-04-30 +var VersionDef string = "1.30.0" // 2019-05-05 // Compatible version is the version used to mark compatible archives (templates, configuration). // It is usually major.minor.0, except when we are at version 0.x, when // every revision may bring incompatibility -var CompatibleVersion string = "1.29.0" // 2019-04-30 +var CompatibleVersion string = "1.30.0" // 2019-05-05 diff --git a/cookbook/recipes.go b/cookbook/recipes.go index 27133f88..e210c62f 100644 --- a/cookbook/recipes.go +++ b/cookbook/recipes.go @@ -23,6 +23,7 @@ import ( "strings" "github.com/alexeyco/simpletable" + "github.com/datacharmer/dbdeployer/common" "github.com/datacharmer/dbdeployer/defaults" "github.com/datacharmer/dbdeployer/globals" @@ -260,7 +261,7 @@ func GetRecipe(recipeName, flavor string) (string, int, error) { return text, ErrNoRecipeFound, fmt.Errorf("recipe %s not found", recipeName) } latestVersions := make(map[string]string) - for _, version := range []string{"5.0", "5.1", "5.5", "5.6", "5.7", "8.0"} { + for _, version := range globals.SupportedMySQLVersions { latest := GetLatestVersion(version, common.MySQLFlavor) if latest != "" { latestVersions[version] = latest @@ -273,11 +274,10 @@ func GetRecipe(recipeName, flavor string) (string, int, error) { if latestVersion == VersionNotFound { versionCode = ErrNoVersionFound } - var data = common.StringMap{ - "Copyright": globals.Copyright, - "TemplateName": recipeName, - "LatestVersion": latestVersion, - } + var data = defaults.DefaultsToMap() + data["Copyright"] = globals.Copyright + data["TemplateName"] = recipeName + data["LatestVersion"] = latestVersion for version, latest := range latestVersions { reDot := regexp.MustCompile(`\.`) versionName := reDot.ReplaceAllString(version, "_") diff --git a/cookbook/recipes_test.go b/cookbook/recipes_test.go index a19c5858..472c8f79 100644 --- a/cookbook/recipes_test.go +++ b/cookbook/recipes_test.go @@ -18,6 +18,7 @@ import ( "fmt" "github.com/datacharmer/dbdeployer/common" "github.com/datacharmer/dbdeployer/compare" + "github.com/datacharmer/dbdeployer/defaults" "github.com/datacharmer/dbdeployer/sandbox" "testing" ) @@ -48,6 +49,14 @@ func TestGetLatestVersion(t *testing.T) { recipe, _, _ = GetRecipe("single", common.MySQLFlavor) compare.OkMatchesString("recipe single", recipe, version, t) } + dummyRecipe := RecipeTemplate{ + Contents: `sandbox prefix ="{{.SandboxPrefix}}" - replication prefix = "{{.MasterSlavePrefix}}" <{{.PxcBasePort}}>`, + } + RecipesList["dummy"] = dummyRecipe + + recipeText, _, _ := GetRecipe("dummy", common.MySQLFlavor) + compare.OkMatchesString("dummy recipe", recipeText, defaults.Defaults().SandboxPrefix, t) + compare.OkMatchesString("dummy recipe", recipeText, defaults.Defaults().MasterSlavePrefix, t) err = sandbox.RemoveMockEnvironment(sandbox.DefaultMockDir) compare.OkIsNil("removing mock environment", err, t) diff --git a/defaults/defaults.go b/defaults/defaults.go index 5e349caa..a8598832 100644 --- a/defaults/defaults.go +++ b/defaults/defaults.go @@ -27,47 +27,43 @@ import ( ) type DbdeployerDefaults struct { - Version string `json:"version"` - SandboxHome string `json:"sandbox-home"` - SandboxBinary string `json:"sandbox-binary"` - UseSandboxCatalog bool `json:"use-sandbox-catalog"` - LogSBOperations bool `json:"log-sb-operations"` - LogDirectory string `json:"log-directory"` - CookbookDirectory string `json:"cookbook-directory"` - - //UseConcurrency bool `json:"use-concurrency"` - MasterSlaveBasePort int `json:"master-slave-base-port"` - GroupReplicationBasePort int `json:"group-replication-base-port"` - GroupReplicationSpBasePort int `json:"group-replication-sp-base-port"` - FanInReplicationBasePort int `json:"fan-in-replication-base-port"` - AllMastersReplicationBasePort int `json:"all-masters-replication-base-port"` - MultipleBasePort int `json:"multiple-base-port"` - // GaleraBasePort int `json:"galera-base-port"` - PxcBasePort int `json:"pxc-base-port"` - NdbBasePort int `json:"ndb-base-port"` - NdbClusterPort int `json:"ndb-cluster-port"` - GroupPortDelta int `json:"group-port-delta"` - MysqlXPortDelta int `json:"mysqlx-port-delta"` - AdminPortDelta int `json:"admin-port-delta"` - MasterName string `json:"master-name"` - MasterAbbr string `json:"master-abbr"` - NodePrefix string `json:"node-prefix"` - SlavePrefix string `json:"slave-prefix"` - SlaveAbbr string `json:"slave-abbr"` - SandboxPrefix string `json:"sandbox-prefix"` - MasterSlavePrefix string `json:"master-slave-prefix"` - GroupPrefix string `json:"group-prefix"` - GroupSpPrefix string `json:"group-sp-prefix"` - MultiplePrefix string `json:"multiple-prefix"` - FanInPrefix string `json:"fan-in-prefix"` - AllMastersPrefix string `json:"all-masters-prefix"` - ReservedPorts []int `json:"reserved-ports"` - RemoteRepository string `json:"remote-repository"` - RemoteIndexFile string `json:"remote-index-file"` - // GaleraPrefix string `json:"galera-prefix"` - PxcPrefix string `json:"pxc-prefix"` - NdbPrefix string `json:"ndb-prefix"` - Timestamp string `json:"timestamp"` + Version string `json:"version"` + SandboxHome string `json:"sandbox-home"` + SandboxBinary string `json:"sandbox-binary"` + UseSandboxCatalog bool `json:"use-sandbox-catalog"` + LogSBOperations bool `json:"log-sb-operations"` + LogDirectory string `json:"log-directory"` + CookbookDirectory string `json:"cookbook-directory"` + MasterSlaveBasePort int `json:"master-slave-base-port"` + GroupReplicationBasePort int `json:"group-replication-base-port"` + GroupReplicationSpBasePort int `json:"group-replication-sp-base-port"` + FanInReplicationBasePort int `json:"fan-in-replication-base-port"` + AllMastersReplicationBasePort int `json:"all-masters-replication-base-port"` + MultipleBasePort int `json:"multiple-base-port"` + PxcBasePort int `json:"pxc-base-port"` + NdbBasePort int `json:"ndb-base-port"` + NdbClusterPort int `json:"ndb-cluster-port"` + GroupPortDelta int `json:"group-port-delta"` + MysqlXPortDelta int `json:"mysqlx-port-delta"` + AdminPortDelta int `json:"admin-port-delta"` + MasterName string `json:"master-name"` + MasterAbbr string `json:"master-abbr"` + NodePrefix string `json:"node-prefix"` + SlavePrefix string `json:"slave-prefix"` + SlaveAbbr string `json:"slave-abbr"` + SandboxPrefix string `json:"sandbox-prefix"` + MasterSlavePrefix string `json:"master-slave-prefix"` + GroupPrefix string `json:"group-prefix"` + GroupSpPrefix string `json:"group-sp-prefix"` + MultiplePrefix string `json:"multiple-prefix"` + FanInPrefix string `json:"fan-in-prefix"` + AllMastersPrefix string `json:"all-masters-prefix"` + ReservedPorts []int `json:"reserved-ports"` + RemoteRepository string `json:"remote-repository"` + RemoteIndexFile string `json:"remote-index-file"` + PxcPrefix string `json:"pxc-prefix"` + NdbPrefix string `json:"ndb-prefix"` + Timestamp string `json:"timestamp"` } const ( @@ -85,19 +81,17 @@ var ( ConfigurationFile string = path.Join(ConfigurationDir, ConfigurationFileName) CustomConfigurationFile string = "" SandboxRegistry string = path.Join(ConfigurationDir, SandboxRegistryName) - SandboxRegistryLock string = path.Join(ConfigurationDir, SandboxRegistryLockName) + SandboxRegistryLock string = path.Join(common.GlobalTempDir(), SandboxRegistryLockName) LogSBOperations bool = common.IsEnvSet("DBDEPLOYER_LOGGING") factoryDefaults = DbdeployerDefaults{ - Version: common.CompatibleVersion, - SandboxHome: path.Join(homeDir, "sandboxes"), - SandboxBinary: path.Join(homeDir, "opt", "mysql"), - - UseSandboxCatalog: true, - LogSBOperations: false, - LogDirectory: path.Join(homeDir, "sandboxes", "logs"), - CookbookDirectory: "recipes", - //UseConcurrency : true, + Version: common.CompatibleVersion, + SandboxHome: path.Join(homeDir, "sandboxes"), + SandboxBinary: path.Join(homeDir, "opt", "mysql"), + UseSandboxCatalog: true, + LogSBOperations: false, + LogDirectory: path.Join(homeDir, "sandboxes", "logs"), + CookbookDirectory: "recipes", MasterSlaveBasePort: 11000, GroupReplicationBasePort: 12000, GroupReplicationSpBasePort: 13000, @@ -105,24 +99,23 @@ var ( AllMastersReplicationBasePort: 15000, MultipleBasePort: 16000, PxcBasePort: 18000, - // GaleraBasePort: 17000, - NdbBasePort: 19000, - NdbClusterPort: 20000, - GroupPortDelta: 125, - MysqlXPortDelta: 10000, - AdminPortDelta: 11000, - MasterName: "master", - MasterAbbr: "m", - NodePrefix: "node", - SlavePrefix: "slave", - SlaveAbbr: "s", - SandboxPrefix: "msb_", - MasterSlavePrefix: "rsandbox_", - GroupPrefix: "group_msb_", - GroupSpPrefix: "group_sp_msb_", - MultiplePrefix: "multi_msb_", - FanInPrefix: "fan_in_msb_", - AllMastersPrefix: "all_masters_msb_", + NdbBasePort: 19000, + NdbClusterPort: 20000, + GroupPortDelta: 125, + MysqlXPortDelta: 10000, + AdminPortDelta: 11000, + MasterName: "master", + MasterAbbr: "m", + NodePrefix: "node", + SlavePrefix: "slave", + SlaveAbbr: "s", + SandboxPrefix: "msb_", + MasterSlavePrefix: "rsandbox_", + GroupPrefix: "group_msb_", + GroupSpPrefix: "group_sp_msb_", + MultiplePrefix: "multi_msb_", + FanInPrefix: "fan_in_msb_", + AllMastersPrefix: "all_masters_msb_", ReservedPorts: []int{ 1186, // MySQL Cluster 3306, // MySQL Server regular port @@ -131,10 +124,9 @@ var ( }, RemoteRepository: "https://raw.githubusercontent.com/datacharmer/mysql-docker-minimal/master/dbdata", RemoteIndexFile: "available.json", - // GaleraPrefix: "galera_msb_", - NdbPrefix: "ndb_msb_", - PxcPrefix: "pxc_msb_", - Timestamp: time.Now().Format(time.UnixDate), + NdbPrefix: "ndb_msb_", + PxcPrefix: "pxc_msb_", + Timestamp: time.Now().Format(time.UnixDate), } currentDefaults DbdeployerDefaults ) @@ -219,7 +211,6 @@ func ValidateDefaults(nd DbdeployerDefaults) bool { checkInt("multiple-base-port", nd.MultipleBasePort, minPortValue, maxPortValue) && checkInt("fan-in-base-port", nd.FanInReplicationBasePort, minPortValue, maxPortValue) && checkInt("all-masters-base-port", nd.AllMastersReplicationBasePort, minPortValue, maxPortValue) && - // checkInt("galera-base-port", nd.GaleraBasePort, minPortValue, maxPortValue) && checkInt("pxc-base-port", nd.PxcBasePort, minPortValue, maxPortValue) && checkInt("ndb-base-port", nd.NdbBasePort, minPortValue, maxPortValue) && checkInt("ndb-cluster-port", nd.NdbClusterPort, minPortValue, maxPortValue) && @@ -236,7 +227,6 @@ func ValidateDefaults(nd DbdeployerDefaults) bool { nd.MultipleBasePort != nd.AllMastersReplicationBasePort && nd.MultipleBasePort != nd.NdbBasePort && nd.MultipleBasePort != nd.NdbClusterPort && - // nd.MultipleBasePort != nd.GaleraBasePort && nd.MultipleBasePort != nd.PxcBasePort && nd.MultiplePrefix != nd.GroupSpPrefix && nd.MultiplePrefix != nd.GroupPrefix && @@ -246,7 +236,6 @@ func ValidateDefaults(nd DbdeployerDefaults) bool { nd.MultiplePrefix != nd.AllMastersPrefix && nd.MasterAbbr != nd.SlaveAbbr && nd.MultiplePrefix != nd.NdbPrefix && - // nd.MultiplePrefix != nd.GaleraPrefix && nd.MultiplePrefix != nd.PxcPrefix && nd.SandboxHome != nd.SandboxBinary if !noConflicts { @@ -265,7 +254,6 @@ func ValidateDefaults(nd DbdeployerDefaults) bool { nd.GroupSpPrefix != "" && nd.MultiplePrefix != "" && nd.PxcPrefix != "" && - // nd.GaleraPrefix != "" && nd.NdbPrefix != "" && nd.SandboxHome != "" && nd.SandboxBinary != "" && @@ -328,8 +316,6 @@ func UpdateDefaults(label, value string, storeDefaults bool) { newDefaults.LogDirectory = value case "cookbook-directory": newDefaults.CookbookDirectory = value - //case "use-concurrency": - // new_defaults.UseConcurrency = common.TextToBool(value) case "master-slave-base-port": newDefaults.MasterSlaveBasePort = common.Atoi(value) case "group-replication-base-port": @@ -346,8 +332,6 @@ func UpdateDefaults(label, value string, storeDefaults bool) { newDefaults.NdbBasePort = common.Atoi(value) case "ndb-cluster-port": newDefaults.NdbClusterPort = common.Atoi(value) - // case "galera-base-port": - // new_defaults.GaleraBasePort = common.Atoi(value) case "pxc-base-port": newDefaults.PxcBasePort = common.Atoi(value) case "group-port-delta": @@ -386,8 +370,6 @@ func UpdateDefaults(label, value string, storeDefaults bool) { newDefaults.RemoteIndexFile = value case "reserved-ports": newDefaults.ReservedPorts = strToSlice("reserved-ports", value) - // case "galera-prefix": - // new_defaults.GaleraPrefix = value case "pxc-prefix": newDefaults.PxcPrefix = value case "ndb-prefix": @@ -423,3 +405,85 @@ func LoadConfiguration() { time.Sleep(1000 * time.Millisecond) } } + +// Converts the defaults to a string map, +// useful to access single values from other operations +func DefaultsToMap() common.StringMap { + currentDefaults = Defaults() + return common.StringMap{ + "Version": currentDefaults.Version, + "version": currentDefaults.Version, + "SandboxHome": currentDefaults.SandboxHome, + "sandbox-home": currentDefaults.SandboxHome, + "SandboxBinary": currentDefaults.SandboxBinary, + "sandbox-binary": currentDefaults.SandboxBinary, + "UseSandboxCatalog": currentDefaults.UseSandboxCatalog, + "use-sandbox-catalog": currentDefaults.UseSandboxCatalog, + "LogSBOperations": currentDefaults.LogSBOperations, + "log-sb-operations": currentDefaults.LogSBOperations, + "LogDirectory": currentDefaults.LogDirectory, + "log-directory": currentDefaults.LogDirectory, + "CookbookDirectory": currentDefaults.CookbookDirectory, + "cookbook-directory": currentDefaults.CookbookDirectory, + "MasterSlaveBasePort": currentDefaults.MasterSlaveBasePort, + "master-slave-basePort": currentDefaults.MasterSlaveBasePort, + "GroupReplicationBasePort": currentDefaults.GroupReplicationBasePort, + "group-replication-base-port": currentDefaults.GroupReplicationBasePort, + "GroupReplicationSpBasePort": currentDefaults.GroupReplicationSpBasePort, + "group-replication-sp-base-port": currentDefaults.GroupReplicationSpBasePort, + "FanInReplicationBasePort": currentDefaults.FanInReplicationBasePort, + "fanin-replication-base-port": currentDefaults.FanInReplicationBasePort, + "AllMastersReplicationBasePort": currentDefaults.AllMastersReplicationBasePort, + "all-masters-replication-base-port": currentDefaults.AllMastersReplicationBasePort, + "MultipleBasePort": currentDefaults.MultipleBasePort, + "multiple-base-port": currentDefaults.MultipleBasePort, + "PxcBasePort": currentDefaults.PxcBasePort, + "pxc-base-port": currentDefaults.PxcBasePort, + "NdbBasePort": currentDefaults.NdbBasePort, + "ndb-base-port": currentDefaults.NdbBasePort, + "NdbClusterPort": currentDefaults.NdbClusterPort, + "ndb-cluster-port": currentDefaults.NdbClusterPort, + "GroupPortDelta": currentDefaults.GroupPortDelta, + "group-port-delta": currentDefaults.GroupPortDelta, + "MysqlXPortDelta": currentDefaults.MysqlXPortDelta, + "mysqlx-port-delta": currentDefaults.MysqlXPortDelta, + "AdminPortDelta": currentDefaults.AdminPortDelta, + "admin-port-delta": currentDefaults.AdminPortDelta, + "MasterName": currentDefaults.MasterName, + "master-name": currentDefaults.MasterName, + "MasterAbbr": currentDefaults.MasterAbbr, + "master-abbr": currentDefaults.MasterAbbr, + "NodePrefix": currentDefaults.NodePrefix, + "node-prefix": currentDefaults.NodePrefix, + "SlavePrefix": currentDefaults.SlavePrefix, + "slave-prefix": currentDefaults.SlavePrefix, + "SlaveAbbr": currentDefaults.SlaveAbbr, + "slave-abbr": currentDefaults.SlaveAbbr, + "SandboxPrefix": currentDefaults.SandboxPrefix, + "sandbox-prefix": currentDefaults.SandboxPrefix, + "MasterSlavePrefix": currentDefaults.MasterSlavePrefix, + "master-slave-prefix": currentDefaults.MasterSlavePrefix, + "GroupPrefix": currentDefaults.GroupPrefix, + "group-prefix": currentDefaults.GroupPrefix, + "GroupSpPrefix": currentDefaults.GroupSpPrefix, + "group-sp-prefix": currentDefaults.GroupSpPrefix, + "MultiplePrefix": currentDefaults.MultiplePrefix, + "multiple-prefix": currentDefaults.MultiplePrefix, + "FanInPrefix": currentDefaults.FanInPrefix, + "fan-in-prefix": currentDefaults.FanInPrefix, + "AllMastersPrefix": currentDefaults.AllMastersPrefix, + "all-masters-prefix": currentDefaults.AllMastersPrefix, + "ReservedPorts": currentDefaults.ReservedPorts, + "reserved-ports": currentDefaults.ReservedPorts, + "RemoteRepository": currentDefaults.RemoteRepository, + "remote-repository": currentDefaults.RemoteRepository, + "RemoteIndexFile": currentDefaults.RemoteIndexFile, + "remote-index-file": currentDefaults.RemoteIndexFile, + "PxcPrefix": currentDefaults.PxcPrefix, + "pxc-prefix": currentDefaults.PxcPrefix, + "NdbPrefix": currentDefaults.NdbPrefix, + "ndb-prefix": currentDefaults.NdbPrefix, + "Timestamp": currentDefaults.Timestamp, + "timestamp": currentDefaults.Timestamp, + } +} diff --git a/docs/API/API-1.30.md b/docs/API/API-1.30.md new file mode 100644 index 00000000..2ebef723 --- /dev/null +++ b/docs/API/API-1.30.md @@ -0,0 +1,941 @@ +This is the list of commands and modifiers available for +dbdeployer 1.30.0 as of 05-May-2019 07:21 UTC + +# main + $ dbdeployer -h + dbdeployer makes MySQL server installation an easy task. + Runs single, multiple, and replicated sandboxes. + + Usage: + dbdeployer [command] + + Available Commands: + admin sandbox management tasks + cookbook Shows dbdeployer samples + defaults tasks related to dbdeployer defaults + delete delete an installed sandbox + delete-binaries delete an expanded tarball + deploy deploy sandboxes + export Exports the command structure in JSON format + global Runs a given command in every sandbox + help Help about any command + info Shows information about dbdeployer environment samples + remote Manages remote tarballs + sandboxes List installed sandboxes + unpack unpack a tarball into the binary directory + usage Shows usage of installed sandboxes + versions List available versions + + Flags: + --config string configuration file (default "$HOME/.dbdeployer/config.json") + -h, --help help for dbdeployer + --sandbox-binary string Binary repository (default "$HOME/opt/mysql") + --sandbox-home string Sandbox deployment directory (default "$HOME/sandboxes") + --version version for dbdeployer + + Use "dbdeployer [command] --help" for more information about a command. + + + $ dbdeployer-docs tree + - admin + - capabilities + - lock + - unlock + - upgrade + - cookbook + - create + - list + - show + - defaults + - export + - load + - reset + - show + - store + - templates + - describe + - export + - import + - list + - reset + - show + - update + - delete + - delete-binaries + - deploy + - multiple + - replication + - single + - export + - global + - restart + - start + - status + - stop + - test + - test-replication + - use + - info + - defaults + - version + - remote + - download + - list + - sandboxes + - unpack + - usage + - versions + + +## admin + $ dbdeployer admin -h + Runs commands related to the administration of sandboxes. + + Usage: + dbdeployer admin [command] + + Aliases: + admin, manage + + Available Commands: + capabilities Shows capabilities of a given flavor [and optionally version] + lock Locks a sandbox, preventing deletion + unlock Unlocks a sandbox + upgrade Upgrades a sandbox to a newer version + + Flags: + -h, --help help for admin + + + $ dbdeployer admin capabilities -h + Shows the capabilities of all flavors. + If a flavor is specified, only the capabilities of that flavor are shown. + If also a version is specified, we show what that version supports + + Usage: + dbdeployer admin capabilities [flavor [version]] [flags] + + Examples: + dbdeployer admin capabilities + dbdeployer admin capabilities mysql + dbdeployer admin capabilities mysql 5.7.11 + dbdeployer admin capabilities mysql 5.7.13 + + + Flags: + -h, --help help for capabilities + + + $ dbdeployer admin lock -h + Prevents deletion for a given sandbox. + Note that the deletion being prevented is only the one occurring through dbdeployer. + Users can still delete locked sandboxes manually. + + Usage: + dbdeployer admin lock sandbox_name [flags] + + Aliases: + lock, preserve + + Flags: + -h, --help help for lock + + + $ dbdeployer admin unlock -h + Removes lock, allowing deletion of a given sandbox + + Usage: + dbdeployer admin unlock sandbox_name [flags] + + Aliases: + unlock, unpreserve + + Flags: + -h, --help help for unlock + + + $ dbdeployer admin upgrade -h + Upgrades a sandbox to a newer version. + The sandbox with the new version must exist already. + The data directory of the old sandbox will be moved to the new one. + + Usage: + dbdeployer admin upgrade sandbox_name newer_sandbox [flags] + + Examples: + dbdeployer admin upgrade msb_8_0_11 msb_8_0_12 + + Flags: + --dry-run Shows upgrade operations, but don't execute them + -h, --help help for upgrade + --verbose Shows upgrade operations + + + +## cookbook + $ dbdeployer cookbook -h + Shows practical examples of dbdeployer usages, by creating usage scripts. + + Usage: + dbdeployer cookbook [command] + + Aliases: + cookbook, recipes, samples + + Available Commands: + create creates a script for a given recipe + list Shows available dbdeployer samples + show Shows the contents of a given recipe + + Flags: + --flavor string For which flavor this recipe is + -h, --help help for cookbook + + + $ dbdeployer cookbook create -h + creates a script for a given recipe + + Usage: + dbdeployer cookbook create recipe_name or ALL [flags] + + Aliases: + create, make + + Flags: + -h, --help help for create + + + $ dbdeployer cookbook list -h + Shows list of available cookbook recipes + + Usage: + dbdeployer cookbook list [flags] + + Flags: + -h, --help help for list + --sort-by string Sort order for the list (name, flavor, script) (default "name") + + + $ dbdeployer cookbook show -h + Shows the contents of a given recipe, without actually running it + + Usage: + dbdeployer cookbook show recipe_name [flags] + + Flags: + -h, --help help for show + --raw Shows the recipe without variable substitution + + + +## defaults + $ dbdeployer defaults -h + Runs commands related to the administration of dbdeployer, + such as showing the defaults and saving new ones. + + Usage: + dbdeployer defaults [command] + + Aliases: + defaults, config + + Available Commands: + export Export current defaults to a given file + load Load defaults from file + reset Remove current defaults file + show shows defaults + store Store current defaults + templates Templates management + update Change defaults value + + Flags: + -h, --help help for defaults + + + $ dbdeployer defaults export -h + Saves current defaults to a user-defined file + + Usage: + dbdeployer defaults export filename [flags] + + Flags: + -h, --help help for export + + + $ dbdeployer defaults load -h + Reads defaults from file and saves them to dbdeployer configuration file ($HOME/.dbdeployer/config.json) + + Usage: + dbdeployer defaults load file_name [flags] + + Aliases: + load, import + + Flags: + -h, --help help for load + + + $ dbdeployer defaults reset -h + Removes current dbdeployer configuration file ($HOME/.dbdeployer/config.json) + Afterwards, dbdeployer will use the internally stored defaults. + + Usage: + dbdeployer defaults reset [flags] + + Aliases: + reset, remove + + Flags: + -h, --help help for reset + + + $ dbdeployer defaults show -h + Shows currently defined defaults + + Usage: + dbdeployer defaults show [flags] + + Aliases: + show, list + + Flags: + -h, --help help for show + + + $ dbdeployer defaults store -h + Saves current defaults to dbdeployer configuration file ($HOME/.dbdeployer/config.json) + + Usage: + dbdeployer defaults store [flags] + + Flags: + -h, --help help for store + + + +## defaults templates + $ dbdeployer defaults templates -h + The commands in this section show the templates used + to create and manipulate sandboxes. + + Usage: + dbdeployer defaults templates [command] + + Aliases: + templates, template, tmpl, templ + + Available Commands: + describe Describe a given template + export Exports templates to a directory + import imports templates from a directory + list list available templates + reset Removes all template files + show Show a given template + + Flags: + -h, --help help for templates + + + $ dbdeployer defaults templates describe -h + Describe a given template + + Usage: + dbdeployer defaults templates describe template_name [flags] + + Aliases: + describe, descr, structure, struct + + Flags: + -h, --help help for describe + --with-contents Shows complete structure and contents + + + $ dbdeployer defaults templates export -h + Exports a group of templates (or "ALL") to a given directory + + Usage: + dbdeployer defaults templates export group_name directory_name [template_name] [flags] + + Flags: + -h, --help help for export + + + $ dbdeployer defaults templates import -h + Imports a group of templates (or "ALL") from a given directory + + Usage: + dbdeployer defaults templates import group_name directory_name [template_name] [flags] + + Flags: + -h, --help help for import + + + $ dbdeployer defaults templates list -h + list available templates + + Usage: + dbdeployer defaults templates list [group] [flags] + + Flags: + -h, --help help for list + -s, --simple Shows only the template names, without description + + + $ dbdeployer defaults templates reset -h + Removes all template files that were imported and starts using internal values. + + Usage: + dbdeployer defaults templates reset [flags] + + Aliases: + reset, remove + + Flags: + -h, --help help for reset + + + $ dbdeployer defaults templates show -h + Show a given template + + Usage: + dbdeployer defaults templates show template_name [flags] + + Flags: + -h, --help help for show + + + $ dbdeployer defaults update -h + Updates one field of the defaults. Stores the result in the dbdeployer configuration file. + Use "dbdeployer defaults show" to see which values are available + + Usage: + dbdeployer defaults update label value [flags] + + Examples: + + $ dbdeployer defaults update master-slave-base-port 17500 + + + Flags: + -h, --help help for update + + + +## delete + $ dbdeployer delete -h + Stops the sandbox (and its depending sandboxes, if any), and removes it. + Warning: this command is irreversible! + + Usage: + dbdeployer delete sandbox_name (or "ALL") [flags] + + Aliases: + delete, remove, destroy + + Examples: + + $ dbdeployer delete msb_8_0_4 + $ dbdeployer delete rsandbox_5_7_21 + + Flags: + --concurrent Runs multiple deletion tasks concurrently. + --confirm Requires confirmation. + -h, --help help for delete + --skip-confirm Skips confirmation with multiple deletions. + + + +## delete-binaries + $ dbdeployer delete-binaries -h + Removes the given directory and all its subdirectories. + It will fail if the directory is still used by any sandbox. + Warning: this command is irreversible! + + Usage: + dbdeployer delete-binaries binaries_dir_name [flags] + + Examples: + + $ dbdeployer delete-binaries 8.0.4 + $ dbdeployer delete ps5.7.25 + + Flags: + -h, --help help for delete-binaries + --skip-confirm Skips confirmation. + + + +## deploy + $ dbdeployer deploy -h + Deploys single, multiple, or replicated sandboxes + + Usage: + dbdeployer deploy [command] + + Available Commands: + multiple create multiple sandbox + replication create replication sandbox + single deploys a single sandbox + + Flags: + --base-port int Overrides default base-port (for multiple sandboxes) + --binary-version string Specifies the version when the basedir directory name does not contain it (i.e. it is not x.x.xx) + --bind-address string defines the database bind-address (default "127.0.0.1") + --client-from string Where to get the client binaries from + --concurrent Runs multiple sandbox deployments concurrently + --custom-mysqld string Uses an alternative mysqld (must be in the same directory as regular mysqld) + -p, --db-password string database password (default "msandbox") + -u, --db-user string database user (default "msandbox") + --defaults strings Change defaults on-the-fly (--defaults=label:value) + --disable-mysqlx Disable MySQLX plugin (8.0.11+) + --enable-admin-address Enables admin address (8.0.14+) + --enable-general-log Enables general log for the sandbox (MySQL 5.1+) + --enable-mysqlx Enables MySQLX plugin (5.7.12+) + --expose-dd-tables In MySQL 8.0+ shows data dictionary tables + --flavor string Defines the tarball flavor (MySQL, NDB, Percona Server, etc) + --flavor-in-prompt Add flavor values to prompt + --force If a destination sandbox already exists, it will be overwritten + --gtid enables GTID + -h, --help help for deploy + --history-dir string Where to store mysql client history (default: in sandbox directory) + --init-general-log uses general log during initialization (MySQL 5.1+) + -i, --init-options strings mysqld options to run during initialization + --keep-server-uuid Does not change the server UUID + --log-directory string Where to store dbdeployer logs (default "$HOME/sandboxes/logs") + --log-sb-operations Logs sandbox operations to a file + --my-cnf-file string Alternative source file for my.sandbox.cnf + -c, --my-cnf-options strings mysqld options to add to my.sandbox.cnf + --native-auth-plugin in 8.0.4+, uses the native password auth plugin + --port int Overrides default port + --port-as-server-id Use the port number as server ID + --post-grants-sql strings SQL queries to run after loading grants + --post-grants-sql-file string SQL file to run after loading grants + --pre-grants-sql strings SQL queries to run before loading grants + --pre-grants-sql-file string SQL file to run before loading grants + --remote-access string defines the database access (default "127.%") + --repl-crash-safe enables Replication crash safe + --rpl-password string replication password (default "rsandbox") + --rpl-user string replication user (default "rsandbox") + --sandbox-directory string Changes the default sandbox directory + --skip-load-grants Does not load the grants + --skip-report-host Does not include report host in my.sandbox.cnf + --skip-report-port Does not include report port in my.sandbox.cnf + --skip-start Does not start the database server + --socket-in-datadir Create socket in datadir instead of $TMPDIR + --use-template strings [template_name:file_name] Replace existing template with one from file + + + $ dbdeployer deploy multiple -h + Creates several sandboxes of the same version, + without any replication relationship. + For this command to work, there must be a directory $HOME/opt/mysql/5.7.21, containing + the binary files from mysql-5.7.21-$YOUR_OS-x86_64.tar.gz + Use the "unpack" command to get the tarball into the right directory. + + Usage: + dbdeployer deploy multiple MySQL-Version [flags] + + Examples: + + $ dbdeployer deploy multiple 5.7.21 + + + Flags: + -h, --help help for multiple + -n, --nodes int How many nodes will be installed (default 3) + + + $ dbdeployer deploy replication -h + The replication command allows you to deploy several nodes in replication. + Allowed topologies are "master-slave" for all versions, and "group", "all-masters", "fan-in" + for 5.7.17+. + Topologies "pcx" and "ndb" are available for binaries of type Percona Xtradb Cluster and MySQL Cluster. + For this command to work, there must be a directory $HOME/opt/mysql/5.7.21, containing + the binary files from mysql-5.7.21-$YOUR_OS-x86_64.tar.gz + Use the "unpack" command to get the tarball into the right directory. + + Usage: + dbdeployer deploy replication MySQL-Version [flags] + + Examples: + + $ dbdeployer deploy replication 5.7 # deploys highest revision for 5.7 + $ dbdeployer deploy replication 5.7.21 # deploys a specific revision + $ dbdeployer deploy replication /path/to/5.7.21 # deploys a specific revision in a given path + # (implies topology = master-slave) + + $ dbdeployer deploy --topology=master-slave replication 5.7 + # (explicitly setting topology) + + $ dbdeployer deploy --topology=group replication 5.7 + $ dbdeployer deploy --topology=group replication 8.0 --single-primary + $ dbdeployer deploy --topology=all-masters replication 5.7 + $ dbdeployer deploy --topology=fan-in replication 5.7 + $ dbdeployer deploy --topology=pxc replication pxc5.7.25 + $ dbdeployer deploy --topology=ndb replication ndb8.0.14 + + + Flags: + -h, --help help for replication + --master-ip string Which IP the slaves will connect to (default "127.0.0.1") + --master-list string Which nodes are masters in a multi-source deployment (default "1,2") + --ndb-nodes int How many NDB nodes will be installed (default 3) + -n, --nodes int How many nodes will be installed (default 3) + --read-only-slaves Set read-only for slaves + --repl-history-dir uses the replication directory to store mysql client history + --semi-sync Use semi-synchronous plugin + --single-primary Using single primary for group replication + --slave-list string Which nodes are slaves in a multi-source deployment (default "3") + --super-read-only-slaves Set super-read-only for slaves + -t, --topology string Which topology will be installed (default "master-slave") + + + $ dbdeployer deploy single -h + single installs a sandbox and creates useful scripts for its use. + MySQL-Version is in the format x.x.xx, and it refers to a directory named after the version + containing an unpacked tarball. The place where these directories are found is defined by + --sandbox-binary (default: $HOME/opt/mysql.) + For example: + dbdeployer deploy single 5.7 # deploys the latest release of 5.7.x + dbdeployer deploy single 5.7.21 # deploys a specific release + dbdeployer deploy single /path/to/5.7.21 # deploys a specific release in a given path + + For this command to work, there must be a directory $HOME/opt/mysql/5.7.21, containing + the binary files from mysql-5.7.21-$YOUR_OS-x86_64.tar.gz + Use the "unpack" command to get the tarball into the right directory. + + Usage: + dbdeployer deploy single MySQL-Version [flags] + + Flags: + -h, --help help for single + --master Make the server replication ready + --prompt string Default prompt for the single client (default "mysql") + + + +## export + $ dbdeployer export -h + Exports the command line structure, with examples and flags, to a JSON structure. + If a command is given, only the structure of that command and below will be exported. + Given the length of the output, it is recommended to pipe it to a file or to another command. + + Usage: + dbdeployer export [command [sub-command]] [ > filename ] [ | command ] [flags] + + Aliases: + export, dump + + Flags: + --force-output-to-terminal display output to terminal regardless of pipes being used + -h, --help help for export + + + +## global + $ dbdeployer global -h + This command can propagate the given action through all sandboxes. + + Usage: + dbdeployer global [command] + + Examples: + + $ dbdeployer global use "select version()" + $ dbdeployer global status + $ dbdeployer global stop + + + Available Commands: + restart Restarts all sandboxes + start Starts all sandboxes + status Shows the status in all sandboxes + stop Stops all sandboxes + test Tests all sandboxes + test-replication Tests replication in all sandboxes + use Runs a query in all sandboxes + + Flags: + -h, --help help for global + + + $ dbdeployer global restart -h + Restarts all sandboxes + + Usage: + dbdeployer global restart [options] [flags] + + Flags: + -h, --help help for restart + + + $ dbdeployer global start -h + Starts all sandboxes + + Usage: + dbdeployer global start [options] [flags] + + Flags: + -h, --help help for start + + + $ dbdeployer global status -h + Shows the status in all sandboxes + + Usage: + dbdeployer global status [flags] + + Flags: + -h, --help help for status + + + $ dbdeployer global stop -h + Stops all sandboxes + + Usage: + dbdeployer global stop [flags] + + Flags: + -h, --help help for stop + + + $ dbdeployer global test -h + Tests all sandboxes + + Usage: + dbdeployer global test [flags] + + Aliases: + test, test_sb, test-sb + + Flags: + -h, --help help for test + + + $ dbdeployer global test-replication -h + Tests replication in all sandboxes + + Usage: + dbdeployer global test-replication [flags] + + Aliases: + test-replication, test_replication + + Flags: + -h, --help help for test-replication + + + $ dbdeployer global use -h + Runs a query in all sandboxes. + It does not check if the query is compatible with every version deployed. + For example, a query using @@port won't run in MySQL 5.0.x + + Usage: + dbdeployer global use {query} [flags] + + Examples: + + $ dbdeployer global use "select @@server_id, @@port" + + Flags: + -h, --help help for use + + + +## info + $ dbdeployer info -h + Shows current information about defaults and environment. + + Usage: + dbdeployer info [command] + + Available Commands: + defaults displays a defaults value + version displays the latest version available + + Flags: + --flavor string For which flavor this info is + -h, --help help for info + + + $ dbdeployer info defaults -h + Displays one field of the defaults. + + Usage: + dbdeployer info defaults field-name [flags] + + Examples: + + $ dbdeployer info defaults master-slave-base-port + + + Flags: + -h, --help help for defaults + + + $ dbdeployer info version -h + Displays the latest version available for deployment. + If a short version is indicated (such as 5.7, or 8.0), only the versions belonging to that short + version are searched. + If "all" is indicated after the short version, displays all versions belonging to that short version. + + Usage: + dbdeployer info version [short-version|all] [all] [flags] + + Examples: + + # Shows the latest version available + $ dbdeployer info version + 8.0.16 + + # shows the latest version belonging to 5.7 + $ dbdeployer info version 5.7 + 5.7.26 + + # shows the latest version for every short version + $ dbdeployer info version all + 5.0.96 5.1.73 5.5.53 5.6.41 5.7.26 8.0.16 + + # shows all the versions for a given short version + $ dbdeployer info version 8.0 all + 8.0.11 8.0.12 8.0.13 8.0.14 8.0.15 8.0.16 + + + Flags: + -h, --help help for version + + + +## remote + $ dbdeployer remote -h + Manages remote tarballs + + Usage: + dbdeployer remote [command] + + Available Commands: + download download a remote tarball into a local file + list list remote tarballs + + Flags: + -h, --help help for remote + + + $ dbdeployer remote download -h + If no file name is given, the file name will be .tar.xz + + Usage: + dbdeployer remote download version [file-name] [flags] + + Aliases: + download, get + + Flags: + -h, --help help for download + + + $ dbdeployer remote list -h + list remote tarballs + + Usage: + dbdeployer remote list [version] [flags] + + Aliases: + list, index + + Flags: + -h, --help help for list + + + +## sandboxes + $ dbdeployer sandboxes -h + Lists all sandboxes installed in $SANDBOX_HOME. + If sandboxes are installed in a different location, use --sandbox-home to + indicate where to look. + Alternatively, using --catalog will list all sandboxes, regardless of where + they were deployed. + + Usage: + dbdeployer sandboxes [flags] + + Aliases: + sandboxes, installed, deployed + + Flags: + --catalog Use sandboxes catalog instead of scanning directory + --flavor Shows flavor in sandbox list + --full-info Shows all info in table format + --header Shows header with catalog output + -h, --help help for sandboxes + --table Shows sandbox list as a table + + + +## unpack + $ dbdeployer unpack -h + If you want to create a sandbox from a tarball (.tar.gz or .tar.xz), you first need to unpack it + into the sandbox-binary directory. This command carries out that task, so that afterwards + you can call 'deploy single', 'deploy multiple', and 'deploy replication' commands with only + the MySQL version for that tarball. + If the version is not contained in the tarball name, it should be supplied using --unpack-version. + If there is already an expanded tarball with the same version, a new one can be differentiated with --prefix. + + Usage: + dbdeployer unpack MySQL-tarball [flags] + + Aliases: + unpack, extract, untar, unzip, inflate, expand + + Examples: + + $ dbdeployer unpack mysql-8.0.4-rc-linux-glibc2.12-x86_64.tar.gz + Unpacking tarball mysql-8.0.4-rc-linux-glibc2.12-x86_64.tar.gz to $HOME/opt/mysql/8.0.4 + + $ dbdeployer unpack --prefix=ps Percona-Server-5.7.21-linux.tar.gz + Unpacking tarball Percona-Server-5.7.21-linux.tar.gz to $HOME/opt/mysql/ps5.7.21 + + $ dbdeployer unpack --unpack-version=8.0.18 --prefix=bld mysql-mybuild.tar.gz + Unpacking tarball mysql-mybuild.tar.gz to $HOME/opt/mysql/bld8.0.18 + + + Flags: + --flavor string Defines the tarball flavor (MySQL, NDB, Percona Server, etc) + -h, --help help for unpack + --overwrite Overwrite the destination directory if already exists + --prefix string Prefix for the final expanded directory + --shell Unpack a shell tarball into the corresponding server directory + --target-server string Uses a different server to unpack a shell tarball + --unpack-version string which version is contained in the tarball + --verbosity int Level of verbosity during unpack (0=none, 2=maximum) (default 1) + + + +## usage + $ dbdeployer usage -h + Shows syntax and examples of tools installed in database sandboxes. + + Usage: + dbdeployer usage [single|multiple] [flags] + + Flags: + -h, --help help for usage + + + +## versions + $ dbdeployer versions -h + List available versions + + Usage: + dbdeployer versions [flags] + + Aliases: + versions, available + + Flags: + --by-flavor Shows versions list by flavor + --flavor string Get only versions of the given flavor + -h, --help help for versions + + diff --git a/docs/dbdeployer_completion.sh b/docs/dbdeployer_completion.sh index 5313ac79..6d733ec0 100644 --- a/docs/dbdeployer_completion.sh +++ b/docs/dbdeployer_completion.sh @@ -392,7 +392,6 @@ _dbdeployer_cookbook_create() flags+=("--flavor=") flags+=("--sandbox-binary=") flags+=("--sandbox-home=") - flags+=("--sort-by=") must_have_one_flag=() must_have_one_noun=() @@ -413,11 +412,11 @@ _dbdeployer_cookbook_list() flags_with_completion=() flags_completion=() + flags+=("--sort-by=") flags+=("--config=") flags+=("--flavor=") flags+=("--sandbox-binary=") flags+=("--sandbox-home=") - flags+=("--sort-by=") must_have_one_flag=() must_have_one_noun=() @@ -444,7 +443,6 @@ _dbdeployer_cookbook_show() flags+=("--flavor=") flags+=("--sandbox-binary=") flags+=("--sandbox-home=") - flags+=("--sort-by=") must_have_one_flag=() must_have_one_noun=() @@ -473,7 +471,6 @@ _dbdeployer_cookbook() flags_completion=() flags+=("--flavor=") - flags+=("--sort-by=") flags+=("--config=") flags+=("--sandbox-binary=") flags+=("--sandbox-home=") @@ -1438,6 +1435,80 @@ _dbdeployer_global() noun_aliases=() } +_dbdeployer_info_defaults() +{ + last_command="dbdeployer_info_defaults" + + command_aliases=() + + commands=() + + flags=() + two_word_flags=() + local_nonpersistent_flags=() + flags_with_completion=() + flags_completion=() + + flags+=("--config=") + flags+=("--flavor=") + flags+=("--sandbox-binary=") + flags+=("--sandbox-home=") + + must_have_one_flag=() + must_have_one_noun=() + noun_aliases=() +} + +_dbdeployer_info_version() +{ + last_command="dbdeployer_info_version" + + command_aliases=() + + commands=() + + flags=() + two_word_flags=() + local_nonpersistent_flags=() + flags_with_completion=() + flags_completion=() + + flags+=("--config=") + flags+=("--flavor=") + flags+=("--sandbox-binary=") + flags+=("--sandbox-home=") + + must_have_one_flag=() + must_have_one_noun=() + noun_aliases=() +} + +_dbdeployer_info() +{ + last_command="dbdeployer_info" + + command_aliases=() + + commands=() + commands+=("defaults") + commands+=("version") + + flags=() + two_word_flags=() + local_nonpersistent_flags=() + flags_with_completion=() + flags_completion=() + + flags+=("--flavor=") + flags+=("--config=") + flags+=("--sandbox-binary=") + flags+=("--sandbox-home=") + + must_have_one_flag=() + must_have_one_noun=() + noun_aliases=() +} + _dbdeployer_remote_download() { last_command="dbdeployer_remote_download" @@ -1668,6 +1739,7 @@ _dbdeployer_root_command() aliashash["dump"]="export" fi commands+=("global") + commands+=("info") commands+=("remote") commands+=("sandboxes") if [[ -z "${BASH_VERSION}" || "${BASH_VERSINFO[0]}" -gt 3 ]]; then diff --git a/globals/globals.go b/globals/globals.go index 6a7f7575..1310ffdc 100644 --- a/globals/globals.go +++ b/globals/globals.go @@ -150,17 +150,19 @@ const ( // Instantiated in cmd/export.go - ForceOutputToTermLabel = "force-output-to-terminal" - ExportSandboxDir = "sandbox-dir" - ExportVersionDir = "version-dir" - ExportTemplateGroup = "template-group" - ExportTemplateName = "template-name" - ExportString = "string" - ExportInteger = "integer" - ExportBoolean = "boolean" - ExportCookbookName = "cookbook-name" - ExportTopology = "topology" - ExportAllowedTopologies = "allowed-topologies" + ForceOutputToTermLabel = "force-output-to-terminal" + ExportSandboxDir = "sandbox-dir" + ExportVersionDir = "version-dir" + ExportTemplateGroup = "template-group" + ExportTemplateName = "template-name" + ExportString = "string" + ExportInteger = "integer" + ExportBoolean = "boolean" + ExportCookbookName = "cookbook-name" + ExportTopology = "topology" + ExportAllowedTopologies = "allowed-topologies" + ExportSupportedAllVersions = "supported-all-versions" + ExportSupportedMysqlVersions = "supported-mysql-versions" // Instantiated in sandbox package AutoCnfName = "auto.cnf" @@ -357,8 +359,13 @@ var AllowedTopologies = []string{ NdbLabel, } +// This structure is not used directly by dbdeployer. +// It is here to be used by third party applications that +// use metadata exported using cmd.Export() var ExportReferenceData = map[string]interface{}{ - ExportAllowedTopologies: AllowedTopologies, + ExportAllowedTopologies: AllowedTopologies, + ExportSupportedAllVersions: SupportedAllVersions, + ExportSupportedMysqlVersions: SupportedMySQLVersions, } var ( @@ -375,6 +382,14 @@ var ( "kill", "ls", "mkdir", "mv", "printf", "rm", "seq", "sh", "sleep", "stat", "tail", "test", "[", "touch", "tr", "wc"} + SupportedMySQLVersions = []string{ + "4.1", "5.0", "5.1", "5.5", "5.6", "5.7", "8.0", + } + SupportedAllVersions = []string{ + "4.1", "5.0", "5.1", "5.5", "5.6", "5.7", "8.0", + "10.0", "10.1", "10.2", "10.3", "10.4", + } + // Extra executables needed for PXC NeededPxcExecutables = []string{"rsync", "lsof"} ) diff --git a/mkreadme/readme_template.md b/mkreadme/readme_template.md index 1ac1ff7d..31ff1e48 100644 --- a/mkreadme/readme_template.md +++ b/mkreadme/readme_template.md @@ -11,7 +11,7 @@ Documentation updated for version {{.Version}} ({{.Date}}) - [Main operations](#Main-operations) - [Database server flavors](#Database-server-flavors) - [Getting remote tarballs](#Getting-remote-tarballs) -- [Practical examples](#Practical-examples) +- [Practical examples (cookbook)](#Practical-examples) - [Standard and non-standard basedir names](#Standard-and-non-standard-basedir-names) - [Using short version numbers](#Using-short-version-numbers) - [Multiple sandboxes, same version and type](#Multiple-sandboxes-same-version-and-type) @@ -31,6 +31,7 @@ Documentation updated for version {{.Version}} ({{.Date}}) - [Dedicated admin address](#Dedicated-admin-address) - [Obtaining sandbox metadata](#Obtaining-sandbox-metadata) - [Replication between sandboxes](#Replication-between-sandboxes) +- [Using dbdeployer in scripts](#Using-dbdeployer-in-scripts) - [Compiling dbdeployer](#Compiling-dbdeployer) - [Generating additional documentation](#Generating-additional-documentation) - [Command line completion](#Command-line-completion) @@ -941,6 +942,57 @@ Examples: ~/sandboxes/group_8_0_15_2/replicate_from ms_8_0_15_1 ``` +# Using dbdeployer in scripts + +dbdeployer has been designed to simplify automated operations. Using it in scripts is easy, as shown in the [cookbook examples](#Practical-examples). +In addition to run operations on sandboxes, dbdeployer can also provide information about the environment in a way that is suitable for scripting. + +For example, if you want to deploy a sandbox using the most recent 5.7 binaries, you may run `dbdeployer versions`, look which versions are available, and pick the most recent one. But dbdeployer 1.30.0 can aytomate this procedure using `dbdeployer info version 5.7`. This command will print the latest 5.7 binaries to the standard output, allowing us to create dynamic scripts such as: + +```bash +# the absolute latest version +latest=$(dbdeployer info version) +latest57=$(dbdeployer info version 5.7) +latest80=$(dbdeployer info version 8.0) + +if [ -z "$latest" ] +then + echo "No versions found" + exit 1 +fi + +echo "The latest version is $latest" + +if [ -n "$latest57" ] +then + echo "# latest for 5.7 : $latest57" + dbdeployer deploy single $latest57 +fi + +if [ -n "$latest80" ] +then + echo "# latest for 8.0 : $latest80" + dbdeployer deploy single $latest80 +fi +``` + + {{dbdeployer info version -h}} + +Similarly to `versions`, the `defaults` subcommand allows us to get dbdeployer metadata in a way that can be used in scripts + + {{dbdeployer info defaults -h}} + +For example + +``` +$ dbdeployer info defaults sandbox-prefix +msb_ + +$ dbdeployer info defaults master-slave-ptrefix +rsandbox_ +``` +You can ask for any fields from the defaults (see `dbdeployer defaults list` for the field names). + # Compiling dbdeployer Should you need to compile your own binaries for dbdeployer, follow these steps: