Code cleanup and more docs

- Code reformatting and minor refactoring
- Fixed syntax error in code for tree.go
- Added coding sample minimal-sandbox2.go

Changelog

@@ -1,3 +1,7 @@
+1.4.2 06-May-2018
+ - Code reformatting and minor refactoring 
+ - Fixed syntax error in code for tree.go
+ - Added coding sample minimal-sandbox2.go
 1.4.1 05-May-2018
  - Merged pull request #11 from percona-csalguero/include_version_in_dir
  - Fixed Issue #12 "deploying a sandbox with invalid version does not fail"

README.md

@@ -3,7 +3,7 @@
 [DBdeployer](https://github.com/datacharmer/dbdeployer) is a tool that deploys MySQL database servers easily.
 This is a port of [MySQL-Sandbox](https://github.com/datacharmer/mysql-sandbox), originally written in Perl, and re-designed from the ground up in [Go](https://golang.org). See the [features comparison](https://github.com/datacharmer/dbdeployer/blob/master/docs/features.md) for more detail.
 
-Documentation updated for version 1.4.1 (05-May-2018 20:43 UTC)
+Documentation updated for version 1.4.2 (06-May-2018 14:42 UTC)
 
 ## Installation
 
@@ -13,7 +13,7 @@ Get the one for your O.S. from [dbdeployer releases](https://github.com/datachar
 
 For example:
 
- $ VERSION=1.4.1
+ $ VERSION=1.4.2
  $ origin=https://github.com/datacharmer/dbdeployer/releases/download/$VERSION
  $ wget $origin/dbdeployer-$VERSION.linux.tar.gz
  $ tar -xzf dbdeployer-$VERSION.linux.tar.gz
@@ -47,7 +47,7 @@ For example:
 The program doesn't have any dependencies. Everything is included in the binary. Calling *dbdeployer* without arguments or with ``--help`` will show the main help screen.
 
  $ dbdeployer --version
- dbdeployer version 1.4.1
+ dbdeployer version 1.4.2
 
 
  $ dbdeployer -h
@@ -459,7 +459,7 @@ Here's how:
   3306,
   33060
   ],
-  "timestamp": "Sat May 5 22:43:32 CEST 2018"
+  "timestamp": "Sun May 6 16:42:39 CEST 2018"
   }
 
 
@@ -496,7 +496,7 @@ Here's how:
   3306,
   33060
   ],
-  "timestamp": "Sat May 5 22:43:32 CEST 2018"
+  "timestamp": "Sun May 6 16:42:39 CEST 2018"
   }
 
 
@@ -674,18 +674,18 @@ Should you need to compile your own binaries for dbdeployer, follow these steps:
 2. Run ``go get github.com/datacharmer/dbdeployer``. This will import all the code that is needed to build dbdeployer.
 3. Change directory to ``$GOPATH/src/github.com/datacharmer/dbdeployer``.
 4. From the folder ``./pflag``, copy the file ``string_slice.go`` to ``$GOPATH/src/github.com/spf13/pflag``.
-5. Run ``./build.sh {linux|OSX} 1.4.1``
-6. If you need the docs enabled binaries (see the section "Generating additional documentation") run ``MKDOCS=1 ./build.sh {linux|OSX} 1.4.1``
+5. Run ``./build.sh {linux|OSX} 1.4.2``
+6. If you need the docs enabled binaries (see the section "Generating additional documentation") run ``MKDOCS=1 ./build.sh {linux|OSX} 1.4.2``
 
 ## Generating additional documentation
 
 Between this file and [the API API list](https://github.com/datacharmer/dbdeployer/blob/master/docs/API/API-1.1.md), you have all the existing documentation for dbdeployer.
 Should you need additional formats, though, dbdeployer is able to generate them on-the-fly. Tou will need the docs-enabled binaries: in the distribution list, you will find:
 
-* dbdeployer-1.4.1-docs.linux.tar.gz
-* dbdeployer-1.4.1-docs.osx.tar.gz
-* dbdeployer-1.4.1.linux.tar.gz
-* dbdeployer-1.4.1.osx.tar.gz
+* dbdeployer-1.4.2-docs.linux.tar.gz
+* dbdeployer-1.4.2-docs.osx.tar.gz
+* dbdeployer-1.4.2.linux.tar.gz
+* dbdeployer-1.4.2.osx.tar.gz
 
 The executables containing ``-docs`` in their name have the same capabilities of the regular ones, but in addition they can run the *hidden* command ``tree``, with alias ``docs``.
 

cmd/root.go

@@ -82,5 +82,8 @@ func init() {
  set_pflag(rootCmd,"sandbox-binary", "", "SANDBOX_BINARY", defaults.Defaults().SandboxBinary, "Binary repository", false)
 
  rootCmd.InitDefaultVersionFlag()
+
+ // Indicates that we're using dbdeployer command line interface
+ // rather than calling its sandbox creation functions from other apps.
  defaults.UsingDbDeployer = true
 }

common/version.go

@@ -15,7 +15,7 @@
 
 package common
 
-var VersionDef string = "1.4.1" // 2018-05-05
+var VersionDef string = "1.4.2" // 2018-05-06
 
 // Compatible version is the version used to mark compatible archives (templates, configuration).
 // It is usually major.minor.0, except when we are at version 0.x, when

concurrent/concurrent.go

@@ -68,6 +68,8 @@ func start_task (num int, w *sync.WaitGroup, tasks CommonChan) {
  }
 }
 
+// Run several tasks in parallel
+
 func RunParallelTasks( priority_level int, operations ExecCommands ) {
  tasks := make(CommonChan, 64)
 
@@ -83,6 +85,43 @@ func RunParallelTasks( priority_level int, operations ExecCommands ) {
  }
 }
 
+/*
+// Given a list of tasks with different priorities
+// This function organizes the queued tasks by priority
+// and runs concurrently the tasks with the same priority
+// until no task is left in the queue.
+// For example we may have:
+ priority command
+ 1 /some/path/init_db
+ 2 /some/path/start
+ 3 /some/path/load_grants
+ 1 /some/other/path/init_db
+ 2 /some/other/path/start
+ 3 /some/other/path/load_grants
+ 1 /some/alternative/path/init_db
+ 2 /some/alternative/path/start
+ 3 /some/alternative/path/load_grants
+
+ This function will receive the commands, and re-arrange them as follows
+ run concurrently: {
+ 1 /some/path/init_db
+ 1 /some/other/path/init_db
+ 1 /some/alternative/path/init_db
+ }
+
+ run concurrently: {
+ 2 /some/path/start
+ 2 /some/other/path/start
+ 2 /some/alternative/path/start
+ }
+ 
+ run concurrently: {
+ 3 /some/path/load_grants
+ 3 /some/other/path/load_grants
+ 3 /some/alternative/path/load_grants
+ }
+*/
+
 func RunParallelTasksByPriority ( exec_lists []ExecutionList) {
  maxPriority := 0
  if len(exec_lists) == 0 {

defaults/defaults.go

@@ -77,6 +77,11 @@ var (
  StarLine string = strings.Repeat("*", LineLength)
  DashLine string = strings.Repeat("-", LineLength)
  HashLine string = strings.Repeat("#", LineLength)
+
+ // This variable is changed to true when the "cmd" package is activated,
+ // meaning that we're using the command line interface of dbdeployer.
+ // It is used to make decisions whether to write messages to the screen
+ // when calling sandbox creation functions from other apps.
  UsingDbDeployer bool = false
 
  factoryDefaults = DbdeployerDefaults{

docs/coding/dbdeployer-as-library.md

@@ -92,5 +92,43 @@ See the sample source file ``minimal-sandbox.go`` for a working example.
 
 If you want to create multiple sandboxes, things are a bit more complicated. ``dbdeployer`` uses a concurrent execution engine that needs to be used with care.
 
+Function ``CreateSingleSandbox`` returns a slice of ``concurrent.ExecutionList``, a structure made of a priority index and commands to run. When sandboxes are created with concurrency, ``CreateSingleSandbox`` will create the sandbox directory and all the scripts, but won't run any expensive tasks, such as database initialization and start. Instead, it will add those commands to the execution list. The calling function (replication or multiple sandbox call) will queue all the execution lists, and then pass the final list to ``defaults.RunParallelTasksByPriority`` which organizes the tasks by priorities and then runs concurrently the ones that have the same priority level until no task is left in the queue.
+
+For example, we may have:
+
+ priority command
+ 1 /some/path/init_db
+ 2 /some/path/start
+ 3 /some/path/load_grants
+ 1 /some/other/path/init_db
+ 2 /some/other/path/start
+ 3 /some/other/path/load_grants
+ 1 /some/alternative/path/init_db
+ 2 /some/alternative/path/start
+ 3 /some/alternative/path/load_grants
+
+``RunParallelTasksByPriority`` will receive the commands, and re-arrange them as follows
+
+```
+run concurrently: {
+ 1 /some/path/init_db
+ 1 /some/other/path/init_db
+ 1 /some/alternative/path/init_db
+}
+
+run concurrently: {
+ 2 /some/path/start
+ 2 /some/other/path/start
+ 2 /some/alternative/path/start
+}
+
+run concurrently: {
+ 3 /some/path/load_grants
+ 3 /some/other/path/load_grants
+ 3 /some/alternative/path/load_grants
+}
+```
+Instead of having 9 commands running sequentially, we will have three groups of concurrent commands. Each group depends on the completion of the previous one (we can't run ``start`` if ``init_db`` did not finish.)
+
 Look at the invocation of the replication command in ``cmd/replication.go`` for an example of how to prepare the SandboxDef structure before calling the relevant function.
 

docs/coding/minimal-sandbox.go

@@ -1,43 +1,68 @@
+// This is a sample source file that shows how
+// to create a MySQL sandbox using dbdeployer code
+// from another Go program.
 package main
 
 import (
- "os"
- "github.com/datacharmer/dbdeployer/sandbox"
  "github.com/datacharmer/dbdeployer/common"
  "github.com/datacharmer/dbdeployer/defaults"
+ "github.com/datacharmer/dbdeployer/sandbox"
+ "os"
 )
 
 func main() {
+ // Searches for expanded sandboxes in $HOME/opt/mysql
+ sandbox_binary := os.Getenv("HOME") + "/opt/mysql"
+
+ // For this to work, we need to have
+ // a MySQL tarball expanded in $HOME/opt/mysql/5.7.22
  version := "5.7.22"
+
+ // Creates sandboxes in $HOME/sandboxes
  sandbox_home := os.Getenv("HOME") + "/sandboxes"
- sandbox_binary := os.Getenv("HOME") + "/opt/mysql"
+
+ // MySQL will look for binaries in $HOME/opt/mysql/5.7.22
  basedir := sandbox_binary + "/" + version
+
+ // The unique port for this sandbox
  port := 5722
+
+ // Username and password for this sandbox
  user := "msandbox"
- password := "psandbox"
+ password := "msandbox"
 
  if !common.DirExists(sandbox_home) {
  common.Mkdir(sandbox_home)
  }
 
- var sdef = sandbox.SandboxDef{
- Version: version, 
- Basedir: basedir,
- SandboxDir: sandbox_home,
- DirName: "msb_5_7_22",
- LoadGrants:true,
- InstalledPorts:[]int{1186, 3306, 33060},
- Port: port,
- DbUser: user,
- DbPassword: password,
- RplUser: "r" + user,
- RplPassword: "r" + password,
- RemoteAccess:"127.%",
- BindAddress:"127.0.0.1",
+ // Minimum data to be filled for a simple sandbox.
+ // See sandbox/sandbox.go for the full description
+ // of this data structure
+ var sdef = sandbox.SandboxDef{
+ Version: version,
+ Basedir: basedir,
+ SandboxDir: sandbox_home,
+ DirName: "msb_5_7_22",
+ LoadGrants: true,
+ InstalledPorts: []int{1186, 3306, 33060},
+ Port: port,
+ DbUser: user,
+ DbPassword: password,
+ RplUser: "r" + user,
+ RplPassword: "r" + password,
+ RemoteAccess: "127.%",
+ BindAddress: "127.0.0.1",
  }
 
+ // Calls the sandbox creation
  sandbox.CreateSingleSandbox(sdef)
+
+ // Invokes the sandbox self-testing script
  common.Run_cmd(sandbox_home + "/msb_5_7_22/test_sb")
+
+ // Removes the sandbox from disk
  sandbox.RemoveSandbox(sandbox_home, "msb_5_7_22", false)
- defaults.DeleteFromCatalog(sandbox_home+"/msb_5_7_22")
+
+ // Removes the sandbox from dbdeployer catalog
+ defaults.DeleteFromCatalog(sandbox_home + "/msb_5_7_22")
 }

docs/coding/minimal-sandbox2.go

@@ -0,0 +1,96 @@
+// This is a sample source file that shows how
+// to create two MySQL sandboxes using dbdeployer code
+// from another Go program.
+package main
+
+import (
+ "github.com/datacharmer/dbdeployer/common"
+ "github.com/datacharmer/dbdeployer/defaults"
+ "github.com/datacharmer/dbdeployer/sandbox"
+ "os"
+)
+
+func main() {
+ // Searches for expanded sandboxes in $HOME/opt/mysql
+ sandbox_binary := os.Getenv("HOME") + "/opt/mysql"
+
+ // Creates sandboxes in $HOME/sandboxes
+ sandbox_home := os.Getenv("HOME") + "/sandboxes"
+
+ // For this to work, we need to have
+ // a MySQL tarball expanded in $HOME/opt/mysql/5.7.22
+ version1 := "5.7.22"
+ version2 := "5.6.25"
+
+ sandbox_name1 := "msb_5_7_22"
+ sandbox_name2 := "msb_5_6_25"
+
+ // The unique ports for these sandboxes
+ port1 := 5722
+ port2 := 5625
+
+ // MySQL will look for binaries in $HOME/opt/mysql/5.7.22
+ basedir1 := sandbox_binary + "/" + version1 // This is what dbdeployer expects
+ // i.e. a name containing the full version
+
+ // MySQL will look for binaries in $HOME/opt/mysql/my-5.6
+ basedir2 := sandbox_binary + "/my-5.6" // This is a deviation from dbdeployer
+ // paradigm, using a non-standard name
+ // for the base directory
+
+ // Username and password for this sandbox
+ user := "msandbox"
+ password := "msandbox"
+
+ // Creates the base target directory if it doesn't exist
+ if !common.DirExists(sandbox_home) {
+ common.Mkdir(sandbox_home)
+ }
+
+ // Minimum data to be filled for a simple sandbox.
+ // See sandbox/sandbox.go for the full description
+ // of this data structure
+ var sdef = sandbox.SandboxDef{
+ Version: version1,
+ Basedir: basedir1,
+ SandboxDir: sandbox_home,
+ DirName: sandbox_name1,
+ LoadGrants: true,
+ // This is the list of ports to ignore
+ // when checking if the designated port is
+ // used or not.
+ // Try changing the Port item to 3306.
+ // You will see that the sandbox will install using 3307
+ InstalledPorts: []int{1186, 3306, 33060},
+ Port: port1,
+ DbUser: user,
+ DbPassword: password,
+ RplUser: "r" + user,
+ RplPassword: "r" + password,
+ RemoteAccess: "127.%",
+ BindAddress: "127.0.0.1",
+ }
+
+ // Calls the sandbox creation
+ sandbox.CreateSingleSandbox(sdef)
+
+ sdef.Version = version2
+ sdef.Basedir = basedir2
+ sdef.DirName = sandbox_name2
+ sdef.Port = port2
+
+ // Calls the sandbox creation for the second sandbox
+ sandbox.CreateSingleSandbox(sdef)
+
+ // Invokes the sandbox self-testing script
+ common.Run_cmd(sandbox_home + "/" + sandbox_name1 + "/test_sb")
+ common.Run_cmd(sandbox_home + "/" + sandbox_name2 + "/test_sb")
+
+ // Removes the sandbox from disk
+ sandbox.RemoveSandbox(sandbox_home, sandbox_name1, false)
+ sandbox.RemoveSandbox(sandbox_home, sandbox_name2, false)
+
+ // Removes the sandbox from dbdeployer catalog
+ defaults.DeleteFromCatalog(sandbox_home + "/" + sandbox_name1)
+ defaults.DeleteFromCatalog(sandbox_home + "/" + sandbox_name2)
+}