Re-organize commands under more specific click groups (elastic#356)

* Restructure commands under more specific click groups
* standardize CLI error handling
* add global debug options
* move es and kibana clients into their click groups
* move commands and groups to dedicated files 
* distinguish variable names for better env/config parsing

.github/workflows/pythonpackage.yml

@@ -30,11 +30,11 @@ jobs:
 
     - name: Python License Check
       run: |
-        python -m detection_rules license-check
+        python -m detection_rules dev license-check
         
     - name: Build release package
       run: |
-        python -m detection_rules build-release
+        python -m detection_rules dev build-release
 
     - name: Archive production artifacts
       uses: actions/upload-artifact@v2

CLI.md

@@ -25,8 +25,8 @@ Currently supported arguments:
 * elasticsearch_url
 * kibana_url
 * cloud_id
-* username
-* password
+* *_username (kibana and es)
+* *_password (kibana and es)
 
 #### Using environment variables
 
@@ -92,14 +92,79 @@ This will also strip additional fields and prompt for missing required fields.
 <a id="note-3">\* Note</a>: This will attempt to parse ALL files recursively within a specified directory.
 
 
+## Commands using Elasticsearch and Kibana clients
+
+Commands which connect to Elasticsearch or Kibana are embedded under the subcommands:
+* es
+* kibana
+
+These command groups will leverage their respective clients and will automatically use parsed config options if
+defined, otherwise arguments should be passed to the sub-command as:
+
+`python -m detection-rules kibana -u <username> -p <password> upload-rule <...>`
+
+```console
+python -m detection_rules es -h
+
+Usage: detection_rules es [OPTIONS] COMMAND [ARGS]...
+
+  Commands for integrating with Elasticsearch.
+
+Options:
+  -e, --elasticsearch-url TEXT
+  --cloud-id TEXT
+  -u, --es-user TEXT
+  -p, --es-password TEXT
+  -t, --timeout INTEGER         Timeout for elasticsearch client
+  -h, --help                    Show this message and exit.
+
+Commands:
+  collect-events  Collect events from Elasticsearch.
+```
+
+```console
+python -m detection_rules kibana -h
+
+Usage: detection_rules kibana [OPTIONS] COMMAND [ARGS]...
+
+  Commands for integrating with Kibana.
+
+Options:
+  -k, --kibana-url TEXT
+  --cloud-id TEXT
+  -u, --kibana-user TEXT
+  -p, --kibana-password TEXT
+  -t, --timeout INTEGER       Timeout for kibana client
+  -h, --help                  Show this message and exit.
+
+Commands:
+  upload-rule  Upload a list of rule .toml files to Kibana.
+```
+
+
 ## Uploading rules to Kibana
 
-Toml formatted rule files can be uploaded as custom rules using the `kibana-upload` command. To upload more than one 
+Toml formatted rule files can be uploaded as custom rules using the `kibana upload-rule` command. To upload more than one 
 file, specify multiple files at a time as individual args. This command is meant to support uploading and testing of 
 rules and is not intended for production use in its current state.
 
 ```console
-python -m detection_rules kibana-upload my-rules/example_custom_rule.toml
+python -m detection_rules kibana upload-rule -h
+
+Kibana client:
+Options:
+  -k, --kibana-url TEXT
+  --cloud-id TEXT
+  -u, --kibana-user TEXT
+  -p, --kibana-password TEXT
+  -t, --timeout INTEGER       Timeout for kibana client
+
+Usage: detection_rules kibana upload-rule [OPTIONS] TOML_FILES...
+
+  Upload a list of rule .toml files to Kibana.
+
+Options:
+  -h, --help  Show this message and exit.
 ```
 
 _*To load a custom rule, the proper index must be setup first. The simplest way to do this is to click 
@@ -130,3 +195,11 @@ rules. This is based on the hash of the rule in the following format:
 * sha256 hash
 
 As a result, all cases where rules are shown or converted to JSON are not just simple conversions from TOML.
+
+## Debugging
+
+Most of the CLI errors will print a concise, user friendly error. To enable debug mode and see full error stacktraces,
+you can define `"debug": true` in your config file, or run `python -m detection-rules -d <commands...>`.
+
+Precedence goes to the flag over the config file, so if debug is enabled in your config and you run 
+`python -m detection-rules --no-debug`, debugging will be disabled.

Makefile

@@ -35,7 +35,7 @@ pytest: $(VENV) deps
 .PHONY: license-check
 license-check: $(VENV) deps
 	@echo "LICENSE CHECK"
-	$(PYTHON) -m detection_rules license-check
+	$(PYTHON) -m detection_rules dev license-check
 
 .PHONY: lint
 lint: $(VENV) deps
@@ -48,12 +48,12 @@ test: $(VENV) lint pytest
 .PHONY: release
 release: deps
 	@echo "RELEASE: $(app_name)"
-	$(PYTHON) -m detection_rules build-release
+	$(PYTHON) -m detection_rules dev build-release
 	rm -rf dist
 	mkdir dist
 	cp -r releases/*/*.zip dist/
 
 .PHONY: kibana-commit
 kibana-commit: deps
 	@echo "PREP KIBANA-COMMIT: $(app_name)"
-	$(PYTHON) -m detection_rules kibana-commit
+	$(PYTHON) -m detection_rules dev kibana-commit

README.md

@@ -55,22 +55,23 @@ Usage: detection_rules [OPTIONS] COMMAND [ARGS]...
   Commands for detection-rules repository.
 
 Options:
-  -h, --help  Show this message and exit.
+  -d, --debug / -n, --no-debug  Print full exception stacktrace on errors
+  -h, --help                    Show this message and exit.
 
 Commands:
-  build-release         Assemble all the rules into Kibana-ready release files.
-  create-rule           Create a new rule TOML file.
-  es                    Helper commands for integrating with Elasticsearch.
-  kibana-diff           Diff rules against their version represented in...
-  load-from-file        Load rules from file(s).
-  mass-update           Update multiple rules based on eql results.
-  rule-search           Use EQL to search the rules.
-  test                  Run unit tests over all of the rules.
-  toml-lint             Cleanup files with some simple toml formatting.
-  update-lock-versions  Update rule hashes in version.lock.json file...
-  validate-all          Check if all rules validates against a schema.
-  validate-rule         Check if a rule staged in rules dir validates...
-  view-rule             View an internal rule or specified rule file.
+  create-rule     Create a detection rule.
+  dev             Commands for development and management by internal...
+  es              Commands for integrating with Elasticsearch.
+  import-rules    Import rules from json, toml, or Kibana exported rule...
+  kibana          Commands for integrating with Kibana.
+  mass-update     Update multiple rules based on eql results.
+  normalize-data  Normalize Elasticsearch data timestamps and sort.
+  rule-search     Use KQL or EQL to find matching rules.
+  test            Run unit tests over all of the rules.
+  toml-lint       Cleanup files with some simple toml formatting.
+  validate-all    Check if all rules validates against a schema.
+  validate-rule   Check if a rule staged in rules dir validates against a...
+  view-rule       View an internal rule or specified rule file.
 ```
 
 The [contribution guide](CONTRIBUTING.md) describes how to use the `create-rule` and `test` commands to create and test a new rule when contributing to Detection Rules.

detection_rules/__init__.py

@@ -3,8 +3,10 @@
 # you may not use this file except in compliance with the Elastic License.
 
 """Detection rules."""
+from . import devtools
 from . import docs
 from . import eswrap
+from . import kbwrap
 from . import main
 from . import mappings
 from . import misc
@@ -14,8 +16,10 @@
 from . import utils
 
 __all__ = (
+    'devtools',
     'docs',
     'eswrap',
+    'kbwrap',
     'mappings',
     "main",
     'misc',