README.md

Query TXN2 data by Account, Model and index pattern. Save a Query and execute a saved Query.

Configuration

Configuration is inherited from txn2/micro. The following configuration is specific to query:

Flag	Environment Variable	Description
-esServer	ELASTIC_SERVER	Elasticsearch Server (default "http://elasticsearch:9200")
-provisionServer	PROVISION_SERVER	Provision Server (txn2/provision)
-authCache	AUTH_CACHE	Seconds to cache key (BasicAuth) authentication.

Routes

Method	Route Pattern	Description
POST	run/:account	Run a query to test it. This action does not save the query.
GET	exec/:account/:id	Executes a saved query.
GET	system/exec/:account/:id	Executes a saved system query.
POST	upsert/:account	Upsert (save) a query.
GET	get/:account/:id	Get a saved query.
POST	search/:account	Search for saved queries.

Local Development

The project includes a docker-compose.yml file with Elasticsearch, Logstash, Kibana, Cerebro, txn2/rxtx, txn2/rtbeat, txn2/provision and txn2/tm:

docker-compose up

Add test account with txn2/provision:

curl -X POST \
  http://localhost:8070/account \
  -H 'Content-Type: application/json' \
  -d '{
    "id": "test",
    "description": "This is a test account",
    "display_name": "Test Organization",
    "active": true,
    "access_keys": [
        {
            "name": "test",
            "key": "PDWgYr3bQGNoLptBRDkLTGQcRmCMqLGRFpXoXJ8xMPsMLMg3LHvWpJgDu2v3LYBA",
            "description": "Generic access key 2",
            "active": true
        }
    ],
    "modules": [
        "wx",
        "data_science"
    ]
}'

Add an admin User with with txn2/provision given access to the test account:

curl -X POST \
  http://localhost:8070/user \
  -H 'Content-Type: application/json' \
  -d '{
	"id": "test",
	"description": "Test User admin",
	"display_name": "Test User",
	"active": true,
	"sysop": false,
	"password": "eWidL7UtiWJABHgn8WA",
	"sections_all": false,
	"sections": ["api", "config", "data"],
	"accounts": ["test"],
	"admin_accounts": ["test"]
}'

Get a user Token from txn2/provision with the User's id and password:

TOKEN=$(curl -s -X POST \
          http://localhost:8070/authUser?raw=true \
          -d '{
        	"id": "test",
        	"password": "eWidL7UtiWJABHgn8WA"
        }') && echo $TOKEN

Insert a sample Model into the test account using txn2/tm running on port 8085:

curl -X POST http://localhost:8085/model/test \
 -H "Authorization: Bearer $TOKEN" \
 -H 'Content-Type: application/json' \
 -d '{
  "machine_name": "some_metrics",
  "display_name": "Some Metrics",
  "description": "A sample model describing some metrics sent through rxtx",
  "fields": [
    {
      "machine_name": "device_id",
      "display_name": "Device ID",
      "data_type": "keyword"
    },
    {
      "machine_name": "random_number",
      "display_name": "Random Number",
      "data_type": "integer"
    },
    {
      "machine_name": "another_number",
      "display_name": "Another Number",
      "data_type": "integer"
    }
  ]
}'

Within Elasticsearch there is now a template _template/test-data-some_metrics for the account test describing txn2/rxtx/txn2/rtbeat inbound data matching index pattern test-data-some_metrics-*. Send some sample data to Elasticsearch through txn2/rxtx and wait for the batch interval (specified in the docker-compose) to complete:

curl -X POST \
  http://localhost:8090/rx/test/some_metrics/sample-data \
  -H 'Content-Type: application/json' \
  -d "{
  \"device_id\": \"12345\",
  \"random_number\": \"${RANDOM}\",
  \"another_number\": 12345
}"

The fields in the data sent to txn2/rxtx should match the fields described in the txn2/tm Model. Although the value for "random_number" is represented here as a string, the template mapping added with the Model instructs Elasticsearch to index it as an integer.

Example Queries

Run query from source. Configure it to use the services running from docker-compose above.

go run ./cmd/query.go --esServer=http://localhost:9200 --provisionServer=http://localhost:8070 --tokenKey="somesharedkey"

Run a Query:

curl -X POST \
  http://localhost:8080/run/test \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "machine_name": "count_some_metrics",
    "display_name": "Get all records from the some_metrics index.",
    "description": "Return all matches",
    "model": "some_metrics",
    "idx_pattern": "-ts-*",
    "query": {
      "size": 0,
	  "query": {
	    "match_all": {}
	  }
	}
}'

Upsert a Query:

(must have admin access to account)

curl -X POST \
  http://localhost:8080/upsert/test \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "machine_name": "count_some_metrics",
    "display_name": "Get all records from the some_metrics index.",
    "description": "Return all matches",
    "model": "some_metrics",
    "idx_pattern": "-ts-*",
    "query": {
      "size": 0,
	  "query": {
	    "match_all": {}
	  }
	}
}'

Search for queries:

curl -X POST \
  http://localhost:8080/search/test \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
  "size": 10,
  "query": {
    "match_all": {}
  }
}'

Get a Query:

curl -X GET \
  http://localhost:8080/get/test/count_some_metrics \
  -H "Authorization: Bearer $TOKEN"

Execute a Query:

curl -X GET \
  http://localhost:8080/exec/test/count_some_metrics \
  -H "Authorization: Bearer $TOKEN"

Execute a Query with BasicAuth using an Account AccessKey:

curl -X GET \
  http://localhost:8080/exec/test/count_some_metrics \
  -H 'Authorization: Basic dGVzdDpQRFdnWXIzYlFHTm9McHRCUkRrTFRHUWNSbUNNcUxHUkZwWG9YSjh4TVBzTUxNZzNMSHZXcEpnRHUydjNMWUJB'

Query Templates

Query templates use Go's build in template system along with Sprig. The following example tests a query with /run and demonstrates a query string override of default values provided in the "parameters" section:

curl -X POST \
  'http://localhost:8080/run/test?size=0&idx=2019.%2A' \
  -H 'Authorization: Basic dGVzdDpQRFdnWXIzYlFHTm9McHRCUkRrTFRHUWNSbUNNcUxHUkZwWG9YSjh4TVBzTUxNZzNMSHZXcEpnRHUydjNMWUJB' \
  -d '{
    "machine_name": "get_some_metrics",
    "display_name": "Get records from the some_metrics index.",
    "description": "Get some_metrics",
    "model": "some_metrics",
    "idx_pattern": "-ts-{{ index . \"idx\" }}",
    "query_template": "{\"size\": {{ index . \"size\" }},\"query\": {\"match_all\": {}}}",
	"parameters": [
		{
			"machine_name": "size",
			"default_value": "1"
		},
		{
			"machine_name": "idx",
			"default_value": "2019.*"
		}
	]
}'

Release Packaging

Build test release:

goreleaser --skip-publish --rm-dist --skip-validate

Build and release:

GITHUB_TOKEN=$GITHUB_TOKEN goreleaser --rm-dist