Create a template system for the graphite serializer

closes influxdata#925
closes influxdata#879

CHANGELOG.md

@@ -2,6 +2,11 @@
 
 ### Release Notes
 - Breaking change in the dovecot input plugin. See Features section below.
+- Graphite output templates are now supported. See
+https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md#graphite
+- Possible breaking change for the librato and graphite outputs. Telegraf will
+no longer insert field names when the field is simply named `value`. This is
+because the `value` field is redundant in the graphite/librato context.
 
 ### Features
 - [#976](https://github.com/influxdata/telegraf/pull/976): Reduce allocations in the UDP and statsd inputs.
@@ -11,6 +16,7 @@
 - [#943](https://github.com/influxdata/telegraf/pull/943): http_response input plugin. Thanks @Lswith!
 - [#939](https://github.com/influxdata/telegraf/pull/939): sysstat input plugin. Thanks @zbindenren!
 - [#998](https://github.com/influxdata/telegraf/pull/998): **breaking change** enabled global, user and ip queries in dovecot plugin. Thanks @mikif70!
+- [#1001](https://github.com/influxdata/telegraf/pull/1001): Graphite serializer templates.
 
 ### Bugfixes
 - [#968](https://github.com/influxdata/telegraf/issues/968): Processes plugin gets unknown state when spaces are in (command name)

docs/DATA_FORMATS_OUTPUT.md

@@ -1,5 +1,11 @@
 # Telegraf Output Data Formats
 
+Telegraf is able to serialize metrics into the following output data formats:
+
+1. [InfluxDB Line Protocol](https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md#influx)
+1. [JSON](https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md#json)
+1. [Graphite](https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md#graphite)
+
 Telegraf metrics, like InfluxDB
 [points](https://docs.influxdata.com/influxdb/v0.10/write_protocols/line/),
 are a combination of four basic parts:
@@ -29,8 +35,7 @@ config option, for example, in the `file` output plugin:
   ## Files to write to, "stdout" is a specially handled file.
   files = ["stdout"]
 
-  ## Data format to output. 
-
+  ## Data format to output.
   ## Each data format has it's own unique set of configuration options, read
   ## more about them here:
   ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md
@@ -42,35 +47,46 @@ config option, for example, in the `file` output plugin:
 Each data_format has an additional set of configuration options available, which
 I'll go over below.
 
-## Influx:
+# Influx:
 
 There are no additional configuration options for InfluxDB line-protocol. The
 metrics are serialized directly into InfluxDB line-protocol.
 
-#### Influx Configuration:
+### Influx Configuration:
 
 ```toml
 [[outputs.file]]
   ## Files to write to, "stdout" is a specially handled file.
   files = ["stdout", "/tmp/metrics.out"]
 
-  ## Data format to output. 
-
+  ## Data format to output.
   ## Each data format has it's own unique set of configuration options, read
   ## more about them here:
   ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md
   data_format = "influx"
 ```
 
-## Graphite:
+# Graphite:
 
-The Graphite data format translates Telegraf metrics into _dot_ buckets.
-The format is:
+The Graphite data format translates Telegraf metrics into _dot_ buckets. A
+template can be specified for the output of Telegraf metrics into Graphite
+buckets. The default template is:
 
 ```
-[prefix].[host tag].[all tags (alphabetical)].[measurement name].[field name] value timestamp
+template = "host.tags.measurement.field"
 ```
 
+In the above template, we have four parts:
+
+1. _host_ is a tag key. This can be any tag key that is in the Telegraf
+metric(s). If the key doesn't exist, it will be ignored. If it does exist, the
+tag value will be filled in.
+1. _tags_ is a special keyword that outputs all remaining tag values, separated
+by dots and in alphabetical order (by tag key). These will be filled after all
+tag keys are filled.
+1. _measurement_ is a special keyword that outputs the measurement name.
+1. _field_ is a special keyword that outputs the field name.
+
 Which means the following influx metric -> graphite conversion would happen:
 
 ```
@@ -80,28 +96,28 @@ tars.cpu-total.us-east-1.cpu.usage_user 0.89 1455320690
 tars.cpu-total.us-east-1.cpu.usage_idle 98.09 1455320690
 ```
 
-`prefix` is a configuration option when using the graphite output data format.
-
-#### Graphite Configuration:
+### Graphite Configuration:
 
 ```toml
 [[outputs.file]]
   ## Files to write to, "stdout" is a specially handled file.
   files = ["stdout", "/tmp/metrics.out"]
 
-  ## Data format to output. 
-
+  ## Data format to output.
   ## Each data format has it's own unique set of configuration options, read
   ## more about them here:
   ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md
-  data_format = "influx"
+  data_format = "graphite"
 
+  # prefix each graphite bucket
   prefix = "telegraf"
+  # graphite template
+  template = "host.tags.measurement.field"
 ```
 
-## Json:
+# JSON:
 
-The Json data format serialized Telegraf metrics in json format. The format is:
+The JSON data format serialized Telegraf metrics in json format. The format is:
 
 ```json
 {
@@ -119,15 +135,14 @@ The Json data format serialized Telegraf metrics in json format. The format is:
 }
 ```
 
-#### Json Configuration:
+### JSON Configuration:
 
 ```toml
 [[outputs.file]]
   ## Files to write to, "stdout" is a specially handled file.
   files = ["stdout", "/tmp/metrics.out"]
 
-  ## Data format to output. 
-
+  ## Data format to output.
   ## Each data format has it's own unique set of configuration options, read
   ## more about them here:
   ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md

etc/telegraf.conf

@@ -178,6 +178,9 @@
 #   servers = ["localhost:2003"]
 #   ## Prefix metrics name
 #   prefix = ""
+#   ## Graphite output template
+#   ## see https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md
+#   template = "host.tags.measurement.field"
 #   ## timeout in seconds for the write connection to graphite
 #   timeout = 2
 
@@ -251,22 +254,20 @@
 # [[outputs.librato]]
 #   ## Librator API Docs
 #   ## http://dev.librato.com/v1/metrics-authentication
-# 
 #   ## Librato API user
 #   api_user = "[email protected]" # required.
-# 
 #   ## Librato API token
 #   api_token = "my-secret-token" # required.
-# 
-#   ### Debug
+#   ## Debug
 #   # debug = false
-# 
-#   ### Tag Field to populate source attribute (optional)
-#   ### This is typically the _hostname_ from which the metric was obtained.
+#   ## Tag Field to populate source attribute (optional)
+#   ## This is typically the _hostname_ from which the metric was obtained.
 #   source_tag = "host"
-# 
 #   ## Connection timeout.
 #   # timeout = "5s"
+#   ## Output Name Template (same as graphite buckets)
+#   ## see https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md#graphite
+#   template = "host.tags.measurement.field"
 
 
 # # Configuration for MQTT server to send metrics to

internal/config/config.go

@@ -850,8 +850,17 @@ func buildSerializer(name string, tbl *ast.Table) (serializers.Serializer, error
 		}
 	}
 
+	if node, ok := tbl.Fields["template"]; ok {
+		if kv, ok := node.(*ast.KeyValue); ok {
+			if str, ok := kv.Value.(*ast.String); ok {
+				c.Template = str.Value
+			}
+		}
+	}
+
 	delete(tbl.Fields, "data_format")
 	delete(tbl.Fields, "prefix")
+	delete(tbl.Fields, "template")
 	return serializers.NewSerializer(c)
 }
 

plugins/outputs/graphite/README.md

@@ -1,13 +1,34 @@
 # Graphite Output Plugin
 
-This plugin writes to [Graphite](http://graphite.readthedocs.org/en/latest/index.html) via raw TCP.
+This plugin writes to [Graphite](http://graphite.readthedocs.org/en/latest/index.html)
+via raw TCP.
+
+## Configuration:
+
+```toml
+# Configuration for Graphite server to send metrics to
+[[outputs.graphite]]
+  ## TCP endpoint for your graphite instance.
+  servers = ["localhost:2003"]
+  ## Prefix metrics name
+  prefix = ""
+  ## Graphite output template
+  ## see https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md
+  template = "host.tags.measurement.field"
+  ## timeout in seconds for the write connection to graphite
+  timeout = 2
+```
 
 Parameters:
 
-    Servers []string
-    Prefix  string
-    Timeout int
+    Servers  []string
+    Prefix   string
+    Timeout  int
+    Template string
 
 * `servers`: List of strings, ["mygraphiteserver:2003"].
 * `prefix`: String use to prefix all sent metrics.
-* `timeout`: Connection timeout in second.
+* `timeout`: Connection timeout in seconds.
+* `template`: Template for graphite output format, see
+https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md
+for more details.

plugins/outputs/graphite/graphite.go

@@ -16,17 +16,21 @@ import (
 
 type Graphite struct {
 	// URL is only for backwards compatability
-	Servers []string
-	Prefix  string
-	Timeout int
-	conns   []net.Conn
+	Servers  []string
+	Prefix   string
+	Template string
+	Timeout  int
+	conns    []net.Conn
 }
 
 var sampleConfig = `
   ## TCP endpoint for your graphite instance.
   servers = ["localhost:2003"]
   ## Prefix metrics name
   prefix = ""
+  ## Graphite output template
+  ## see https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md
+  template = "host.tags.measurement.field"
   ## timeout in seconds for the write connection to graphite
   timeout = 2
 `
@@ -72,7 +76,7 @@ func (g *Graphite) Description() string {
 func (g *Graphite) Write(metrics []telegraf.Metric) error {
 	// Prepare data
 	var bp []string
-	s, err := serializers.NewGraphiteSerializer(g.Prefix)
+	s, err := serializers.NewGraphiteSerializer(g.Prefix, g.Template)
 	if err != nil {
 		return err
 	}

plugins/outputs/graphite/graphite_test.go

@@ -54,7 +54,7 @@ func TestGraphiteOK(t *testing.T) {
 	m1, _ := telegraf.NewMetric(
 		"mymeasurement",
 		map[string]string{"host": "192.168.0.1"},
-		map[string]interface{}{"mymeasurement": float64(3.14)},
+		map[string]interface{}{"myfield": float64(3.14)},
 		time.Date(2010, time.November, 10, 23, 0, 0, 0, time.UTC),
 	)
 	m2, _ := telegraf.NewMetric(
@@ -90,10 +90,10 @@ func TCPServer(t *testing.T, wg *sync.WaitGroup) {
 	reader := bufio.NewReader(conn)
 	tp := textproto.NewReader(reader)
 	data1, _ := tp.ReadLine()
-	assert.Equal(t, "my.prefix.192_168_0_1.mymeasurement 3.14 1289430000", data1)
+	assert.Equal(t, "my.prefix.192_168_0_1.mymeasurement.myfield 3.14 1289430000", data1)
 	data2, _ := tp.ReadLine()
-	assert.Equal(t, "my.prefix.192_168_0_1.mymeasurement.value 3.14 1289430000", data2)
+	assert.Equal(t, "my.prefix.192_168_0_1.mymeasurement 3.14 1289430000", data2)
 	data3, _ := tp.ReadLine()
-	assert.Equal(t, "my.prefix.192_168_0_1.my_measurement.value 3.14 1289430000", data3)
+	assert.Equal(t, "my.prefix.192_168_0_1.my_measurement 3.14 1289430000", data3)
 	conn.Close()
 }

plugins/outputs/librato/librato.go

@@ -21,6 +21,7 @@ type Librato struct {
 	NameFromTags bool
 	SourceTag    string
 	Timeout      internal.Duration
+	Template     string
 
 	apiUrl string
 	client *http.Client
@@ -29,22 +30,20 @@ type Librato struct {
 var sampleConfig = `
   ## Librator API Docs
   ## http://dev.librato.com/v1/metrics-authentication
-
   ## Librato API user
   api_user = "[email protected]" # required.
-
   ## Librato API token
   api_token = "my-secret-token" # required.
-
-  ### Debug
+  ## Debug
   # debug = false
-
-  ### Tag Field to populate source attribute (optional)
-  ### This is typically the _hostname_ from which the metric was obtained.
+  ## Tag Field to populate source attribute (optional)
+  ## This is typically the _hostname_ from which the metric was obtained.
   source_tag = "host"
-
   ## Connection timeout.
   # timeout = "5s"
+  ## Output Name Template (same as graphite buckets)
+  ## see https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md#graphite
+  template = "host.tags.measurement.field"
 `
 
 type LMetrics struct {
@@ -152,17 +151,13 @@ func (l *Librato) Description() string {
 	return "Configuration for Librato API to send metrics to."
 }
 
-func (l *Librato) buildGaugeName(m telegraf.Metric, fieldName string) string {
-	// Use the GraphiteSerializer
-	graphiteSerializer := graphite.GraphiteSerializer{}
-	return graphiteSerializer.SerializeBucketName(m, fieldName)
-}
-
 func (l *Librato) buildGauges(m telegraf.Metric) ([]*Gauge, error) {
 	gauges := []*Gauge{}
+	serializer := graphite.GraphiteSerializer{Template: l.Template}
+	bucket := serializer.SerializeBucketName(m.Name(), m.Tags())
 	for fieldName, value := range m.Fields() {
 		gauge := &Gauge{
-			Name:        l.buildGaugeName(m, fieldName),
+			Name:        graphite.InsertField(bucket, fieldName),
 			MeasureTime: m.Time().Unix(),
 		}
 		if !gauge.verifyValue(value) {