Cannot use Kubernetes DNS as URL

[Coolfeather2]

time="2017-10-16T11:45:46Z" level=warning msg="Task failed" _block=spin _module=scheduler-task consecutive failure limit=10 consecutive failures=6 error="rpc error: code = 2 desc = Error Connecting to graphite at grafana-backend.grafana:2003. Error: dial tcp: lookup grafana-backend.grafana on 169.254.169.254:53: no such host" task-id=dda70bc3-d1ec-4282-9c83-eb34fa8da251 task-name=Task-dda70bc3-d1ec-4282-9c83-eb34fa8da251

time="2017-10-16T11:45:56Z" level=error msg="error with publisher job" _module=scheduler-job block=run error="rpc error: code = 2 desc = Error Connecting to graphite at grafana-backend.grafana:2003. Error: dial tcp: lookup grafana-backend.grafana on 169.254.169.254:53: no such host" job-type=publisher plugin-config=map[port:{2003} prefix_tags:{} prefix:{snap.cluster-1.gke-cluster-1-default-pool-5a6d0c6d-f8f4} server:{grafana-backend.grafana}] plugin-name=graphite plugin-version=-1

time="2017-10-16T11:45:56Z" level=warning msg="Publish job failed" _block=submit-publish-job _module=scheduler-workflow parent-node-type=collector publish-name=graphite publish-version=-1 task-id=dda70bc3-d1ec-4282-9c83-eb34fa8da251 task-name=Task-dda70bc3-d1ec-4282-9c83-eb34fa8da251

My container and node dashboards have no data
also no cluster disk usage on cluster dashboard

[Coolfeather2]

changing the graphite write address to the service instead of dns name fixed the issue, but would still rather use dns for better deployment practice if it is possible

[woodsaj]

In order to be able to collect certain metrics, the agent runs with "hostNetwork: true". Due to using the hostNetwork , the pod running the agent will use the DNS servers defined in the hosts /etc/resolv.conf
And this typically doesnt list the address of the k8s internal DNS servers.

[woodsaj]

This is a DUP of grafana/kubernetes-app#38

[woodsaj]

However, looks like kubernetes/kubernetes#17406 has been resolved so we can probably address this.

Though as it would restrict the app to support k8s 1.6 or newer, i am not sure we should right now.

[Coolfeather2]

Okay, thanks for addressing this