Track progress for Kubernetes 1.8 / kubernetes-kafka v3.0.0

[solsson]

Manifests:

• Update to 1.8 workload API versions
  • 8e2de05
  • 03ddbf1
• Storage Retain (Use k8s 1.8 storage class syntax and set Retain by default #85)
• Retain for minikube too

Test open PRs:

• Zookeeper liveness (Add Zookeeper liveness probes #76, d0a5088)
• metrics-improve-scrape-times (f89bd74)
  • test with Exemplify use of prometheus-operator kubernetes-assert#4
• Remove overrides in manifest command; use property files #77 remove overrides (any votes there?)
  • if not the whole PR, maybe ed9fe4e as fix for server properties not being changed #72?
• kafka-manager Adds yahoo/kafka-manger #83 + 03ddbf1
• kubernetes events to kafka (Addon: Stream all Kubernetes events to a topic #60) 1.8-kube-events

Test tests:

• kafkacat based, delete brokers etc
• java based, delete brokers etc

Structure:

• Addons -> master, selected using kubectl apply [folder]
  • See "1.8-" branches like 1.8-logs-streaming
  • See label v3.1

[solsson]

• rename manifests from NNname.yml to NN-name.yml?

[solsson]

Found an explanation for why selector: needs to be added to workload API manifests:
From daemonset but I guess it applies to other resources as well:
"The pod selector will no longer be defaulted when left empty. Selector defaulting was not compatible with kubectl apply."

[solsson]

An update on testing, based on #79 (comment). I lack the Kafka experiment to interpret current results, so I think I want 3 tests that do essentially the same but with different clients.

What we want to assert is basically "uptime" in the face of re-scheduled broker and zk pods, caused by things like node downtime, cluster upgrades or zone outages. Tests continuously "bootstrap" + consume from a topic with 2 replicas (b8bfda8) + regularly produce messages + assert that those messages get consumed.

Measurements of throughput etc will have to wait until we have Prometheus monitoring up and running (#49 + Yolean/kubernetes-monitoring + ServiceMonitors + rules).

The three tests are:

• The kafka image's console producer + consumer (test-produce-consume)
• Kafkacat (Update tests to act as long-running consumers #79)
• https://github.com/Yolean/kafka-test-failover (i.e. the java client with some typical usage pattern/config)

As a complement it'll be interesting to have kafkacat with new bootstrap for every assert run, i.e. https://github.com/Yolean/kubernetes-kafka/blob/master/test/basic-with-kafkacat.yml prior to #79)

[solsson]

Killing pods and watching test readiness I tend to need a tab that does a human readable variant of #60. My new favorite oneliner 😄 (with alias from solsson/kubectx#1 (comment)):

while :; do k get pods --all-namespaces -w; done | gawk '{ print strftime("%FT%T"), $0; fflush() }'

[solsson]

Tests have been put into practice in #79 (comment)

Huge improvement that we don't spin up 2 JVMs every 10s :)

[solsson]

Closing this messy ticket in favor of https://github.com/Yolean/kubernetes-kafka/milestone/1, with scope reductions -> https://github.com/Yolean/kubernetes-kafka/milestone/2.