Detail Document Needed to Setup consul-registration-hook

[ashishchalak]

Is there any detailed documentation available to setup consul-registration-hook with kubernetes. My requirement is to register PODS and its IP address with Consul.

[ashishchalak]

[myhost:/home/users/ashishrajshek/consul-registration-hook/cmd/consul-registration-hook]$ export KUBERNETES_SERVICE_HOST=10.73.28.78
[myhost:/home/users/ashishrajshek/consul-registration-hook/cmd/consul-registration-hook]$ export KUBERNETES_SERVICE_PORT=6443
[myhost:/home/users/ashishrajshek/consul-registration-hook/cmd/consul-registration-hook]$ ./consul-registration-hook register k8s
2020/07/14 05:43:55 consul-registration-hook (version: )
2020/07/14 05:43:55 Registering services using data from Kubernetes API
2020/07/14 05:43:55 error getting services to register: unable create K8S API client: couldn't initialize client: open /var/run/secrets/kubernetes.io/serviceaccount/namespace: no such file or directory

[ojagodzinski]

Are You running this command from inside of the running POD? In /var/run/secrets/kubernetes.io/serviceaccount/ should be default serviceaccount data as K8S documentation states:

https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/
https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/#accessing-the-api-from-a-pod

[ashishchalak]

Hi Oskar,

Thanks for your quick response. Let me check via POD and get back.

[ashishchalak]

What is the value I should be passing in GS_URL, KUBERNETES_POD_NAME, KUBERNETES_POD_NAMESPACE and HOST_IP in below yaml file.
I have a consul running on http://10.73.28.78:8500

apiVersion: v1
kind: Pod
metadata:
name: pod-with-consul-hook
labels:
consul: service-name
spec:
initContainers:

• name: hook-init-container
  image: google/cloud-sdk:alpine
  imagePullPolicy: Always
  command: ["/bin/sh"]
  args: ["-c", "gsutil cat ${GS_URL} | tar -C /hooks -zxvf -"]
  env:
  • name: GS_URL
    value: 'http://10.73.28.78:8500'
    volumeMounts:
  • name: hooks
    mountPath: /hooks
    containers:
• name: service-with-consul-hook-container
  image: python:2
  command: ["python", "-m", "SimpleHTTPServer", "8080"]
  env:
  • name: KUBERNETES_POD_NAME
    value: 'ashishpodname'
  • name: KUBERNETES_POD_NAMESPACE
    value: 'default'
  • name: HOST_IP
    value: 'http://10.73.28.78:8500'
  • name: CONSUL_HTTP_ADDR
    value: "$(HOST_IP):8500"
    ports:
  • containerPort: 8080
    volumeMounts:
  • name: hooks
    mountPath: /hooks
    lifecycle:
    postStart:
    exec:
    command: ["/bin/sh", "-c", "/hooks/consul-registration-hook register k8s"]
    preStop:
    exec:
    command: ["/bin/sh", "-c", "/hooks/consul-registration-hook deregister k8s"]
    volumes:
• name: hooks
  emptyDir: {}

[ojagodzinski]

error:

2020/07/14 05:43:55 error getting services to register: unable create K8S API client: couldn't initialize client: open /var/run/secrets/kubernetes.io/serviceaccount/namespace: no such file or directory

Has nothing to do with environment variables and their values. Problem lies in the lack of a serviceaccount directory inside POD. There (in POD where hook is executed) should be directory as below:

~$kubectl run my-shell --rm -i --tty --image ubuntu:latest -- bash
root@my-shell-67f75f7478-kds4r:/# ls -al /var/run/secrets/kubernetes.io/serviceaccount/
drwxr-xr-x 2 root root  100 Jul 14 00:00 ..2020_07_14_00_00_00
lrwxrwxrwx 1 root root   31 Jul 14 00:00 ..data -> ..2020_07_14_00_00_00
lrwxrwxrwx 1 root root   13 Jul 14 00:00 ca.crt -> ..data/ca.crt
lrwxrwxrwx 1 root root   16 Jul 14 00:00 namespace -> ..data/namespace
lrwxrwxrwx 1 root root   12 Jul 14 00:00 token -> ..data/token

[ashishchalak]

I also have access to Service Account directory

[myhost:/home/users/ashishrajshek]$ kubectl run my-shell --rm -i --tty --image ubuntu:latest -- bash
If you don't see a command prompt, try pressing enter.
root@my-shell:/# ls -al /var/run/secrets/kubernetes.io/serviceaccount/
total 0
drwxrwxrwt 3 root root 140 Jul 14 17:20 .
drwxr-xr-x 3 root root 28 Jul 14 17:21 ..
drwxr-xr-x 2 root root 100 Jul 14 17:20 ..2020_07_14_17_20_58.384039542
lrwxrwxrwx 1 root root 31 Jul 14 17:20 ..data -> ..2020_07_14_17_20_58.384039542
lrwxrwxrwx 1 root root 13 Jul 14 17:20 ca.crt -> ..data/ca.crt
lrwxrwxrwx 1 root root 16 Jul 14 17:20 namespace -> ..data/namespace
lrwxrwxrwx 1 root root 12 Jul 14 17:20 token -> ..data/token
root@my-shell:/#

[ashishchalak]

Getting this exception when trying to register via POD.

$kubectl describe pod pod-with-consul-hook

Events:
Type Reason Age From Message

---

Normal Scheduled default-scheduler Successfully assigned default/pod-with-consul-hook to myhost
Normal Pulled 12s kubelet, myhost Container image "python:2" already present on machine
Normal Created 12s kubelet, myhost Created container service-with-consul-hook-container
Normal Started 12s kubelet, myhost Started container service-with-consul-hook-container
Warning FailedPostStartHook 1s kubelet, myhost Exec lifecycle hook ([/bin/sh -c /hooks/hooks/consul-registration-hook register k8s]) for Container "service-with-consul-hook-container" in Pod "pod-with-consul-hook_default(51624215-6b8a-4d64-9790-cf0d0902fd1a)" failed - error: command '/bin/sh -c /hooks/hooks/consul-registration-
hook register k8s' exited with 1:
2020/07/14 18:00:56 consul-registration-hook (version: )
2020/07/14 18:00:56 Registering services using data from Kubernetes API
2020/07/14 18:01:05 unable to get pod data from API: unable to get pod data from API: kubernetes api: Failure 403 pods "pod-with-consul-hook" is forbidden: User "system:serviceaccount:default:default" cannot get resource "pods" in API group "" in the namespace "default"
2020/07/14 18:01:06 error getting services to register: unable to get pod data from API: could not get valid Pod data after 10s

[ojagodzinski]

2020/07/14 18:01:05 unable to get pod data from API: unable to get pod data from API: kubernetes api: Failure 403 pods "pod-with-consul-hook" is forbidden: User "system:serviceaccount:default:default" cannot get resource "pods" in API group "" in the namespace "default"

serviceaccount injected to POD must have broader permissions.

https://kubernetes.io/docs/reference/access-authn-authz/rbac/#service-account-permissions

[ashishchalak]

I elevated the permissions to admin role.

In POD Events i get the below exceptions:

Warning FailedPostStartHook 3m17s kubelet, myhost.com Exec lifecycle hook ([/bin/sh -c /hooks/hooks/consul-registration-hook register k8s]) for Container "service-with-consul-hook-container" in Pod "pod-with-consul-hook_default(624441b4-5fb8-432d-a72c-ab12a4015551)" failed - error: command '/bin/sh -c /hooks/hooks/consul-registration-hook register k8s' exited with 1:
2020/07/15 08:13:02 consul-registration-hook (version: )
2020/07/15 08:13:02 Registering services using data from Kubernetes API
2020/07/15 08:13:12 error getting services to register: unable to get pod data from API: could not get valid Pod data after 10s, message: "

---

When I try to run same command from inside the POD, Service is registered.

[myhost:/home/users/ashishrajshek/consul-registration-hook]$ kubectl exec -it pod-with-consul-hook sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.

/hooks/hooks/consul-registration-hook register k8s

2020/07/15 08:35:21 consul-registration-hook (version: )
2020/07/15 08:35:21 Registering services using data from Kubernetes API
2020/07/15 08:35:21 Won't include failure domain data in registration: failure domain labels don't exist
2020/07/15 08:35:21 no port configuration (PORT_DEFINITIONS)
2020/07/15 08:35:21 Found 1 services to register
2020/07/15 08:35:21 Registering "service-name" service in Consul agent

/hooks/hooks/consul-registration-hook deregister k8s

2020/07/15 08:35:46 consul-registration-hook (version: )
2020/07/15 08:35:46 Deregistering services using data from Kubernetes API
2020/07/15 08:35:46 Won't include failure domain data in registration: failure domain labels don't exist
2020/07/15 08:35:46 no port configuration (PORT_DEFINITIONS)
2020/07/15 08:35:46 Found 1 services to deregister
2020/07/15 08:35:46 Deregistering "10.244.0.212_8080" service in Consul agent

[ojagodzinski]

When I try to run same command from inside the POD, Service is registered.

So everything works, hook should be executed from inside of the POD.

[ashishchalak]

Are logs stored anywhere inside POD while registering and de registering services. If yes which location?

[ojagodzinski]

Hook prints logs to stdout/err.

[ashishchalak]

Any idea on this exception:

Warning FailedPostStartHook 40s (x3 over 66s) kubelet, myhost.com Exec lifecycle hook ([/bin/sh -c /hooks/hooks/consul-registration-hook register k8s]) for Container "cidtestserver" in Pod "cidtestserver-2_default(a56d696a-e3df-4a07-8d86-65a36e38f284)" failed - error: command '/bin/sh -c /hooks/hooks/consul-registration-hook register k8s' exited with 126: /bin/sh: /hooks/hooks/consul-registration-hook: Permission denied
, message: "/bin/sh: /hooks/hooks/consul-registration-hook: Permission denied\n"