Metrics collector fails to create watcher

[gigabyte132]

What happened?

I tried running the example enas-cpu experiment with a StdOut collector and the experiment fails to run due to an error in the metrics-collector container

2024/09/24 13:19:08 FATAL -- failed to create Watcher
goroutine 18 [running]:
runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:26 +0x5e
github.com/nxadm/tail/util.Fatal({0xe14a9b?, 0xc000282000?}, {0x0, 0x0, 0x0})
        /go/pkg/mod/github.com/nxadm/[email protected]/util/util.go:23 +0x8b
github.com/nxadm/tail/watch.(*InotifyTracker).run(0xc0002b6000)
        /go/pkg/mod/github.com/nxadm/[email protected]/watch/inotify_tracker.go:220 +0x68
created by github.com/nxadm/tail/watch.init.func1 in goroutine 17
        /go/pkg/mod/github.com/nxadm/[email protected]/watch/inotify_tracker.go:55 +0x14e 

There is a related issue to this #1769 . Since then katib has migrated from using the hpcloud library for tailing to nxadm but it seems like I ran into the same exact issue regardless. This is with the following version of the metrics collector image https://hub.docker.com/layers/kubeflowkatib/file-metrics-collector/v1beta1-867c40a/images/sha256-3ab68e0932dd6c2028592dd7a7443ba4970e54f91ab145d6d35828112780eb0a?context=explore as the change to nxadm wasn't included in the 0.17 release. I have tried both the 0.16 and 0.17 images as well, but the result was the same. I haven't had more time to debug this more in depth (e.g building my own image with extra logs, etc).

What did you expect to happen?

The metrics collector should work normally, I have tried using the File metrics collector and things seem to be fine, although I haven't managed to get any Katib Experiment of any kind working with the StdOut one.

Environment

Kubernetes version:

$ kubectl version
Client Version: v1.29.6
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.2

Katib controller version:

$ kubectl get pods -n kubeflow -l katib.kubeflow.org/component=controller -o jsonpath="{.items[*].spec.containers[*].image}"
kubeflow/kubeflowkatib/katib-controller:v0.16.0

Katib Python SDK version:

$ pip show kubeflow-katib
Name: kubeflow-katib
Version: 0.17.0

Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

[andreyvelich]

Thanks for creating this issue @gigabyte132!
@tariq-hasan @Electronic-Waste Please can you help us to explore this issue ?

/remove-label lifecycle/needs-triage
/area backend

[Electronic-Waste]

I can't reproduce the result with https://github.com/kubeflow/katib/blob/master/examples/v1beta1/nas/enas-cpu.yaml

It turned out to be completed successfully in my environment:

$ kubectl get experiment -n kubeflow 
NAME       TYPE        STATUS   AGE
enas-cpu   Succeeded   True     27m

@gigabyte132 For your reference, my setup environment is:

Kuberentes version:

$ kubectl version
Client Version: v1.30.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.1

Katib controller version:

$ kubectl get pods -n kubeflow -l katib.kubeflow.org/component=controller -o jsonpath="{.items[*].spec.containers[*].image}"
docker.io/kubeflowkatib/katib-controller:latest

Katib Python SDK version:

$ pip show kubeflow-katib
Name: kubeflow-katib
Version: 0.17.0

Maybe you can upgrade the version of katib-controller to latest and try it again?

cc @andreyvelich @tariq-hasan

[tariq-hasan]

One other thing I was just curious about is if @gigabyte132 saw this error only for the enas-cpu experiment or if the error was also seen for other experiments such as darts-cpu and file-metrics-collector.

[gigabyte132]

@tariq-hasan for me, any type of experiment that uses the file-metrics-collector fails with this error