Whether the hyperparameter search algorithm will refer to the value of additionalMetricNames

[kldancer]

If not, is the purpose of the additionalMetricNames parameter just for visualization? It will not affect the experimental results？
Please help me answer this question, thank you very much !!!🙏

[andreyvelich]

That is correct @YKL2436542696

Currently, additionalMetricNames is only used for metrics tracking purposes. So you can get them in the Katib UI or using Katib SDK.
We have an open issue to support multi-objective optimization, but it has not been implemented yet: #1549

[kldancer]

@andreyvelich Thank you。 Can I understand it this way? The strategies for additionalMetricNames in Spec.Objective.MetricStrategies currently have no practical effect. (This may work when multi-objective optimization is supported in the future)

[kldancer]

And I noticed that the metrics values output by the "/katib/fetch_hp_job_info/" interface in the katib-ui module are based on spec.Objective.Type and have nothing to do with Spec.Objective.MetricStrategies。
I'm curious why the metric values are not output according to the strategy defined by Spec.Objective.MetricStrategies 🤯

[andreyvelich]

The strategies for additionalMetricNames in Spec.Objective.MetricStrategies currently have no practical effect.

That's right, we check if Experiment goal is reached based on objective metric name and its metrics strategy: https://github.com/kubeflow/katib/blob/master/pkg/controller.v1beta1/experiment/util/status_util.go#L166

However, we send those metrics to the suggestion service based on the strategies: https://github.com/kubeflow/katib/blob/master/pkg/controller.v1beta1/suggestion/suggestionclient/suggestionclient.go#L419.
So if user implements custom algorithm service that analyses all Trial metrics that might give some value.

[andreyvelich]

I'm curious why the metric values are not output according to the strategy defined by Spec.Objective.MetricStrategies

It's a good point, actually UI should just take Trial metrics results from .status.observation and based on strategies should take the appropriate value, not from metrics logs as we do right now: https://github.com/kubeflow/katib/blob/master/pkg/ui/v1beta1/hp.go#L156-L166
Like what we do for Suggestion Service.
Thanks for reporting!

/bug
/area frontend
/good-first-issue

[google-oss-prow]

@andreyvelich:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

I'm curious why the metric values are not output according to the strategy defined by Spec.Objective.MetricStrategies

It's a good point, actually UI should show just take Trial metrics results from .status.observation, not from metrics logs as we do right now: https://github.com/kubeflow/katib/blob/master/pkg/ui/v1beta1/hp.go#L156-L166
Thanks for reporting!

/bug
/area frontend
/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[andreyvelich]

/kind bug

[live2awesome]

i am interested in solving this
/assign

[xr-dev-saurabh]

/assign

[andreyvelich]

Thank you for your interest @xr-dev-saurabh, as I can see @live2awesome is already working on this issue.
Please feel free to work on other issues that don't have assignee
/assign @live2awesome

[xr-dev-saurabh]

/unassign

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[andreyvelich]

@xr-dev-saurabh @live2awesome Do you still want to work on this issue ?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.