Allow metrics to return tuple of scores #334
Conversation
| if isinstance(score_ret, tuple): | ||
| # The first score in the returned tuple will be used | ||
| # as primary score for this metric | ||
| end_scores[f"{name}_{score}"] = score_ret[0][1] |
There was a problem hiding this comment.
Not sure if this is the right PR to change this, but I hate this JSON schema I defined.
It looks like this:
"test": {
"test_score": 0.5070093274116516,
"test_loss": 11.01280689239502,
"test_event_onset_200ms_fms": 0.5070093274116516,
"test_segment_1s_er": 0.5021833777427673,
"validation_score": 0.4511459767818451,
"epoch": 470,
"time_in_min": 64.55239222049713
},
When I want to crawl over this JSON and find ALL the scores, I resort to the hacky trick of seeing if the string starts with "test" and isn't "test_score" or "test_loss". Kinda gross.
I would prefer ["test"]["test_scores"] = {"onset_fms": ..., ...} etc in addition to ["test"]["test_score"] and ["test"]["test_loss"].
Maybe we should make an issue for this?
| # If the returned score is a tuple, store each subscore as separate entry | ||
| if isinstance(score_ret, tuple): | ||
| # The first score in the returned tuple will be used | ||
| # as primary score for this metric |
There was a problem hiding this comment.
Wait. No. WTF.
We don't have a primary score per metric. We have a primary score AKA optimization criterion AKA target score for each task.
There was a problem hiding this comment.
by priamry score at the metric level, i meant the fms will be primary in any of the optimisation or will behave as the target. like the first one in the tuple will be primary
There was a problem hiding this comment.
added information for this in validate_score_return_type
| @@ -76,9 +111,23 @@ def __init__( | |||
| self.name = name | |||
| self.maximize = maximize | |||
|
|
|||
| def __call__(self, predictions: Any, targets: Any, **kwargs) -> float: | |||
| def __call__(self, *args, **kwargs) -> Union[Tuple[Tuple[str, float], ...], float]: | |||
| assert hasattr(self, "scores"), "Scores for the model should be defined" | ||
| end_scores = {} | ||
| # The first score in the first `self.scores` is the optimization criterion | ||
| for score in self.scores: |
There was a problem hiding this comment.
We probably only need to do this for the primary score, otherwise we are wasting time during validation.
No description provided.