Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix frame buffer bug where predecessors were not checked properly #155

Merged
merged 6 commits into from
Sep 16, 2024

Conversation

cmalinmayor
Copy link
Collaborator

@cmalinmayor cmalinmayor commented Aug 6, 2024

Proposed Change

@bentaculum I also saw issues with the frame buffer not updating when I expected it to. Upon closer inspection, there was a bug in how we were checking the successors. This PR fixes that bug. Sorry!

Previous discussion of possible frame buffer issue #100 (comment)

Types of Changes

What types of changes does your code introduce? Put an x in the boxes that apply.

  • Bugfix (non-breaking change which fixes an issue)
  • New feature or enhancement
  • Documentation update
  • Tests and benchmarks
  • Maintenance (e.g. dependencies, CI, releases, etc.)

Which topics does your change affect? Put an x in the boxes that apply.

  • Loaders
  • Matchers
  • Track Errors
  • Metrics
  • Core functionality (e.g. TrackingGraph, run_metrics, cli, etc.)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

  • I have read the developer/contributing docs.
  • I have added tests that prove that my feature works in various situations or tests the bugfix (if appropriate).
  • I have checked that I maintained or improved code coverage.
  • I have checked the benchmarking action to verify that my changes did not adversely affect performance.
  • I have written docstrings and checked that they render correctly in the Read The Docs build (created after the PR is opened).
  • I have updated the general documentation including Metric descriptions and example notebooks if necessary.

@msschwartz21 msschwartz21 marked this pull request as draft August 6, 2024 19:33
@msschwartz21
Copy link
Collaborator

I'm going to go in and fix the test case for this so that the tests would have actually caught this bug...

@msschwartz21 msschwartz21 self-assigned this Aug 6, 2024
@bentaculum
Copy link
Contributor

Nice catch! Can we generally describe the influence of this fix on the division metrics calculated up to this point? Are old metrics too optimistic/pessimistic?

@cmalinmayor
Copy link
Collaborator Author

@bentaculum The old version incorrectly penalized some (most?) correct divisions when frame_buffer > 0. Specifically, it checked if the successors had the same ID, not if the successors were matched, so if your prediction and candidate nodes had different IDs, it would never count an offset division as correct. For linking only experiments with GT detections, it probably won't change anything, but for experiments with predicted detections, performance should improve for division metrics when frame buffer > 0.

@DragaDoncila
Copy link
Collaborator

DragaDoncila commented Aug 19, 2024

@cmalinmayor @msschwartz21 took a look at this PR and if I'm understanding correctly, the bug was that when checking successors, we were directly using the graph node IDs rather than converting them via the mapper. And the test was passing because our division_graphs are using identical node IDs right? So if we just tweak the get_division_graphs util to use different node IDs that should be a sufficient test that fails on main and passes with this PR?

@msschwartz21
Copy link
Collaborator

@DragaDoncila that's correct! I'll fix the test case in the next week or so. It's an easy fix but life has been busy 😂

@DragaDoncila
Copy link
Collaborator

@msschwartz21 no rush at all, just making sure I understand the change!

* Change node ids on divisions graphs to be unique
* Update how mapper for division graphs is returned so that direction of gt to pred can be flipped
@msschwartz21 msschwartz21 marked this pull request as ready for review September 10, 2024 18:26
@msschwartz21 msschwartz21 added tests Test case related issues topic: metrics fix bug fix labels Sep 10, 2024
Copy link
Collaborator

@DragaDoncila DragaDoncila left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! Thanks for editing the tests @msschwartz21

@msschwartz21 msschwartz21 merged commit ba122c7 into main Sep 16, 2024
13 checks passed
@msschwartz21 msschwartz21 deleted the bugfix_frame_buffer branch September 16, 2024 19:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fix bug fix tests Test case related issues topic: metrics
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants