Skip to content

v2.0.0

Latest
Latest
Compare
Choose a tag to compare
@mike-ferguson mike-ferguson released this 19 Aug 19:58
v2.0.0
ce5c28d
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

What's changed

We are thrilled to introduce Brain-Score 2.0, a major upgrade to the Brain-Score platform that empowers researchers, developers, and enthusiasts in the fields of neuroscience, cognitive science, and artificial intelligence to directly compare computational models of the brain with human and non-human primate behavior and brain recordings.

Here are the Brain-Score 2.0 features we are most excited about:

  1. Flexibility: new plugin system for easier contribution and collaboration: Brain-Score 2.0 introduces a modular plugin system that makes it easy to develop, combine, and contribute data, metrics, benchmarks, and models. Researchers can now more easily submit their own components and stress-test leading models or submit their own state-of-the-art model.
  2. Accessibility: enhanced website and documentation. Brain-Score’s web presence got a facelift. Our revamped website provides an improved user experience with more intuitive navigation, a simplified leaderboard, and detailed tutorials. We believe that making the platform easier to use is crucial for fostering collaboration and innovation.
  3. Language: broadening to a new domain. Brain-Score 2.0 introduces Brain-Score Language, which extends the Brain-Score platform from modeling the visual ventral stream to the human language system. The addition of Brain-Score Language reflects our philosophy that the Brain-Score approach to model benchmarking is fundamentally domain agnostic, and represents a step toward a future multimodal Brain-Score.

There are other quality of life changes that we hope will contribute to the platform's ease of use and sense of community, including:

  1. A centralized organization repo with clear purpose descriptions.
  2. A domain-independent, simplified way of scoring models, as seen here.
  3. Better error reporting for scoring runs, as well as automated Github merge functionality for user contributions via pull requests.
  4. A centralized repo of models to be easily accessible and runnable by any user.
  5. Enhanced FAQ and troubleshooting with the most common errors encountered, as well as their solutions.
  6. A dedicated Slack channel for the community, that is always accepting new members!

...and much more!

Looking Ahead

As Brain-Score grows, we hope that regular version updates and patch notes will be released on a bi-weekly schedule (once every two weeks), and major feature additions will be announced as they are completed. Normally these patch notes would include direct links to PRs or issues, but as the 2.0.0 update was so large, those are omitted in this release notes announcement.

The Brain-Score team has even more features and fixes on the horizon, including:

  1. Switching to a more stable and faster compute cluster in order to produce model scores in a reliable and efficient way. This should result in significantly less scoring errors and downtime.
  2. Updating all our code to support Python 3.11. This is one of our most popular requests, and we hope to have that done by the next release (2.1.0)
  3. Further improved user documentation, with clearer setup and troubleshooting for scoring models (including custom models).
  4. Adding new models to the model repos (we hope to have the top 100 models all included on Github within the next few releases).
  5. Other quality-of-life changes, including adding an option for downloading the leaderboard as a CSV file

Full Changelog: brain-score/vision@v1.2...v1.3

Assets 2
Loading