Skip to content

Latest commit

 

History

History
266 lines (207 loc) · 8.99 KB

README.md

File metadata and controls

266 lines (207 loc) · 8.99 KB
Raw



Oncology FM Evaluation Framework by kaiko.ai

PyPI docs license
paper

Installation • How To Use • Quick Start • Documentation • Datasets • Benchmarks
Contribute • Acknowledgements


eva is an evaluation framework for oncology foundation models (FMs) by kaiko.ai. Check out the documentation for more information.

Highlights:

  • Easy and reliable benchmark of Oncology FMs
  • Supports path-level classification, slide-level classification and semantic segmentation downstream tasks
  • Automatic embedding inference and evaluation of a downstream task
  • Native support of popular medical datasets and models
  • Produce statistics over multiple evaluation fits and multiple metrics

Installation

Simple installation from PyPI:

# to install the core version only
pip install kaiko-eva

# to install the expanded `vision` version
pip install 'kaiko-eva[vision]'

# to install everything
pip install 'kaiko-eva[all]'

To install the latest version of the main branch:

pip install "kaiko-eva[all] @ git+https://github.com/kaiko-ai/eva.git"

You can verify that the installation was successful by executing:

eva --version

How To Use

eva can be used directly from the terminal as a CLI tool as follows:

eva {fit,predict,predict_fit} --config url/or/path/to/the/config.yaml

eva uses jsonargparse to make it easily configurable by automatically generating command line interfaces (CLIs), which allows to call any Python object from the command line. Moreover, the configuration structure is always in sync with the code. Thus, eva can be used either directly from Python or as a CLI tool (recommended).

For more information, please refer to the documentation.

Learn about Configs

The following interfaces are identical:

Python interface Configuration file 
# main.py
# execute with: `python main.py`

from torch import nn

from eva import core
from eva.vision import datasets, transforms

# initialize trainer
trainer = core.Trainer(max_steps=100)

# initialize model
model = core.HeadModule(
  backbone=nn.Flatten(),
  head=nn.Linear(150528, 4),
  criterion=nn.CrossEntropyLoss(),
)

# initialize data
data = core.DataModule(
  datasets=core.DatasetsSchema(
    train=datasets.BACH(
      root="data/bach",
      split="train",
      download=True,
      transforms=transforms.ResizeAndCrop(),
    ),
  ),
  dataloaders=core.DataloadersSchema(
    train=core.DataLoader(batch_size=32),
  ),
)

# perform fit
pipeline = core.Interface()
pipeline.fit(trainer, model=model, data=data)
 
# main.yaml
# execute with: `eva fit --config main.yaml`

---
trainer:
  class_path: eva.Trainer
  init_args:
    max_steps: 100
model:
  class_path: eva.HeadModule
  init_args:
    backbone: torch.nn.Flatten
    head:
      class_path: torch.nn.Linear
      init_args:
        in_features: 150528
        out_features: 4
    criterion: torch.nn.CrossEntropyLoss
data:
  class_path: eva.DataModule
  init_args:
    datasets:
      train:
        class_path: eva.vision.datasets.BACH
        init_args:
          root: ./data/bach
          split: train
          download: true
          transforms: eva.vision.transforms.ResizeAndCrop
    dataloaders:
      train:
        batch_size: 32

The .yaml file defines the functionality of eva by parsing and translating its content to Python objects directly. Native supported configs can be found at the configs directory of the repo, which can be both locally stored or remote.

Quick Start

We define two types of evaluations: online and offline. While online fit uses the backbone (FM) to perform forward passes during the fitting process, offline fit first generates embeddings with the backbone and then fits the model using these embeddings as input, resulting in a faster evaluation.

Here are some examples to get you started:

  • Perform a downstream offline classification evaluation of DINO ViT-S/16 on the BACH dataset with linear probing by first inferring the embeddings and then performing 5 sequential fits:

    export DOWNLOAD_DATA=true
eva predict_fit --config https://raw.githubusercontent.com/kaiko-ai/eva/main/configs/vision/dino_vit/offline/bach.yaml

  • Perform a downstream online segmentation evaluation of DINO ViT-S/16 on the MoNuSAC dataset with the ConvDecoderMS decoder:

    export DOWNLOAD_DATA=true
eva fit --config https://raw.githubusercontent.com/kaiko-ai/eva/main/configs/vision/dino_vit/online/monusac.yaml

For more examples, take a look at the configs and tutorials.

Note

All the datasets that support automatic download in the repo have by default the option to automatically download set to false. For automatic download you have to manually set the environmental variable DOWNLOAD_DATA=true or in the configuration file download=true.

Leaderboards

The following table shows the FMs we have evaluated with eva. For more detailed information about the evaluation process, please refer to our documentation.

Pathology Leaderboard

Contributing

eva is an open source project and welcomes contributions of all kinds. Please checkout the developer and contributing guide for help on how to do so.

All contributors must follow the code of conduct.

Acknowledgements

Our codebase is built using multiple opensource contributions

python pytorch lightning
black isort Ruff Checked with pyright
pdm-managed Nox Built with Material for MkDocs

Citation

If you find this repository useful, please consider giving a star ⭐ and adding the following citation:

@inproceedings{kaiko.ai2024eva,
    title={eva: Evaluation framework for pathology foundation models},
    author={kaiko.ai and Ioannis Gatopoulos and Nicolas K{\"a}nzig and Roman Moser and Sebastian Ot{\'a}lora},
    booktitle={Medical Imaging with Deep Learning},
    year={2024},
    url={https://openreview.net/forum?id=FNBQOPj18N}
}