Skip to content

Commit

Permalink
update README for cRedAnno+
Browse files Browse the repository at this point in the history
  • Loading branch information
@ludles
ludles committed Oct 31, 2022
1 parent 4b32f4e commit 09a737f
Show file tree
Hide file tree
Showing 4 changed files with 683 additions and 406 deletions.
145 changes: 82 additions & 63 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,32 @@

# cRedAnno 🤏

[[`arXiv`](https://arxiv.org/abs/2206.13608)] [[`Poster`](https://ludles.github.io/assets/pdf/poster_credanno.pdf)] [[`Slides`](https://ludles.github.io/assets/pdf/slides_credanno_MICCAI2022.pdf)] [[`Dataset`](https://wiki.cancerimagingarchive.net/display/Public/LIDC-IDRI)]

**C**onsiderably **Red**ucing **Anno**tation Need in Self-Explanatory Models

- v2 (cRedAnno+):
[[`arXiv`](https://arxiv.org/abs/2210.16097)]

- v1 (cRedAnno):
[[`arXiv`](https://arxiv.org/abs/2206.13608)]
[[`Poster`](https://ludles.github.io/assets/pdf/poster_credanno.pdf)]
[[`Slides`](https://ludles.github.io/assets/pdf/slides_credanno_MICCAI2022.pdf)]

- [[`Dataset`](https://wiki.cancerimagingarchive.net/display/Public/LIDC-IDRI)]

------

## Method illustration

<div align="center">
<img src="./imgs/cRedAnno_plus_Intro_dark.svg" alt="cRedAnno_plus_Intro_dark" width="600px" />
</div>

## Performance overview

- Annotation reduction performance:
- Comparison between cRedAnno+ and cRedAnno:

<div align="center">
<img src="./imgs/anno_reduce.svg" alt="anno_reduce" width="400px" />
<img src="./imgs/anno_reduce+.svg" alt="anno_reduce" width="400px" />
</div>

- Prediction accuracy (%) of nodule attributes and malignancy:
Expand All @@ -41,73 +54,62 @@
<td colspan="9" style="text-align:left;">Full annotation</td>
</tr>
<tr>
<td style="text-align:left;">cRedAnno (50-NN)</td>
<td>94.93</td>
<td>92.72</td>
<td>95.58</td>
<td>93.76</td>
<td>91.29</td>
<td>92.72</td>
<td>94.67</td>
<td style="text-align:center;">87.52</td>
</tr>
<tr>
<td style="text-align:left;">cRedAnno (250-NN)</td>
<td>96.36</td>
<td>92.59</td>
<td>96.23</td>
<td>94.15</td>
<td>90.90</td>
<td>92.33</td>
<td>92.72</td>
<td style="text-align:center;">88.95</td>
<td style="text-align:left;">cRedAnno+</td>
<td>96.32 ±0.61</td>
<td>95.88 ±0.15</td>
<td>97.23 ±0.20</td>
<td>96.23 ±0.23</td>
<td>93.93 ±0.87</td>
<td>94.06 ±0.60</td>
<td>97.01 ±0.26</td>
<td style="text-align:center;">87.56 ±0.61</td>
</tr>
<tr>
<td style="text-align:left;">cRedAnno (trained)</td>
<td>95.84</td>
<td>95.97</td>
<td>97.40</td>
<td>96.49</td>
<td>94.15</td>
<td>94.41</td>
<td>97.01</td>
<td style="text-align:center;">88.30</td>
<td colspan="9" style="text-align:left;">Partial annotation</td>
</tr>
<tr>
<td colspan="9" style="text-align:left;">Partial annotation</td>
<td style="text-align:left;">cRedAnno (10%)</td>
<td>96.06 ±2.02</td>
<td>93.76 ±0.85</td>
<td>95.97 ±0.69</td>
<td>94.37 ±0.79</td>
<td>93.06 ±0.27</td>
<td>93.15 ±0.33</td>
<td>95.49 ±0.85</td>
<td style="text-align:center;">86.65 ±1.39</td>
</tr>
<tr>
<td style="text-align:left;">cRedAnno (10%, 50-NN)</td>
<td>94.93</td>
<td>92.07</td>
<td>96.75</td>
<td>94.28</td>
<td>92.59</td>
<td>91.16</td>
<td>94.15</td>
<td style="text-align:center;">87.13</td>
<td style="text-align:left;">cRedAnno+ (10%)</td>
<td>96.23 ±0.45</td>
<td>92.72 ±1.66</td>
<td>95.71 ±0.47</td>
<td>90.03 ±3.68</td>
<td>93.89 ±1.41</td>
<td>93.67 ±0.64</td>
<td>92.41 ±1.05</td>
<td style="text-align:center;">87.86 ±1.99</td>
</tr>
<tr>
<td style="text-align:left;">cRedAnno (10%, 150-NN)</td>
<td>95.32</td>
<td>89.47</td>
<td>97.01</td>
<td>93.89</td>
<td>91.81</td>
<td>90.51</td>
<td>92.85</td>
<td style="text-align:center;">88.17</td>
<td style="text-align:left;">cRedAnno (1%)</td>
<td>93.98 ±2.09</td>
<td>89.68 ±3.52</td>
<td>94.02 ±2.30</td>
<td>91.94 ±1.17</td>
<td>91.03 ±1.72</td>
<td>90.81 ±1.56</td>
<td>93.63 ±0.47</td>
<td style="text-align:center;">80.02 ±8.56</td>
</tr>
<tr>
<td style="text-align:left;">cRedAnno (1%, trained) 🤏</td>
<td>91.81</td>
<td>93.37</td>
<td>96.49</td>
<td>90.77</td>
<td>89.73</td>
<td>92.33</td>
<td>93.76</td>
<td style="text-align:center;">86.09</td>
<td style="text-align:left;">cRedAnno+ (1%)</td>
<td>95.84 ±0.34</td>
<td>92.67 ±1.24</td>
<td>95.97 ±0.45</td>
<td>91.03 ±4.65</td>
<td>93.54 ±0.87</td>
<td>92.72 ±1.19</td>
<td>92.67 ±1.50</td>
<td style="text-align:center;">86.22 ±2.51</td>
</tr>
</tbody>
</table>
Expand All @@ -131,7 +133,7 @@ Use [`extract_LIDC_IDRI_nodules.py`](./extract_LIDC_IDRI_nodules.py) to extract

### Training

#### Unsupervised feature extraction
#### 1. Unsupervised feature extraction

Following [DINO](https://github.com/facebookresearch/dino), to train on the extracted nodules:

Expand All @@ -141,8 +143,22 @@ python -m torch.distributed.launch --nproc_per_node=2 main_dino.py --arch vit_sm

The reported results start from the ImageNet-pretrained full weights provided for [`ViT-S/16`](https://dl.fbaipublicfiles.com/dino/dino_deitsmall16_pretrain/dino_deitsmall16_pretrain_full_checkpoint.pth), which should be put under `./logs/vits16_pretrain_full_2d_ann/`.

#### Supervised prediction
#### 2. Semi-supervised prediction

- Sparse seeding:

```bash
python eval_linear_joint_recycle.py --pretrained_weights ./logs/vits16_pretrain_full_2d_ann/checkpoint.pth --data_path /path_to_extracted_dir --output_dir ./logs/vits16_pretrain_full_2d_ann --label_frac 0.01 --lr 0.0005 --seed 42 --mode seed
```
- Semi-supervised active learning:

```bash
python eval_linear_joint_recycle.py --pretrained_weights ./logs/vits16_pretrain_full_2d_ann/checkpoint.pth --data_path /path_to_extracted_dir --output_dir ./logs/vits16_pretrain_full_2d_ann --label_frac 0.1 --lr 0.0005 --seed 42 --mode boost
```

<details>
<summary>v1 (click to expand)</summary>

To train the predictors:

```bash
Expand All @@ -155,6 +171,9 @@ or use the k-NN classifiers:
python eval_knn_joint.py --pretrained_weights ./logs/vits16_pretrain_full_2d_ann/checkpoint.pth --data_path /path_to_extracted_dir --output_dir ./logs/vits16_pretrain_full_2d_ann --label_frac 0.01
```

</details>


In both cases, `--label_frac` controls the used fraction of annotations.

The results are saved in `pred_results_*.csv` files under specified `--output_dir`.
Expand Down
Loading

0 comments on commit 09a737f

Please sign in to comment.