Skip to content

Latest commit

 

History

History
769 lines (449 loc) · 26.5 KB

table-form-results.md

File metadata and controls

769 lines (449 loc) · 26.5 KB
Raw

Accuracy Delta by Model, Fault Type, Dataset

The following shows the average accuracy deltas in full table form.

CIFAR10

CIFAR10, ConvNet, Mislabelling

Fault Amount Baseline LS LC RL KD Ens
0 0 9.59 6.27 7.64 7.18 5.19
10 18.41 11.61 6.24 18.66 10.49 11.09
30 24.49 13.34 8.69 42.45 16.06 14.17
50 20.42 17.42 11.6 46.93 30.06 20.37

CIFAR10, ConvNet, Removal

Fault Amount Baseline LS RL KD Ens
0 0 9.59 7.64 7.18 5.19
10 10.49 9.84 7.87 7.76 8.08
30 37.13 9.72 10.32 9.78 18.93
50 13.84 10.84 18.14 11.36 13.69

CIFAR10, ConvNet, Repetition

Fault Amount Baseline LS RL KD Ens
0 0 9.59 7.64 7.18 5.19
10 29.59 9.14 6.04 7.65 7.14
30 29.52 9.61 5.64 7.65 11.14
50 6.87 8.83 5.73 7.62 5.98

CIFAR10, DeconvNet, Mislabelling

Fault Amount Baseline LS LC RL KD Ens
0 0 6.29 9.62 8.65 9.14 7.35
10 40.17 7.78 7.39 8.59 13.66 9.71
30 31.5 10.31 11.34 15.27 30.3 11
50 62.81 15.66 14.69 34.96 41.51 17.06

CIFAR10, DeconvNet, Removal

Fault Amount Baseline LS RL KD Ens
0 0 6.29 8.65 9.14 7.35
10 12.01 6.63 9.18 9.16 7.55
30 58.63 8.33 10.1 12.35 13.73
50 21.21 8.72 11.09 11.94 13.06

CIFAR10, DeconvNet, Repetition

Fault Amount Baseline LS RL KD Ens
0 0 6.29 8.65 9.14 7.35
10 8.15 6.29 7.12 9.8 6.38
30 28.05 6.8 5.88 11.41 9.3
50 17.15 6.5 5.85 8.75 5.95

CIFAR10, MobileNet, Mislabelling

Fault Amount Baseline LS RL KD Ens
0 0 11.5 30.2 25.3 1.1
10 18.8 14.6 38.2 28 5
30 21.1 25.2 48.5 45.4 6.7
50 54.7 38.9 41.7 59.9 4.1

CIFAR10, MobileNet, Removal

Fault Amount Baseline LS RL KD Ens
0 0 11.5 30.2 25.3 1.1
10 36.1 14.1 36.2 24 3.4
30 0 14.8 30.2 24.5 3.6
50 0 15.8 43.3 27.2 2.2

CIFAR10, MobileNet, Repetition

Fault Amount Baseline LS RL KD Ens
0 0 11.5 30.2 25.3 1.1
10 15.6 13.7 36.4 19.9 4.4
30 11.1 13.2 21.3 19.6 3.2
50 10 10.7 26.7 24.3 4

CIFAR10, ResNet18, Mislabelling

Fault Amount Baseline LS LC RL KD Ens
0 0 5.58 3.18 3.9 15.81 6.71
10 17.06 10.28 4.62 4.77 32.26 16.5
30 43.01 17.2 5.95 7.35 47.4 19.45
50 23.76 26.12 7.47 13.79 64.43 25.32

CIFAR10, ResNet18, Removal

Fault Amount Baseline LS RL KD Ens
0 0 5.58 3.9 15.81 6.71
10 20.36 5.58 4.19 23.37 11.85
30 21.88 6.58 5.09 22.03 23.66
50 17.26 8.51 7.02 28.91 17.57

CIFAR10, ResNet18, Repetition

Fault Amount Baseline LS RL KD Ens
0 0 5.58 3.9 15.81 6.71
10 13.36 5.99 3.34 18.56 10.33
30 13.1 5.47 3.25 14.09 15
50 34.11 5.77 3.16 18.91 9.67

CIFAR10, ResNet50, Mislabelling

Fault Amount Baseline LS LC RL KD Ens
0 0 2.84 3.69 5.94 18.74 3.94
10 30.75 9.05 4.79 7.5 29.96 13.08
30 16.66 31.01 7.81 20.77 49.13 16.64
50 55.92 53.02 16.4 39.78 61.49 22.84

CIFAR10, ResNet50, Removal

Fault Amount Baseline LS RL KD Ens
0 0 2.84 5.94 18.74 3.94
10 24.89 3.03 5.51 15.85 10.25
30 31.15 3.72 7.68 19.29 19.58
50 18.91 4.87 11.42 20.64 14.69

CIFAR10, ResNet50, Repetition

Fault Amount Baseline LS RL KD Ens
0 0 2.84 5.94 18.74 3.94
10 11.41 2.62 5.92 16.38 8.89
30 17.72 2.43 4.7 17.17 12.76
50 12.71 2.38 4.61 17.77 8.14

CIFAR10, VGG11, Mislabelling

Fault Amount Baseline LS LC RL KD Ens
0 0 3.25 3.03 6.67 8.2 6.5
10 21.89 8.09 6.02 8 14.81 16.37
30 18.33 22.67 7.31 12.73 29.56 19.1
50 23.39 41.89 11.04 23.97 49.55 25.83

CIFAR10, VGG11, Removal

Fault Amount Baseline LS RL KD Ens
0 0 3.25 6.67 8.2 6.5
10 11.04 4.52 7.55 10.09 12.01
30 16.37 3.99 9.87 11.19 23.37
50 25.76 5.53 10.8 14.34 17.8

CIFAR10, VGG11, Repetition

Fault Amount Baseline LS RL KD Ens
0 0 3.25 6.67 8.2 6.5
10 15.26 3.37 6.72 9.04 9.92
30 25.4 3 5.91 8.84 15.37
50 11.96 3.36 5.5 8.52 9.17

CIFAR10, VGG16, Mislabelling

Fault Amount Baseline LS LC RL KD Ens
0 0 2.82 2.86 8.31 4.42 8.17
10 27.69 3.88 4.82 15.7 8.09 14.06
30 25.69 6.79 5.02 19.67 17.47 17.28
50 56.63 13.24 8.8 46.26 37.31 23.08

CIFAR10, VGG16, Removal

Fault Amount Baseline LS RL KD Ens
0 0 2.82 8.31 4.42 8.17
10 29.07 2.86 8.93 4.69 10.46
30 23.97 2.81 15.18 5.75 21.98
50 27.78 4.65 15.13 7.8 16.75

CIFAR10, VGG16, Repetition

Fault Amount Baseline LS RL KD Ens
0 0 2.82 8.31 4.42 8.17
10 25.63 2.71 8.85 4.38 8.89
30 26.78 2.94 6.36 4.29 13.86
50 7.8 2.75 5.22 4.25 8.05



GTSRB

GTSRB, ConvNet, Mislabelling

Fault Amount Baseline LS LC RL KD Ens
0 0 3.11 16.94 29.09 1.14 0.52
10 5.47 3.48 14.95 39.29 2.3 2.49
30 3.65 4.63 23.1 42.09 3.83 4.25
50 5.08 8.09 21.76 58.04 4.64 7.85

GTSRB, ConvNet, Removal

Fault Amount Baseline LS RL KD Ens
0 0 3.11 29.09 1.14 0.52
10 2.51 3.71 32.76 1.16 1.06
30 3.14 4.5 51.03 1.68 0.97
50 4.21 4.3 42.07 2.23 1.5

GTSRB, ConvNet, Repetition

Fault Amount Baseline LS RL KD Ens
0 0 3.11 29.09 1.14 0.52
10 2.15 2.88 30.74 1.22 0.87
30 1.89 3.21 27.41 1.42 0.85
50 2.08 3.81 30.88 1.49 0.98

GTSRB, DeconvNet, Mislabelling

Fault Amount Baseline LS LC RL KD Ens
0 0 3.7 40.81 20.49 1.92 1.41
10 6.87 4.36 40.53 22.16 6.52 4.25
30 10.01 7.22 43 27.51 13.26 5.97
50 6.31 13.61 38.73 28.21 24.72 9.42

GTSRB, DeconvNet, Removal

Fault Amount Baseline LS RL KD Ens
0 0 3.7 20.49 1.92 1.41
10 2.37 4.05 23.71 2.61 1.91
30 2.64 5.36 20.46 3.05 1.92
50 3.28 4.84 24.61 3.3 2.71

GTSRB, DeconvNet, Repetition

Fault Amount Baseline LS RL KD Ens
0 0 3.7 20.49 1.92 1.41
10 1.78 4.32 20.65 1.87 1.49
30 1.95 3.22 17.92 1.92 1.42
50 2.05 3.49 18.13 1.92 1.45

GTSRB, MobileNet, Mislabelling

Fault Amount Baseline LS RL KD Ens
0 0 7.22 11.05 4.44 1.16
10 14.24 6.91 12.77 9.63 3.27
30 14.32 14.24 17.58 16.04 4.73
50 31.19 23.04 30.98 42.98 8.12

GTSRB, MobileNet, Removal

Fault Amount Baseline LS RL KD Ens
0 0 7.22 11.05 4.44 1.16
10 7.56 9.19 7.17 4.25 1.81
30 9.17 7.1 14.7 5.85 1.47
50 11.5 8.68 16.48 7.1 2.12

GTSRB, MobileNet, Repetition

Fault Amount Baseline LS RL KD Ens
0 0 7.22 11.05 4.44 1.16
10 5.59 5.99 6.88 4.46 1.46
30 7.42 4.29 5.75 3.88 1.27
50 4.84 31.38 5.58 3.91 1.49

GTSRB, ResNet18, Mislabelling

Fault Amount Baseline LS LC RL KD Ens
0 0 4.25 5.77 4.18 1.82 1.9
10 21.37 4.61 10.14 3.99 23.75 4.04
30 11.2 5.27 15.05 9.06 41.51 5.68
50 8.25 9.58 15.3 14.7 58.18 9.01

GTSRB, ResNet18, Removal

Fault Amount Baseline LS RL KD Ens
0 0 4.25 4.18 1.82 1.9
10 3.62 4.62 9.24 1.66 2.14
30 3.58 5.11 12.3 2.19 2.08
50 5.46 6.57 13.39 4.23 2.68

GTSRB, ResNet18, Repetition

Fault Amount Baseline LS RL KD Ens
0 0 4.25 4.18 1.82 1.9
10 4.05 4.12 2.99 1.46 1.78
30 3.08 3.59 2.07 1.49 1.52
50 3.69 3.54 1.6 1.8 1.69

GTSRB, ResNet50, Mislabelling

Fault Amount Baseline LS LC RL KD Ens
0 0 1.92 10.91 3.65 1.27 1.9
10 26.51 2.33 10.92 5.34 21.48 4.17
30 11.46 3.42 17.62 11.73 41.57 5.77
50 25.1 4.4 26.17 22.44 53.64 9.44

GTSRB, ResNet50, Removal

Fault Amount Baseline LS RL KD Ens
0 0 1.92 3.65 1.27 1.9
10 4.02 1.44 6.39 1.07 1.94
30 6.15 2.12 7.77 1.38 1.94
50 6.77 1.93 8.64 2.2 2.86

GTSRB, ResNet50, Repetition

Fault Amount Baseline LS RL KD Ens
0 0 1.92 3.65 1.27 1.9
10 3.91 1.67 3.67 0.91 1.73
30 3.08 1.52 2.14 1.04 1.49
50 3.4 1.42 0.89 2.58 1.51

GTSRB, VGG11, Mislabelling

Fault Amount Baseline LS LC RL KD Ens
0 0 2.11 7.67 5.04 1.48 0.39
10 7.16 2.54 27.55 6 5.16 2.96
30 19.03 4.31 28.45 14.46 19.42 4.44
50 13.82 3.95 25.99 25.21 35 7.72

GTSRB, VGG11, Removal

Fault Amount Baseline LS RL KD Ens
0 0 2.11 5.04 1.48 0.39
10 3.33 2.36 4.86 1.82 1.23
30 4.32 3.07 6.81 2.26 1.27
50 4.37 3.03 8.55 1.88 1.57

GTSRB, VGG11, Repetition

Fault Amount Baseline LS RL KD Ens
0 0 2.11 5.04 1.48 0.39
10 3.38 2.15 4.5 1.53 0.85
30 3.28 1.87 2.73 2.54 0.86
50 3.51 1.77 2.21 1.39 0.83

GTSRB, VGG16, Mislabelling

Fault Amount Baseline LS LC RL KD Ens
0 0 2.68 10.39 3.18 2.3 1.29
10 6.12 6.91 9.68 5.55 2.57 3.56
30 6.1 5.86 23.45 12.37 5.23 5.05
50 45.55 29.49 18.8 41.5 8.28 8.31

GTSRB, VGG16, Removal

Fault Amount Baseline LS RL KD Ens
0 0 2.68 3.18 2.3 1.29
10 1.77 3.09 5.13 1.44 1.48
30 4.08 4.45 5.14 2.04 2.06
50 2.4 6 8.69 3.12 2.27

GTSRB, VGG16, Repetition

Fault Amount Baseline LS RL KD Ens
0 0 2.68 3.18 2.3 1.29
10 2.44 2.19 3.3 2.03 1.52
30 1.83 1.96 2.26 1.55 1.51
50 1.98 2.09 2.06 1.27 1.61



Pneumonia

Pneumonia, ConvNet, Mislabelling

Fault Amount Baseline LS LC RL KD Ens
0 0 6.43 20.7 28.35 3.13 0.52
10 4.87 8.52 21.22 25.74 10.09 14.78
30 7.3 24.35 21.04 29.91 23.83 26.96
50 60.57 47.13 13.22 60.22 63.87 60.57

Pneumonia, ConvNet, Removal

Fault Amount Baseline LS RL KD Ens
0 0 6.43 28.35 3.13 0.52
10 2.61 5.04 28.7 7.48 1.22
30 1.91 6.61 28.87 4.87 0.7
50 3.48 8.7 30.61 4.7 1.22

Pneumonia, ConvNet, Repetition

Fault Amount Baseline LS RL KD Ens
0 0 6.43 28.35 3.13 0.52
10 2.43 4.17 27.83 4 1.22
30 1.22 4.35 20.87 4 0.87
50 1.74 5.39 24.52 3.48 1.39

Pneumonia, DeconvNet, Mislabelling

Fault Amount Baseline LS LC RL KD Ens
0 0 5.43 14.89 10.33 8.23 2.28
10 20.67 6.83 22.07 9.81 17.16 14.54
30 26.62 13.13 18.21 11.21 26.09 26.44
50 59.97 60.85 21.89 56.57 35.03 59.97

Pneumonia, DeconvNet, Removal

Fault Amount Baseline LS RL KD Ens
0 0 5.43 10.33 8.23 2.28
10 4.2 5.6 10.68 9.28 2.28
30 4.73 8.06 10.51 6.3 2.28
50 3.33 10.51 10.68 10.86 2.28

Pneumonia, DeconvNet, Repetition

Fault Amount Baseline LS RL KD Ens
0 0 5.43 10.33 8.23 2.28
10 3.85 5.08 10.16 7.53 3.15
30 4.2 5.25 9.98 7.18 2.63
50 2.63 6.48 13.31 7.36 2.98

Pneumonia, MobileNet, Mislabelling

Fault Amount Baseline LS RL KD Ens
0 0 4.4 19.37 5.46 1.41
10 39.79 6.69 19.89 12.68 14.79
30 48.24 12.32 22.54 31.87 27.11
50 61.2 57.15 57.85 61.2 61.2

Pneumonia, MobileNet, Removal

Fault Amount Baseline LS RL KD Ens
0 0 4.4 19.37 5.46 1.41
10 3.7 3.52 20.07 7.39 1.58
30 3.17 3.17 21.65 8.8 1.41
50 4.93 4.58 21.48 6.34 2.11

Pneumonia, MobileNet, Repetition

Fault Amount Baseline LS RL KD Ens
0 0 4.4 19.37 5.46 1.41
10 2.29 3.52 17.61 7.92 2.11
30 2.11 4.75 19.37 11.27 1.41
50 3.52 2.82 18.49 5.63 2.11

Pneumonia, ResNet18, Mislabelling

Fault Amount Baseline LS LC RL KD Ens
0 0 3.5 24.52 23.82 5.78 1.58
10 11.56 4.38 15.06 22.77 12.78 12.61
30 12.26 8.41 27.85 25.04 29.07 24.87
50 57.52 60.67 23.64 56.82 52.01 58.75

Pneumonia, ResNet18, Removal

Fault Amount Baseline LS RL KD Ens
0 0 3.5 23.82 5.78 1.58
10 1.75 7.01 24.34 6.48 1.58
30 1.93 4.38 21.72 8.41 1.4
50 2.45 3.33 22.59 9.11 2.1

Pneumonia, ResNet18, Repetition

Fault Amount Baseline LS RL KD Ens
0 0 3.5 23.82 5.78 1.58
10 3.33 6.83 23.47 4.73 2.28
30 3.85 7.53 18.04 8.23 1.93
50 1.93 11.91 23.12 8.93 2.8

Pneumonia, ResNet50, Mislabelling

Fault Amount Baseline LS LC RL KD Ens
0 0 3.2 15.12 19.57 4.09 0.71
10 37.72 5.34 62.44 14.95 13.35 15.48
30 62.26 9.61 29.36 24.73 24.91 27.76
50 62.44 57.99 15.48 59.23 46.09 62.44

Pneumonia, ResNet50, Removal

Fault Amount Baseline LS RL KD Ens
0 0 3.2 19.57 4.09 0.71
10 3.74 4.09 18.86 5.69 1.25
30 1.78 3.56 17.44 6.58 0.71
50 2.49 4.8 19.4 8.01 1.25

Pneumonia, ResNet50, Repetition

Fault Amount Baseline LS RL KD Ens
0 0 3.2 19.57 4.09 0.71
10 2.49 4.09 18.51 3.91 1.25
30 2.31 7.3 18.68 5.16 0.89
50 3.38 3.02 19.75 4.98 1.42

Pneumonia, VGG11, Mislabelling

Fault Amount Baseline LS LC RL KD Ens
0 0 3.98 25.26 20.76 8.3 1.73
10 12.98 12.8 31.31 20.93 9.86 15.4
30 60.74 12.98 27.34 20.24 29.24 27.51
50 60.74 49.31 60.74 60.74 51.38 60.74

Pneumonia, VGG11, Removal

Fault Amount Baseline LS RL KD Ens
0 0 3.98 20.76 8.3 1.73
10 1.21 5.71 19.72 3.46 1.56
30 2.94 4.67 20.93 6.06 1.04
50 6.23 6.06 19.55 4.33 1.38

Pneumonia, VGG11, Repetition

Fault Amount Baseline LS RL KD Ens
0 0 3.98 20.76 8.3 1.73
10 3.98 4.5 21.63 7.79 1.73
30 3.63 7.09 21.45 7.96 1.38
50 0.69 3.98 19.72 5.36 1.73

Pneumonia, VGG16, Mislabelling

Fault Amount Baseline LS LC RL KD Ens
0 0 6.06 32.26 16.58 7.84 1.6
10 15.86 30.3 27.63 15.15 7.49 16.04
30 62.56 16.58 19.43 20.32 28.16 28.34
50 63.09 38.32 24.42 63.09 39.57 63.09

Pneumonia, VGG16, Removal

Fault Amount Baseline LS RL KD Ens
0 0 6.06 16.58 7.84 1.6
10 13.73 4.46 18.36 9.45 1.25
30 7.49 17.11 18 6.6 0.71
50 31.91 20.32 16.58 18.72 1.07

Pneumonia, VGG16, Repetition

Fault Amount Baseline LS RL KD Ens
0 0 6.06 16.58 7.84 1.6
10 3.21 3.74 17.29 6.77 1.25
30 2.32 7.49 15.51 8.73 0.89
50 4.1 1.25 17.29 6.6 1.78