Histogram of frequencies of top-level fault codes

[wasowski]

@ipa-hsd thanks for calculating how many bugs for each fault code we have (Table 4 in the manuscript).

[hsd-dev]

Faults:
BDO:DEPENDENCY : 43
BDO:BUILD : 31
BAD-SMELL : 17
PROGRAMMING:RESOURCES : 16
ROS-SPECIFIC : 15
PROGRAMMING:BROKEN-CONTRACT : 15
MODELS:ROBOT : 14
PROGRAMMING:LOGIC : 13
CONCURRENCY:NO-SYNC : 12
PROGRAMMING:CALCULATIONS : 11
EVOLUTION:PACKAGE : 11
PROGRAMMING:CONTROL-FLOW : 11
EVOLUTION:LIBRARY : 9
PROGRAMMING:VALIDATION : 7
CONFIG:PARAM : 6
NAMING-TYPOS : 6
COPYPASTA : 6
PROGRAMMING:UNINITIALISED : 6
EVOLUTION:DEPRECATION : 5
PROGRAMMING:DATATYPE : 5
PROGRAMMING:MISSING-FEATURE : 5
SYSTEMS:OS : 5
CONFIG:NAMESPACE : 4
CONFIG:REMAPPING : 4
EVOLUTION:PATHS : 4
CONFIG:CONSTANTS : 4
MODELS:TRANSFORMATIONS : 4
CONFIG:ARGS : 4
CONCURRENCY:BAD-SYNC : 4
CONCURRENCY:SIGNALS : 3
EVOLUTION:LANGUAGE : 3
BDO:ORCHESTRATION : 3
PROGRAMMING:UNUSED : 3
EVOLUTION:FIRMWARE : 2
CONFIG:ENCODING : 2
SYSTEMS:CONFIG : 2
SYSTEM:FIRMWARE : 2
PROGRAMMING:PATH : 1
PROGRAMMING:STRING-FORMATTING : 1
SYSTEMS:HARDWARE : 1
EVOLUTION:DOCS : 1
PROGRAMMING:COMPILER-ERROR : 1
CONCURRENCY:TIMING : 1

[hsd-dev]

Failures:
SOFTWARE:RUNTIME : 56
SOFTWARE:BUILD : 50
SYSTEM:UNINTENDED-BEHAVIOUR : 42
SYSTEM:LIVENESS : 35
SYSTEM:MOTION : 22
NONE : 16
SYSTEM:NONE : 16
SOFTWARE:UX : 15
SOFTWARE:CONCURRENCY : 12
SYSTEM:PHYSICAL : 11
SOFTWARE:NETWORK : 8
SYSTEM:SIMULATION : 7
SOFTWARE:PERFORMANCE : 6
HARMLESS : 5
SYSTEM:PERFORMANCE : 5
UNKNOWN : 3
WARNING : 3
MISLEADING : 2

[hsd-dev]

@wasowski please let me know if the changes in #400 (comment) are reasonable. Also note #400 (comment), still around 13 bugs have been labelled with 'SOFTWARE:SILENT', which is not part of the schema.

[wasowski]

Table 4

Note for self: sum top-level codes as per the google doc codebook. BAD-SMELLS should be moved to General Programming. @ipa-hsd calculates sums, @wasowski moves to LaTeX and revises the text.

@ipa-hsd: BAD-SMELL should be renamed to PROGRAMMING:BAD-SMELL in the bug files. Thanks.

Table 5/6

More complex, @ipa-hsd will do a proposal of mapping and we need to discuss the doubtful cases again.

[hsd-dev]

PROGRAMMING : 111
BDO : 77
EVOLUTION : 35
CONFIG : 24
CONCURRENCY : 20
MODELS : 18
ROS-SPECIFIC : 15
SYSTEMS : 8
NAMING-TYPOS : 6
COPYPASTA : 5
SYSTEM : 2

[hsd-dev]

@wasowski

[hsd-dev]

BDO:BUILD : 31
PROGRAMMING:BAD-SMELL : 17
PROGRAMMING:RESOURCES : 16
PROGRAMMING:BROKEN-CONTRACT : 15
ROS-SPECIFIC : 15
MODELS:ROBOT : 14
PROGRAMMING:LOGIC : 13
CONCURRENCY:NO-SYNC : 12
PROGRAMMING:CONTROL-FLOW : 11
EVOLUTION:PACKAGE : 11
PROGRAMMING:CALCULATIONS : 10
EVOLUTION:LIBRARY : 9
PROGRAMMING:VALIDATION : 7
PROGRAMMING:UNINITIALISED : 6
CONFIG:PARAM : 6
NAMING-TYPOS : 6
PROGRAMMING:DATATYPE : 5
PROGRAMMING:MISSING-FEATURE : 5
COPYPASTA : 5
SYSTEMS:OS : 5
EVOLUTION:DEPRECATION : 5
CONCURRENCY:BAD-SYNC : 4
CONFIG:CONSTANTS : 4
MODELS:TRANSFORMATIONS : 4
CONFIG:NAMESPACE : 4
CONFIG:ARGS : 4
EVOLUTION:PATHS : 4
CONFIG:REMAPPING : 4
CONCURRENCY:SIGNALS : 3
PROGRAMMING:UNUSED : 3
EVOLUTION:LANGUAGE : 3
BDO:ORCHESTRATION : 3
SYSTEMS:CONFIG : 2
CONFIG:ENCODING : 2
EVOLUTION:FIRMWARE : 2
SYSTEM:FIRMWARE : 2
CONCURRENCY:TIMING : 1
EVOLUTION:DOCS : 1
PROGRAMMING:PATH : 1
PROGRAMMING:STRING-FORMATTING : 1
SYSTEMS:HARDWARE : 1
PROGRAMMING:COMPILER-ERROR : 1

updated after changing BAD-SMELL to PROGRAMMING:BAD-SMELL

[hsd-dev]

Faults:
PROGRAMMING : 123
BDO : 77
EVOLUTION : 37
CONFIG : 28
MODELS : 20
CONCURRENCY : 20
SYSTEMS : 8

@wasowski These are the latest values. For table 5 and 6, waiting for #410

[hsd-dev]

Based on the recent issues, this list is definitely not final.
SYSTEM failures:

SYSTEM:NONE : 102
SYSTEM:LOSS-OF-CONTROL : 55
SYSTEM:UNINTENDED-MOVEMENT : 34
SYSTEM:LOSS-OF-FUNCTIONALITY : 14
SYSTEM:MODEL-REALITY-DIVERGENCE : 10
SYSTEM:DEGRADED-PERFORMANCE : 7

SOFTWARE failures:

SOFTWARE:BEHAVIORAL : 69
SOFTWARE:BUILD : 53
SOFTWARE:CRASHING : 41
SOFTWARE:LIVENESS : 21
SOFTWARE:UX : 19
SOFTWARE:NONE : 9
SOFTWARE:PERFORMANCE : 4

[hsd-dev]

After all the labels related PRs were merged, these are the final numbers:
Faults:

PROGRAMMING : 123
BDO : 76
EVOLUTION : 37
CONFIG : 28
MODELS : 20
CONCURRENCY : 20
SYSTEMS : 8

Failures:

SOFTWARE:BEHAVIORAL : 68
SOFTWARE:BUILD : 54
SOFTWARE:CRASHING : 40
SOFTWARE:UX : 20
SOFTWARE:LIVENESS : 15
SOFTWARE:NONE : 9
SOFTWARE:NETWORK : 7
SOFTWARE:PERFORMANCE : 2
SYSTEM:NONE : 106
SYSTEM:BEHAVING-INCORRECTLY : 44
SYSTEM:LOSS-OF-FUNCTIONALITY : 42
SYSTEM:UNRESPONSIVE : 24
SYSTEM:MONITORING : 11
SYSTEM:DEGRADED-PERFORMANCE : 7
SYSTEM:LOSS-OF-CONTROL : 2

These numbers also include confidential bugs. Is that fine? @wasowski @ChrisTimperley @gavanderhoorn @git-afsantos ?

[wasowski]

I think it is fine to include confidential as well (not really reproducible, but with so small a number, also not distorting the picture greatly).