Experiments

Schwa is created in a context of research therefore, we must validate our hypothesis with data. There are two main Research Questions:

• RQ1: What are the most important features and their importance (weights)?
• RQ2: Can we decrease the diagnostic cost of Crowbar by using Schwa defect probabilities?

Learning Features (RQ1)

Considering that Schwa outputs a rank, our goal is to compute a probability from Revisions, Fixes and Authors tracked information, so that faulty components have more defect probability that non-faulty components. We call this the distance. By using, Genetic Algorithms, we encoded an individual as {RevisionsWeight, FixesWeight, AuthorsWeight} and the fitness function is distance({RevisionsWeight, FixesWeight, AuthorsWeight}}.

Results

The following tables are the results of running the Genetic Algorithm in a variety of projects, from Open Source to enterprise. The bits precision is 3 and the number of generations is 40. Be aware that, solutions with fitness=0 are not trustable, because they can have more than one solution (individuals) that are 0.

andrefreitas/schwa

Commits	Revisions	Fixes	Authors	Fitnesss
5	0.2857	0.2857	0.4286	0
50	0.7143	0.1429	0.1429	1.0699
100	0.7143	0.1429	0.1429	1.1875

crowbar/libcrowbar

Commits	Revisions	Fixes	Authors	Fitnesss
5	0.1429	0.1429	0.7143	0.3486
50	0.7143	0.1429	0.1429	1.3639
100	0.7143	0.1429	0.1429	0.3387

github/joda-time

Commits	Revisions	Fixes	Authors	Fitnesss
5	0.1429	0.2857	0.5714	0
50	0.1429	0.7143	0.1429	-2.1874
100	0.1429	0.7143	0.1429	0.3893

andrefreitas/feup-cmov-meoarena

Commits	Revisions	Fixes	Authors	Fitnesss
5	0.2857	0.4286	0.2857	0
50	0.7143	0.1429	0.1429	1.6712
100	0.7143	0.1429	0.1429	0.7368

github/mongo-java-driver

Commits	Revisions	Fixes	Authors	Fitnesss
5	0.7143	0.1429	0.1429	0.3486
50	0.7143	0.1429	0.1429	0.1685
100	0.1429	0.7143	0.1429	1.4666

strongstep/scraim

Commits	Revisions	Fixes	Authors	Fitnesss
5	0.4286	0.1429	0.4286	0.3486
50	0.1429	0.1429	0.7143	0.3146
100	0.1429	0.7143	0.1429	0.9399

diogopinela/trainsim

Commits	Revisions	Fixes	Authors	Fitnesss
5	0.1429	0.7143	0.1429	0
50	0.7143	0.1429	0.1429	1.5945
100	0.5714	0.2857	0.1429	2.0425

shiftforward/adstax

Commits	Revisions	Fixes	Authors	Fitnesss
5	0.4286	0.1429	0.4286	0
50	0.4286	0.4286	0.1429	1.9044
100	0.2857	0.4286	0.2857	3.3134

shiftforward/boxer

Commits	Revisions	Fixes	Authors	Fitnesss
5	0.2857	0.1429	0.5714	0.4408
50	0.1429	0.1429	0.7143	0.4457
100	0.1429	0.7143	0.1429	-1.4159

shiftforward/apso

Commits	Revisions	Fixes	Authors	Fitnesss
5	0.7143	0.1429	0.1429	0.9439
50	0.1429	0.7143	0.1429	1.9547
100	0.1429	0.4286	0.4286	1.3927

github/hivedb

Commits	Revisions	Fixes	Authors	Fitnesss
5	0.7143	0.1429	0.1429	0
50	0.1429	0.7143	0.1429	0.1739
100	0.7143	0.1429	0.1429	0.9095

github/teamengine

Commits	Revisions	Fixes	Authors	Fitnesss
5	0.7143	0.1429	0.1429	0
50	0.1429	0.7143	0.1429	1.1356
100	0.7143	0.1429	0.1429	3.1080

github/automatalib

Commits	Revisions	Fixes	Authors	Fitnesss
5	0.1429	0.1429	0.7143	0.3486
50	0.1429	0.7143	0.1429	-0.1952
100	0.4286	0.4286	0.1429	0.5261

github/cdi-tck

Commits	Revisions	Fixes	Authors	Fitnesss
5	0.1429	0.5714	0.2857	0
50	0.2857	0.2857	0.4286	0
100	0.1429	0.7143	0.1429	-0.3247

Crowbar Diagnostic Cost (RQ2)

The diagnostic cost can be described as the average distance a faulty component is from the top of the rank. It is the average because components with the same score can change their positions on the rank, if you run Crowbar multiple times.

github/joda-time

Changes

Commit	Message	Description
8207a55	Introduced a bug	Added a defect in DateTime.java in withZoneRetainfields()
74149c0	Inserted a bug	Added a defect in Duration.java in minus()
22a5f71	Fixed withZoneRetainFields bug	Fixed withZoneRetainfields() bug

8207a55

     public DateTime withZoneRetainFields(DateTimeZone newZone) {
         newZone = DateTimeUtils.getZone(newZone);
         DateTimeZone originalZone = DateTimeUtils.getZone(getZone());
-        if (newZone == originalZone) {
+        if (newZone != originalZone) {
             return this;
         }

74149c0

       public Duration minus(long amount) {
-        return withDurationAdded(amount, -1);
+        return withDurationAdded(amount, -2);
     }

   public Duration minus(ReadableDuration amount) {
-        if (amount == null) {
+        if (amount != null) {
             return this;
         }
         return withDurationAdded(amount.getMillis(), -1);
     }

Schwa configuration

Configuration #1

commits: 20 # maximum commits
features_weights: # sum must be 1
  revisions: 0.25
  fixes: 0.5
  authors: 0.25

Ranks

Rank #1

• Schwa: Priors and Goodnesses
• Commit: 74149c0
• Diagnostic Cost: 1
• Configuration: #1

Pos	Component	Probability
1	root:org:joda:time:DateTime:withZoneRetainFields	1
2	root:org:joda:time:Duration:minus	0.639099
3	root:org:joda:time:Duration:withDurationAdded	0.360901
4	root:org:joda:time:base:AbstractInstant:getZone	5.421E-20
5	root:org:joda:time:DateTimeUtils:getZone	4.702E-38
6	root:org:joda:time:base:BaseDuration:getMillis	1.409E-39

Rank #2

• Schwa: Priors
• Commit: 74149c0
• Diagnostic Cost: 1
• Configuration: #1

Pos	Component	Probability
1	root:org:joda:time:DateTime:withZoneRetainFields	0.999937
2	root:org:joda:time:Duration:minus	0.630768
3	root:org:joda:time:Duration:withDurationAdded	0.369232
4	root:org:joda:time:base:AbstractInstant:getZone	6.282E-5
5	root:org:joda:time:DateTimeUtils:getZone	1.471E-11
6	root:org:joda:time:base:BaseDuration:getMillis	7.443E-14
7	"" :chrono:ZonedChronology:ZonedDateTimeField:set	7.443E-14
8	root:org:joda:time:base:AbstractDateTime:toString	8.981E-302
9	root:org:joda:time:base:AbstractInstant:toString	1.720E-309

Rank #3

• Schwa: Goodnesses
• Commit: 74149c0
• Diagnostic Cost: 1
• Configuration: #1

Pos	Component	Probability
1	root:org:joda:time:DateTime:withZoneRetainFields	1
2	root:org:joda:time:Duration:minus	0.998567
3	root:org:joda:time:Duration:withDurationAdded	0.00143313
4	root:org:joda:time:base:AbstractInstant:getZone	5.421E-20
5	root:org:joda:time:DateTimeUtils:getZone	4.702E-38
6	root:org:joda:time:base:BaseDuration:getMillis	2.20089E-39

Rank #4

• Schwa: Priors and Goodnesses
• Commit: 22a5f71
• Diagnostic Cost: 0
• Configuration: #1

Pos	Component	Probability
1	root:org:joda:time:Duration:minus	0.637027
2	root:org:joda:time:Duration:withDurationAdded	0.362973
3	root:org:joda:time:base:BaseDuration:getMillis	1.40404E-39

Rank #5

• Schwa: Goodnesses
• Commit: 22a5f71
• Diagnostic Cost: 0
• Configuration: #1

Pos	Component	Probability
1	root:org:joda:time:Duration:minus	0.998552
2	root:org:joda:time:Duration:withDurationAdded	0.00144786
3	root:org:joda:time:base:BaseDuration:getMillis	2.20086E-39

Rank #6

• Schwa: Priors
• Commit: 22a5f71
• Diagnostic Cost: 1
• Configuration: #1

Pos	Component	Probability
1	root:org:joda:time:Duration:withDurationAdded	0.62267
2	root:org:joda:time:Duration:minus	0.377321
3	root:org:joda:time:base:BaseDuration:getMillis	5.37537E-13

Rank #7

• Schwa: None
• Commit: 22a5f71
• Diagnostic Cost: 0
• Configuration: #1

Pos	Component	Probability
1	root:org:joda:time:Duration:minus	0.995796
2	root:org:joda:time:Duration:withDurationAdded	0.00418174
3	root:org:joda:time:base:BaseDuration:getMillis	1.41862E-12

github/cdi-tck

Changes

Commit	Message	Description
1bc2e2d	Added bug	Added a defect in ActionSequence.java toString() method

1bc2e2d

         @Override
     public String toString() {
-        return String.format("ActionSequence [name=%s, data=%s]", name, getData());
+        return String.format("ActionSequenc [name=%s, data=%s]", name, getData());
     }
 

Schwa configuration

Configuration #1

commits: 20 # maximum commits
features_weights: # sum must be 1
  revisions: 0.15
  fixes: 0.70
  authors: 0.15

Configuration #2

commits: 20 # maximum commits
features_weights: # sum must be 1
  revisions: 0.7
  fixes: 0.15
  authors: 0.15

Configuration #3

commits: 20 # maximum commits
features_weights: # sum must be 1
  revisions: 1
  fixes: 0
  authors: 0

Ranks

Rank #1

• Schwa: Goodnesses
• Commit: 1bc2e2d
• Diagnostic Cost: 0
• Configuration: #1

Pos	Component	Probability
1	root:org:jboss:cdi:tck:util:ActionSequence:toString	0.6721647100000001
2	root:org:jboss:cdi:tck:impl:ConfigurationImpl:setTestTimeoutFactor	0.4671534600000001
3	root:org:jboss:cdi:tck:util:Timer:start	0.11678837799999997
4	root:org:jboss:cdi:tck:util:Timer:checkConfiguration	0.11678837799999997
5	root:org:jboss:cdi:tck:util:ActionSequence:getSequence	0.08402056899999999

Rank #2

• Schwa: Priors and Goodnesses
• Commit: 1bc2e2d
• Diagnostic Cost: 6
• Configuration: #1

Pos	Component	Probability
1	root:org:jboss:cdi:tck:util:ActionSequence:getData	0.9203061599999998
2	root:org:jboss:cdi:tck:impl:ConfigurationImpl:setTestTimeoutFactor	0.4671532299999999
3	root:org:jboss:cdi:tck:util:Timer:start	0.11678830579999999
4	root:org:jboss:cdi:tck:util:Timer:checkConfiguration	0.11678830579999999
5	root:org:jboss:cdi:tck:impl:ConfigurationImpl:getTestTimeoutFactor	0.058394157799999985
6	root:org:jboss:cdi:tck:util:Timer:setDelay	0.058394157799999985
7	root:org:jboss:cdi:tck:util:ActionSequence:toString	0.05513951799999999
8	root:org:jboss:cdi:tck:util:Timer:reset	0.029197125439999996

Rank #3

• Schwa: Priors
• Commit: 1bc2e2d
• Diagnostic Cost: 8
• Configuration: #1

Pos	Component	Probability
1	root:org:jboss:cdi:tck:util:ActionSequence:getData	0.9215867999999998
2	root:org:jboss:cdi:tck:impl:ConfigurationImpl:setTestTimeoutFactor	0.149885205
3	root:org:jboss:cdi:tck:util:Timer:start	0.13180296399999997
4	root:org:jboss:cdi:tck:util:Timer:checkConfiguration	0.13180296399999997
5	root:org:jboss:cdi:tck:impl:ConfigurationImpl:getTestTimeoutFactor	0.10164503999999999
6	root:org:jboss:cdi:tck:util:Timer:setDelay	0.10164503999999999
7	root:org:jboss:cdi:tck:util:Timer:clearStopConditions	0.07327110100000003
8	root:org:jboss:cdi:tck:util:ActionSequence:toString	0.03731706

Rank #4

• Schwa: None
• Commit: 1bc2e2d
• Diagnostic Cost: 0
• Configuration: #1

Pos	Component	Probability
1	root:org:jboss:cdi:tck:util:ActionSequence:toString	0.46210460000000003
2	root:org:jboss:cdi:tck:impl:ConfigurationImpl:setTestTimeoutFactor	0.32066443000000006
3	root:org:jboss:cdi:tck:util:ActionSequence:getSequence	0.13764563

Rank #5

• Schwa: Priors and Goodnesses
• Commit: 1bc2e2d
• Diagnostic Cost: 6
• Configuration: #2

Pos	Component	Probability
1	root:org:jboss:cdi:tck:util:ActionSequence:getData	0.9203061599999998
2	root:org:jboss:cdi:tck:impl:ConfigurationImpl:setTestTimeoutFactor	0.4671532299999999
3	root:org:jboss:cdi:tck:util:Timer:start	0.11678830579999999
4	root:org:jboss:cdi:tck:util:Timer:checkConfiguration	0.11678830579999999
5	root:org:jboss:cdi:tck:impl:ConfigurationImpl:getTestTimeoutFactor	0.058394157799999985
6	root:org:jboss:cdi:tck:util:Timer:setDelay	0.058394157799999985
7	root:org:jboss:cdi:tck:util:ActionSequence:toString	0.05513951799999999
8	root:org:jboss:cdi:tck:util:Timer:reset	0.029197125439999996
9	root:org:jboss:cdi:tck:util:Timer:clearStopConditions	0.029197125439999996
10	root:org:jboss:cdi:tck:impl:ConfigurationFactory:get	0.007299266660000002
11	root:org:jboss:cdi:tck:util:ActionSequence:getSequence	0.0068924418
12	root:org:jboss:cdi:tck:util:ActionSequence:reset	0.0034462169000000005
13	root:org:jboss:cdi:tck:util:ActionSequence:addAction	0.0034462169000000005
14	root:org:jboss:cdi:tck:util:ActionSequence:add	2.1538901999999997E-4
15	root:org:jboss:cdi:tck:util:ActionSequence:checkStringValue	2.1538901999999997E-4

Rank #6

• Schwa: Priors
• Commit: 1bc2e2d
• Diagnostic Cost: 7
• Configuration: #2

Pos	Component	Probability
1	root:org:jboss:cdi:tck:util:ActionSequence:getData	0.9479065
2	root:org:jboss:cdi:tck:impl:ConfigurationImpl:getTestTimeoutFactor	0.14449913500000003
3	root:org:jboss:cdi:tck:util:Timer:setDelay	0.14449913500000003
4	root:org:jboss:cdi:tck:util:Timer:reset	0.14449913500000003
5	root:org:jboss:cdi:tck:util:Timer:clearStopConditions	0.14449913500000003
6	root:org:jboss:cdi:tck:impl:ConfigurationFactory:get	0.06692622699999999
7	root:org:jboss:cdi:tck:impl:ConfigurationImpl:setTestTimeoutFactor	0.04577338799999999
8	root:org:jboss:cdi:tck:util:ActionSequence:toString	0.016159519
9	root:org:jboss:cdi:tck:util:Timer:start	0.010152520999999998
10	root:org:jboss:cdi:tck:util:Timer:checkConfiguration	0.010152520999999998
11	root:org:jboss:cdi:tck:util:ActionSequence:getSequence	0.00905224
12	root:org:jboss:cdi:tck:util:ActionSequence:reset	0.003970634
13	root:org:jboss:cdi:tck:util:ActionSequence:addAction	0.003970634
14	root:org:jboss:cdi:tck:util:ActionSequence:add	0.002958533
15	root:org:jboss:cdi:tck:util:ActionSequence:checkStringValue	0.002958533

Rank #7

• Schwa: Priors
• Commit: 1bc2e2d
• Diagnostic Cost: 7
• Configuration: #3

Pos	Component	Probability
1	root:org:jboss:cdi:tck:util:ActionSequence:getData	0.9479065
2	root:org:jboss:cdi:tck:impl:ConfigurationImpl:getTestTimeoutFactor	0.14449913500000003
3	root:org:jboss:cdi:tck:util:Timer:setDelay	0.14449913500000003
4	root:org:jboss:cdi:tck:util:Timer:reset	0.14449913500000003
5	root:org:jboss:cdi:tck:util:Timer:clearStopConditions	0.14449913500000003
6	root:org:jboss:cdi:tck:impl:ConfigurationFactory:get	0.06692622699999999
7	root:org:jboss:cdi:tck:impl:ConfigurationImpl:setTestTimeoutFactor	0.04577338799999999
8	root:org:jboss:cdi:tck:util:ActionSequence:toString	0.016159519
9	root:org:jboss:cdi:tck:util:Timer:start	0.010152520999999998
10	root:org:jboss:cdi:tck:util:Timer:checkConfiguration	0.010152520999999998
11	root:org:jboss:cdi:tck:util:ActionSequence:getSequence	0.00905224
12	root:org:jboss:cdi:tck:util:ActionSequence:reset	0.003970634
13	root:org:jboss:cdi:tck:util:ActionSequence:addAction	0.003970634
14	root:org:jboss:cdi:tck:util:ActionSequence:add	0.002958533
15	root:org:jboss:cdi:tck:util:ActionSequence:checkStringValue	0.002958533

Rank #8

• Schwa: Goodness
• Commit: 1bc2e2d
• Diagnostic Cost: 0
• Configuration: #2

Pos	Component	Probability
1	root:org:jboss:cdi:tck:util:ActionSequence:toString	0.6721633400000001
2	root:org:jboss:cdi:tck:impl:ConfigurationImpl:setTestTimeoutFactor	0.3615815700000001
3	root:org:jboss:cdi:tck:util:Timer:start	0.180790814
4	root:org:jboss:cdi:tck:util:Timer:checkConfiguration	0.180790814
5	root:org:jboss:cdi:tck:util:ActionSequence:getSequence	0.08402063200000001
6	root:org:jboss:cdi:tck:impl:ConfigurationImpl:getTestTimeoutFactor	0.04519773600000001
7	root:org:jboss:cdi:tck:util:Timer:setDelay	0.04519773600000001
8	root:org:jboss:cdi:tck:util:Timer:reset	0.04519773600000001
9	root:org:jboss:cdi:tck:util:Timer:clearStopConditions	0.04519773600000001
10	root:org:jboss:cdi:tck:util:ActionSequence:reset	0.042010326
11	root:org:jboss:cdi:tck:util:ActionSequence:addAction	0.042010326
12	root:org:jboss:cdi:tck:util:ActionSequence:getData	0.028512467
13	root:org:jboss:cdi:tck:impl:ConfigurationFactory:get	0.0056497171999999995
14	root:org:jboss:cdi:tck:util:ActionSequence:add	0.0026256411
15	root:org:jboss:cdi:tck:util:ActionSequence:checkStringValue	0.0026256411

Rank #9

• Schwa: Goodness
• Commit: 1bc2e2d
• Diagnostic Cost: 0
• Configuration: #3

Pos	Component	Probability
1	root:org:jboss:cdi:tck:util:ActionSequence:toString	0.6721633400000001
2	root:org:jboss:cdi:tck:impl:ConfigurationImpl:setTestTimeoutFactor	0.3615815700000001
3	root:org:jboss:cdi:tck:util:Timer:start	0.180790814
4	root:org:jboss:cdi:tck:util:Timer:checkConfiguration	0.180790814
5	root:org:jboss:cdi:tck:util:ActionSequence:getSequence	0.08402063200000001
6	root:org:jboss:cdi:tck:impl:ConfigurationImpl:getTestTimeoutFactor	0.04519773600000001
7	root:org:jboss:cdi:tck:util:Timer:setDelay	0.04519773600000001
8	root:org:jboss:cdi:tck:util:Timer:reset	0.04519773600000001
9	root:org:jboss:cdi:tck:util:Timer:clearStopConditions	0.04519773600000001
10	root:org:jboss:cdi:tck:util:ActionSequence:reset	0.042010326
11	root:org:jboss:cdi:tck:util:ActionSequence:addAction	0.042010326
12	root:org:jboss:cdi:tck:util:ActionSequence:getData	0.028512467
13	root:org:jboss:cdi:tck:impl:ConfigurationFactory:get	0.0056497171999999995
14	root:org:jboss:cdi:tck:util:ActionSequence:add	0.0026256411
15	root:org:jboss:cdi:tck:util:ActionSequence:checkStringValue	0.0026256411