Copyright © 2020 Rafael Laboissière <[email protected]>
Released under the terms of the GNU General Public License, version 3 or later. No warranties.
In this study, we present a statistical analysis of the Debian General Resolution: “Init systems and systemd” 2019. Exploratory Factorial Analysis was applied to the tally data, considering each option as a variable and the ranks cast in the ballot as the values for each option. The goal of this study is to understand both the structure of the choices made by the Debian Developers community and how the available options were perceived, in relation to each other. Four factors were obtained from the analysis. The first one can be interpreted as “systemd vs. multiple init implementations”. The second one could be related to the “desire for community cohesion”. The third one follows the result of the Condorcet outcome. Polarization of the community is particularly evidenced by the distribution of scores along the third factor. This study sheds some lights on the subtleties of the voters’ behavior and goes beyond the Condorcet result based on the outranking matrix.
At the end of 2019, a General Resolution regarding Init systems and systemd in Debian in Debian was discussed and voted by the Debian developers. The proposals and amendments period ended on 2019-11-16, the discussion period lasted until 2019-11-22 and the voting period started on 2019-12-07 and finished on 2019-12-27.
The proposals under vote were the following:
- F: Focus on systemd
- B: Systemd but we support exploring alternatives
- A: Support for multiple init systems is Important
- D: Support non-systemd systems, without blocking progress
- H: Support portability, without blocking progress
- E: Support for multiple init systems is Required
- G: Support portability and multiple implementations
Together with those proposals, the voters could also choose the option “further discussion” (fd hereafter). Sam Hartman wrote a voting guide, focusing on the technical effects of each proposal.
Debian uses the Condorcet method for voting general resolutions. In casting their votes, voters ranked the eight options above, possibly with ties. Considering all possible two-way races between options, the Condorcet winner, if there is one, is the one that can beat each other candidate in a two-way race with that candidate.
The outcome of the vote shows the following ordering of the options, starting with the winner: B, F, H, D, G, A, fd, and E.
As with many technical issues related to packaging directives in Debian, the debate has been intense within the debian-vote mailing list and also elsewhere (debian-devel and debian-project mailing lists). The goal of the present study is not to discuss the merit of each proposal but rather to understand, thanks to a statistical analysis, the structure of the vote within the Debian Developers community. In other words, I seek to (1) understand how the different options were perceived in relation to each other, (2) verify whether some options are perceived as redundant with others, and (3) explain the collective behavior of the community with a limited amount of factors.
This was accomplished by applying Exploratory Factor Analysis (EFA) to the tally data. EFA is a statistical method aimed at discovering the underlying structure of a set of variables (the ballot options in our case), yielding a reduced amount of (hopefully sensible) latent factors. We used here the rank of each option as the scale for measuring its importance in each ballot. The final result of the EFA is twofold: first, a limited set of factors on which each variable has a “loading” (factors are linear combination of the variables) and, second, the predicted scores of each ballot along the factor axes.
The ballots were obtained from the tally sheet, which contains the 425 valid votes.
The instructions for casting the vote were the following:
There are 8 choices in the form, which you may rank with numbers between 1 and 8. In the brackets next to your preferred choice, place a 1. Place a 2 in the brackets next to your next choice. Continue until you reach your last choice. Do not enter a number smaller than 1 or larger than 8.
In the EFA done in the present study, the value cast for each option can be
seen as the “note” for that option and the data is appropriate for EFA.
The values are bounded between 1 to 8 and each ballot should contain,
according to the vote instructions, at least one variable with value equal
to 1 (this is the case for the large majority of ballots, except for one).
This somehow “normalizes” the data, in the sense that the voter’s preferred
option has value 1 and all the other options are judged relatively to it.
Notice that the voting software would consider ballots like 12222222 and
18888888 as equivalent, but our analysis will consider them as different,
reflecting the possible intention of the voter (in the case of ballot
18888888, options #2 to #8 have a much lower value, in respect to option
#1, than in ballot 12222222).
Instructions regarding the unranked choices were as follows:
You may skip numbers, leave some choices unranked, and rank options equally. Unranked choices are considered equally the least desired choices, and ranked below all ranked choices.
In the present study, unranked options have a value equal to the least
ranked value minus one. So, for instance, the ballot 1234---- became
12345555 in our study. As an exception, if the least value assigned in
the ballot is 8, then the unranked options are also equal to 8.
The EFA results was done with the function factanal of the R statistical
software. The factor axes were transformed by using the non-orthogonal
promax rotation method. Non-orthogonal rotation of the axis are more
appropriate than orthogonal rotation when factors are correlated.
Moreover, non-orthogonal rotations yield higher eigenvalues for the factors
and may improve the interpretation of the results, since variables may
become more isolated among the factors.
With eight variables in the analysis, the function factanal limits the
number of factors in the range 2 to 4. We will use the maximum number of 4
factors in the analysis shown below.
The output of the R code is the following:
===== Exploratory factorial analysis
Call:
factanal(x = ~F + B + A + D + H + E + G + fd, factors = 4, data = dat,
scores = "regression", rotation = "promax")
Uniquenesses:
F B A D H E G fd
0.056 0.276 0.005 0.414 0.005 0.218 0.748 0.568
Loadings:
Factor1 Factor2 Factor3 Factor4
F 0.937 -0.139 -0.117
B 0.622 -0.412 0.243
A -0.109 1.010
D 0.735
H 1.004
E 0.872 0.104
G 0.153 0.380
fd 0.628 0.400
Factor1 Factor2 Factor3 Factor4
SS loadings 1.682 1.589 1.263 1.115
Proportion Var 0.210 0.199 0.158 0.139
Cumulative Var 0.210 0.409 0.567 0.706
Factor Correlations:
Factor1 Factor2 Factor3 Factor4
Factor1 1.0000 0.0576 -0.498 0.333
Factor2 0.0576 1.0000 0.237 0.085
Factor3 -0.4977 0.2374 1.000 -0.252
Factor4 0.3331 0.0850 -0.252 1.000
Test of the hypothesis that 4 factors are sufficient.
The chi square statistic is 10.1 on 2 degrees of freedom.
The p-value is 0.0064
===== Correlation between factors #1 and #2
Pearson's product-moment correlation
data: sc[, 1] and sc[, 2]
t = 11.212, df = 423, p-value < 2.2e-16
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
0.4018258 0.5487867
sample estimates:
cor
0.4786518
We see that the four factors explain 70.6% of the variance. Even though this is still not enough for significantly accounting for the variance in the data (χ²[2] = 10.1, p < 0.01), we will stick to the four factors, since interesting and sensible interpretations can be inferred from them.
A graphic depiction of the resulting factors is shown in the figure below. The heights of the bars are loading values for each option (notice that entries with small absolute values are not shown in the loading matrix in the textual result above).
The projection of the loading onto the bidimensional plane defined by factors #1 and #2 is shown in the figure below.
The predicted scores along factors #1 and #2 for each voter is shown in the figure below. Each point corresponds to a ballot. They are represented with transparent dots, such that points that appear darker correspond to the superposition of two or more points (the more superimposed points, the darker the dot becomes).
Probability densities of the scores along each factor axis are shown in the figure below. The horizontal axes represent the factor axis, while the vertical axis represent the probability density. The projected scores are shown as a scatter plot below the horizontal axis. For the sake of visualization clarity, random jitter has been added vertically to each point.
From the loading matrix, we can see that the following options can be grouped together: F+B, D+H and G+E. This is not surprising, because both options in each pair have similarities in their proposals. Accordingly, the Debian developers who participated to the vote seem to have perceived those pairs as similar options.
A and fd seem to have be considered as isolated options, as regards the EFA results.
The first factor (21% of explained variance), which has high loadings for F and B options, can be regard as the “systemd vs. multiple init implementations” factor. Notice that option F (which proposes to focus on systemd), has higher loading that B (which is less radical than F) along this first factor. It is also interesting to note that option fd (further discussion) has also a high loading on this factor. This means that the, along this factor, voters tended to get a strong opinion either rejecting or selecting options F and B and, at the same time, considering unacceptable the remaining options.
The second factor (19.9% of explained variance) opposes options D and H to the others. Those two options seem to be less divisive than the others, in particular options F and E. We could interpret factor #2 as the importance that one attaches, or not, to the cohesion of the community, by placing options D and H either higher of lower, respectively, as regards the other options.
The third factor (15.8% of explained variance) reflects almost perfectly the outcome of the vote, options B and E being at opposite ends. The loading values determines the ordering B, F, H, D, A, G, fd, and E. The only difference with the vote outcome is the flipping between options A and G. It is interesting to notice, though, that option G beats option A by only 11 votes. Moreover, options H and D beat option D more severely (155 and 168 votes, respectively) than option A (102 and 122 votes, respectively). We could then call it as the “GR outcome” factor.
The fourth factor (13.9% of explained variance) implies only the A option and reflects how high or low this option was ranked by the voters. We could call it the “proposal *A*” factor.
As the scatter plot of scores on the factor#1 × factor#2 plane shows, those two factors are correlated (R = 0.48, t[423] = 11.2, p < 0.001). This justifies the use of the non-orthogonal rotation in the EFA. Notice that voters who scored high in the “systemd-preferred” options (negative value for factor #1) also scored high in the “community cohesion” direction (negative value for factor #2).
We should issue a caveat here, as regards the assertion above. It may be the case, as it is implicitly admitted in the previous paragraph, that voters who are more inclined to choose systemd-preferred solutions would also care more about community cohesion. That may reflect the reality, but we could also consider that my previous interpretation of #2 as being the “community cohesion” factor is not a fully precise depiction of the situation.
Just for the sake of clarity, let us explain the sense of negative values in the scores. Let us take, for instance, factor #2, which has positive values for options D and H. Voters that have negative values along #2 would have put options D and H below their mean values, i.e. towards he value 1.
Finally, the behavior of the voters is also reflected in the distribution of scores along the factor axes. For instance, the score along factor#3 may reflect the amount of agreement with the final outcome of the vote. The higher this score is, the more the voters have put option B close to value 1 and option E close to value 8. We clearly see a bimodal distribution with peaks in the negative and positive sides of the factor axis. This is an indication of polarization of the community around the issue at stake. Notice that the distribution along factor #1 has also a bimodal distribution, while distribution of the scores along factors #2 and #4 are more unimodal .
As a final remark, please notice that the interpretations above are based on correlations. In any case. we should imply that there is a causality in one direction or the other. For instance, let us take factor #3, which was interpreted as being the “GR outcome.” Of course, this does not mean that voters knew beforehand the outcome of the vote and cast their ballot accordingly, what would be a circular reasoning. Keep in mind that correlation does not imply causation.
The tally sheet of a Condorcet voting system allows for a multifaceted interpretation of vote results. The analysis done in the present study allows us to go beyond the crude result of the outranking matrix and to understand the subtleties in the behavior of the voters of the Debian General Resolution: “Init systems and systemd.”
Many thanks to Sébastien Villemot for his thoughtful comments and
suggestions for improvement of the present study, as well as for revising
the final text. Thanks to Taowa Munene-Tardif for pointing out that there
was one ballot without value 1. Thanks to Gregor Herrmann, Ana Guerrero
López, and Martin Michlmayr for fixing typos and to Sam Hartman for the
helpful discussion on the results.
After publication of this study, my attention has been drawn to a very important issue, regarding the interpretation of factor #2. In the present study, I chose to name it the “community cohesion” factor. This choice of denomination was done solely as a way to distinguish options D+H from the (apparently) more divisive options F+B and G+E. It must be said that, firstly, this choice of naming is totally subjective and does not follow naturally from the EFA results. Secondly, it must also be emphasized that the authors of proposals H and D may not have had “community cohesion” in mind when they wrote their proposals. Moreover, there is no guarantee that the community would attain a higher degree of cohesion, had either option H or D won the vote.
The code for producing the analysis is available in this Git repository. The pre-processing of the tally sheet is done by the Python script process-tally.py and the factorial analysis and plotting of the results in the R script vote-2019-002-factanal.r. There is also a Makefile for automating the process.