FightingVariance.wiki

SimulationCraft works according to Central limit theorem, which claims the following:

Let X1, X2, X3, ... Xn be a sequence of n independent and identically distributed random variables having each finite values of expectation µ and variance σ² > 0. Then the distribution of the sample average of these random variables approaches the normal distribution with a mean µ and variance σ² / n irrespective of the shape of the original distribution.

So if we set iterations=infinity, SimCraft will yield a precise result that will be perfect in comparison with theorycraft - no assumptions, pure modelling, but definitive output.

However, this is impossible and full raid sims take quite a significant amount of time even with few iterations. Thus we have to deal with variance in our results, that's especially noticeable when we calculate scaling, i.e. small differences between two dps.

---

Since r698 SimCraft supports tracking dps values (dps is the most important output characteristic of simulations), and we can make use of those.

Using 3σ-rule we can estimate confidence intervals for precise DPS values, namely:

• There's 95% probability that precise DPS lies within 1.96 * σ / sqrt( n ) neighbourhood of an average DPS

• There's 95% probability that precise DPS difference lies within 1.96 * sqrt( 1.96 ) * σ / sqrt( n ) neighbourhood of difference between two average DPS values (useful for scaling simulations and spec comparisons).

---

SimCraft also reports a series of statistical metrics, e.g. stddev, min and max, range, 10th and 90th percentile and #iterations needed for specific error thresholds.

As of r9633 it is possible to specify a custom confidence level, whereas the default remains 95%.

---

Charts explaining what the confidence means: The y axis tells with which probability the true dps is at a certain point. The confidence level is the filled area with the predetermined volume percent. e.g. with a 95% confidence level, the volume of the filled area is exactly 95% of the whole possible area from -infinity to +infinity.