Some thoughts of defining ULP tolerance of exp op

[BruceDai]

Using ULP based comparison method, we ran exp op float32 tests by WebNN Native DirectML backend, observed that there're fluctuation ULP distances with different inputs.

Input	Output by WebNN Native DirectML	Expected by WebNN Baseline	ULP Distance
-22.99204874532912	1.0343804979218163e-10	1.0343799702128246e-10	8
-7.540661071653204	0.0005310465348884463	0.0005310464459828795	2
-2.543807029272031	0.07856672257184982	0.07856672389748613	0
20.0467240906249	508371712	508372035.3858906	10
4.694365060121655	109.32937622070312	109.32936902288711	1
22.620644348144452	6668381184	6668390398.21978	18
21.558996466377643	2306496512	2306498628.5912433	8
6.216680878598737	501.03753662109375	501.0374649351999	2
24.878306448588418	63754645504	63754553672.34833	22
1.84147599931147	6.305838584899902	6.305838809277489	0
14.240338631858862	1529327.625	1529327.465744619	1
24.46251196998665	42066206720	42066265840.60161	14
10.04045060355589	22935.72265625	22935.715503452648	4
6.695826149584278	809.0220336914062	809.0220314133451	0
5.814379130575155	335.0833740234375	335.0832909338617	3
9.914684439586907	20225.1875	20225.196136128863	4
9.314918894083625	11102.41796875	11102.42569029876	8

Referring to Appendix A: Vulkan Environment for SPIR-V / Precision and Operation of SPIR-V Instructions, there's a single precision definition for exp instruction as following

Instruction	Single precision, unless decorated with RelaxedPrecision	Half precision
exp(x), exp2(x)	3+2×∣x∣ ULP	1+2×∣x∣ ULP

Above table is from Table 85. Precision of GLSL.std.450 Instructions.

Then, we could have these ULP distance information:

Input	ULP Distance of WebNN	ULP Distance of Vulkan
-22.99204875	8	48.98409749
-7.540661072	2	18.08132214
-2.543807029	0	8.087614059
20.04672409	10	43.09344818
4.69436506	1	12.38873012
22.62064435	18	48.2412887
21.55899647	8	46.11799293
6.216680879	2	15.43336176
24.87830645	22	52.7566129
1.841475999	0	6.682951999
14.24033863	1	31.48067726
24.46251197	14	51.92502394
10.0404506	4	23.08090121
6.69582615	0	16.3916523
5.814379131	3	14.62875826
9.91468444	4	22.82936888
9.314918894	8	21.62983779

According to above observation, I don't think the ULP tolerance being a fixed value applies to exp op.

@wchao1115 PTAL, thanks.

[fdwr]

Note the appendix A since moved to chapter 48.

[fdwr]

we ran exp op float32 tests by WebNN Native DirectML backend

Are the float64 input values cast to float32 first before being passed through the baseline reference? Otherwise that can contribute additional differences (it's not enough alone to account for the variance, but a little) since DML would receive a rounded float32 input tensor whereas the baseline would receive the original full precision value. I ask because some of the values don't appear directly representable in float32 but rather between expressible float32 values. e.g.

24.46251196998665_____ as float64 from table above, which is between possible float32 values
24.4625110626220703125 (0x41C3B339) as float32 when rounded down
24.462512969970703125_ (0x41C3B33A) as float32 when rounded up

[BruceDai]

Close it by @fdwr has already provided this tolerance criterial as below on #265 (comment).

Op	Old Proposed ULP Tolerance	float16	float32	notes
exp	2	1 ULP	32 ULP	ULP is typically very small (0 to 2), but negative values can yield larger deltas (e.g. exp(-36.7462921143) yields ULP± 27 on my machine). float16 is actually computed using float32 (so 1 ULP for final roundoff).