Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WPT tests tracker / 2 for remaining ops #338

Closed
BruceDai opened this issue Feb 2, 2023 · 6 comments
Closed

WPT tests tracker / 2 for remaining ops #338

BruceDai opened this issue Feb 2, 2023 · 6 comments
Labels

Comments

@BruceDai
Copy link
Contributor

BruceDai commented Feb 2, 2023

Here's process of WPT tests for remaining 20 ops with @fdwr‘s great efforts on these operations tolerances, thanks much @fdwr!

Note: Set l2Pool2d being P3 is due to issue #278.

I synced with @fdwr about tolerance definitions for gru/gruCell and lstm/lstmCell,since they're not operators but more like entire miniature graphs themselves, we need community for some help of defining them, thanks.

Operations WebNN-Baseline implementation WebNN-Polyfill implementation WPT tests (PR link) Priority Note
hardSwish P1 It's easy to implement
resample2d P1 Required by Semantic Segmentation - DeepLabv3
pRelu P1 It's easy to implement
pad (updated) P1
elu P1 It's easy to implement
instanceNormalization P1 Required by Fast Style Transfer
hardSigmoid P1 It's easy to implement
linear P1 It's easy to implement
softplus pr P1 It's easy to implement
softsign P1 It's easy to implement
reduceL1 P2
reduceL2 P2
reduceLogSum P2
reduceLogSumExp P2
reduceSumSquare P2
gru P2 Required by RNNoise, NSNet
gruCell P2 Required by Recurrent network
lstm P2 Required by Recurrent network
lstmCell P2 Required by Recurrent network
l2Pool2d P3 There's an issue #278
@BruceDai
Copy link
Contributor Author

BruceDai commented Feb 2, 2023

Link to #265

@fdwr
Copy link
Collaborator

fdwr commented Feb 6, 2023

Operator tolerances:

Op float16 float32 notes
elu 18 ULP 18 ULP if a >= 0 then a else (expₑ(a) - 1) * alpha
hardSwish 4 ULP 4 ULP if x <= -threshold then 0 elif x >= threshold else x * (x + threshold) / (2 * threshold) // threshold = 3
gru ? ULP ? ULP
gruCell ? ULP ? ULP
hardSigmoid 2 ULP 2 ULP clamp(a * alpha + beta, 0, 1)); defaults alpha = 0.2, beta = 0.5
instanceNormalization ? ULP ? ULP ⏳ (a - mean) * scale / sqrt(variance + epsilon) + bias ¹
linear 2 ULP 2 ULP x * scale + bias
lstm ? ULP ? ULP
lstmCell ? ULP ? ULP
l2Pool2d IEPOE * 2 + 1 ULP IEPOE * 2 + 1 ULP sqrt(a0^2 + a1^2 + … + an^2)
pad 0 ULP 0 ULP should be exact, no math
prelu 1 ULP 1 ULP if a >= 0 then a else a * slope
reduceL1 <=IEPOE ULP <=IEPOE ULP abs(x1) + abs(x2) + ... + abs(xn)
reduceL2 IEPOE * 2 + 1 ULP IEPOE * 2 + 1 ULP sqrt(a1^2 + a2^2 + ... + an^2)
reduceLogSum IEPOE + 18 ULP IEPOE + 18 ULP log(a1 + a2 + ... + an)
reduceLogSumExp IEPOE*2 + 18 ULP IEPOE*2 + 18 ULP log(exp(a1) + exp(a2) + ... + exp(an))
reduceSumSquare IEPOE * 2 ULP IEPOE * 2 ULP a1^2 + a2^2 + ... + an^2
resample2d ? ULP ? ULP
softplus 18 ULP 18 ULP logₑ(1 + expₑ(a))
softsign 3 ULP 3 ULP x / (1 + abs(a))

Earlier ones here: #265 (comment)

¹ "Instance" normalization is really spatial normalization, where the spatial channels are reduced and normalized, while batch and channel remain independent. So pass a list of axes just beyond the leading batch and channel dimensions (starting
at axis 2 up to the last spatial dimension). So for 4D, it's just mean variance normalization with axes: [2,3]. The mean and variance are computed on demand.

[update] Phase 3+:

Op float16 float32 notes
argMax 0 0
argMin 0 0
cast 1 1
elementwiseIf 0 0
equal 0 0
erf
expand 0 0
fillSequence 2 2
gather 0 0
greater 0 0
greaterOrEqual 0 0
identity 0 0
lesser 0 0
lesserOrEqual 0 0
logicalNot 0 0
meanVarianceNormalization
reciprocal 2 2
reshape 0 0
sqrt 1 1
triangularMatrix 0 0

@anssiko
Copy link
Member

anssiko commented Feb 1, 2024

@BruceDai can you provide on update on these remaining tests, can we close this issue?

@BruceDai
Copy link
Contributor Author

BruceDai commented Feb 2, 2024

Hi @anssiko, I'm sorry for not updating this status timely.

Please see the above updated table of V1 remaining operations, now except tests for these complex gru/gruCell/lstm/lstmCell operations which are WIP by @fdwr and me , others are almost done (remaining prs were under reviewing or approved).

Regarding to V2 transformers operations, they've been already tracked by #485, now there're two remaining prs for constant (fillSequence) and triangular which are discussed on #492 and #494.

Thanks a lot for great supports from @fdwr @huningxin @Honry and @mei1127 ! 👍

@anssiko
Copy link
Member

anssiko commented Feb 5, 2024

@BruceDai @fdwr much thanks for your work on wpt coverage improvements.

I noted this in #240. We are expected to share an update on the wpt coverage by the upcoming CR publication transition expected next month. 100% test coverage is not expected for this transition, but the closer we get the better.

@BruceDai
Copy link
Contributor Author

Since recent gru/gruCell tests and lstm/lstmCell tests landed into WPT, now we have all conformance tests to cover current 78 operators. So I'm going to close this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants