Baseline & Long Range Arena

[kmheckel]

General approach:

• FP training as a reference baseline / target

• PTQ as a baseline for quantization

• QAT results

• (optional) QAT starting from the FP checkpoint (i.e., finetuning)

  • analysis of compute (during training) vs. performance (accuracy) vs. efficiency (# of bits / compression)

• Sequential MNIST @stevenabreu7

  • PTQ results @stevenabreu7

LRA:

• sequential CIFAR @stevenabreu7
  • PTQ results @stevenabreu7
• ListOps @stevenabreu7
  • PTQ results @stevenabreu7
  • improve QAT results (all are bad..)
• Text (imdb) @stevenabreu7
  • PTQ results @stevenabreu7
  • improve QAT results (only W4 and W2 with W_a8 worked..)
• Retrieval @AlessandroPierro
  • PTQ results
• Pathfinder @AlessandroPierro
  • PTQ results
• Path-X @AlessandroPierro
  • PTQ results

Bonus:

• Speech Commands 35

[stevenabreu7]

Isn't LRA with sequential CIFAR-10 instead of MNIST?

[kmheckel]

You're probably right - I didn't copy down all of the benchmarks and was basing the list off what was in the S5 repo so this may not be a perfect list. Will fix it.

…

On Wed, May 1, 2024, 22:45 Steven Abreu ***@***.***> wrote: Isn't LRA with sequential CIFAR-10 instead of MNIST? — Reply to this email directly, view it on GitHub <#9 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMG7YKBFQEIRUXT527CFPMTZAFPBDAVCNFSM6AAAAABHCGBJQWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBZGE4DMNJWHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[stevenabreu7]

We could also just do both (since MNIST is relatively fast)

[AlessandroPierro]

Retrieval

• FP16: in progress
• W8A8: in progress
• W4A8: in progress
• W2A8: in progress

Pathfinder

• FP16: DONE
• W8A8: in progress
• W4A8: in progress
• W2A8: in progress

Path-X