Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conformer workload variants #590

Merged
merged 35 commits into from
Dec 7, 2023
Merged

Conformer workload variants #590

merged 35 commits into from
Dec 7, 2023

Conversation

priyakasimbeg
Copy link
Contributor

@priyakasimbeg priyakasimbeg commented Nov 27, 2023

Merge after Criteo variants PR/568

Add Conformer workload variants:

  • GELU
  • remove post layernorm
  • increase attention temperature to 1.6

Added modeldiff compare tests for the variants.

Note that for GELU in PyTorch we have to override to the default 'approximate' arg to specify the tanh approximation which is used by default in the JAX GELU implementation.

Copy link

github-actions bot commented Nov 27, 2023

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

@priyakasimbeg priyakasimbeg changed the title [WIP] Conformer workload variants Conformer workload variants Nov 28, 2023
@priyakasimbeg priyakasimbeg changed the title Conformer workload variants [WIP] Conformer workload variants Nov 28, 2023
@priyakasimbeg priyakasimbeg changed the title [WIP] Conformer workload variants [BLOCKED on PR/568] Conformer workload variants Nov 28, 2023
@priyakasimbeg priyakasimbeg requested a review from znado November 28, 2023 00:46
@priyakasimbeg priyakasimbeg marked this pull request as ready for review December 5, 2023 16:09
@priyakasimbeg priyakasimbeg requested a review from a team as a code owner December 5, 2023 16:09
znado
znado previously approved these changes Dec 7, 2023
@priyakasimbeg priyakasimbeg dismissed znado’s stale review December 7, 2023 20:16

The merge-base changed after approval.

@priyakasimbeg priyakasimbeg changed the title [BLOCKED on PR/568] Conformer workload variants Conformer workload variants Dec 7, 2023
@priyakasimbeg priyakasimbeg merged commit 5b815f6 into dev Dec 7, 2023
31 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Dec 7, 2023
@priyakasimbeg priyakasimbeg deleted the conformer_workload_variants branch December 14, 2023 23:20
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants