Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merge wc_reference_data_2024.4.yaml and wc_reference_data.yaml #2997

Open
wants to merge 2 commits into
base: develop
Choose a base branch
from
Open

Merge wc_reference_data_2024.4.yaml and wc_reference_data.yaml #2997

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5544aa0
merge wc_reference_data_2024.4.yaml and wc_reference_data.yaml
alexsu52 Sep 30, 2024
7b5fac3
Update wc_reference_data.yaml
alexsu52 Oct 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 8 additions & 7 deletions tests/post_training/data/wc_reference_data.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,35 +7,36 @@ tinyllama_data_aware_backend_OV:
num_int4: 94
num_int8: 124
tinyllama_data_aware_awq_stateful_backend_OV:
metric_value: 0.85616
metric_value: 0.85571
num_int4: 94
num_int8: 124
tinyllama_data_aware_awq_scale_estimation_backend_OV:
metric_value: 0.85502
metric_value: 0.86355
num_int4: 94
num_int8: 124
tinyllama_data_aware_awq_scale_estimation_stateful_backend_OV:
metric_value: 0.85502
metric_value: 0.86355
num_int4: 94
num_int8: 124
tinyllama_int8_data_free_backend_TORCH:
metric_value: 0.95624
num_int4: 0
num_int8: 312
tinyllama_data_aware_gptq_backend_OV:
metric_value: 0.87134
metric_value: 0.86521
num_int4: 94
num_int8: 124
metrics_xfail_reason: "Issue-148819"
tinyllama_scale_estimation_per_channel_backend_OV:
metric_value: 0.80873
metric_value: 0.80853
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't it be marked by xfail, since 197 build failed with 0.80873 reference?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In agreement with @l-bat, I updated the metrics. @l-bat, please, confirm that the result is reproducible on different hardware.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test has become unstable on openvino-nightly again (details at ticket: 152627)

num_int4: 188
num_int8: 124
tinyllama_data_aware_lora_stateful_backend_OV:
metric_value: 0.83446
num_int4: 94
num_int8: 500
tinyllama_NF4_scale_estimation_stateful_per_channel_backend_OV:
metric_value: 0.88663
metric_value: 0.87942
num_int4: 11
num_int8: 290
num_int8: 290
metrics_xfail_reason: "Issue-148819"
22 changes: 0 additions & 22 deletions tests/post_training/data/wc_reference_data_2024.4.yaml
View file
Open in desktop

This file was deleted.

Loading