Skip to content

(TG) TG model perf tests #431

(TG) TG model perf tests

(TG) TG model perf tests #431

Triggered via schedule November 10, 2024 12:03
Status Failure
Total duration 2h 28m 17s
Artifacts 2
build-artifact  /  ...  /  build-docker-image
18s
build-artifact / build-docker-image / build-docker-image
Matrix: build-artifact / build-artifact
Matrix: tg-model-perf-tests / tg-model-perf-tests
Fit to window
Zoom out
Zoom in

Annotations

7 errors, 9 warnings, and 7 notices
tg-model-perf-tests / TG CNN model perf tests
Process completed with exit code 1.
tg-model-perf-tests / TG LLM model perf tests
The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
tg-model-perf-tests / TG LLM model perf tests
Process completed with exit code 2.
pcie-cards-are-being-used-cleanup
Tenstorrent cards seem to be in use. Killing PIDs and exiting unsuccessfully. This can happen if a test hung and is normally an issue with the test, rather than infra.
unsuccessful-reset-cleanup
Unable to reset board successfully, rebooting
tg-model-perf-tests / TG LLM model perf tests
The operation was canceled.
tg-model-perf-tests / TG LLM model perf tests
The action 'Run model perf regression tests' has timed out after 60 minutes.
unsuccessful-reset-attempt-cleanup
Unsuccessful board reset, trying again in 1 minute ...
unsuccessful-reset-attempt-cleanup
Unsuccessful board reset, trying again in 1 minute ...
unsuccessful-reset-attempt-cleanup
Unsuccessful board reset, trying again in 1 minute ...
unsuccessful-reset-attempt-cleanup
Unsuccessful board reset, trying again in 1 minute ...
unsuccessful-reset-attempt-cleanup
Unsuccessful board reset, trying again in 1 minute ...
unsuccessful-reset-attempt-cleanup
Unsuccessful board reset, trying again in 1 minute ...
unsuccessful-reset-attempt-cleanup
Unsuccessful board reset, trying again in 1 minute ...
unsuccessful-reset-attempt-cleanup
Unsuccessful board reset, trying again in 1 minute ...
unsuccessful-reset-attempt-cleanup
Unsuccessful board reset, trying again in 1 minute ...
printing-out-smi-info-cleanup
Touching and printing out SMI info
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful
printing-out-smi-info-cleanup
Touching and printing out SMI info
attempting-reset-cleanup
Attempting to reset card(s).
successful-reset-cleanup
tt-smi reset was successful
reset-successful-cleanup
tt-smi reset was successful

Artifacts

Produced during runtime
Name Size
TTMetal_build_wormhole_b0
51.4 MB
perf-report-csv-CNN-wormhole_b0-
526 Bytes