Enable >2GB models + allow model paths to be passed for generate_artifacts API #20958

carzh · 2024-06-06T20:29:23Z

Description

Alternative design from #20942

Allow users to pass in a model path for the generate_artifacts API.

Motivation and Context

ONNX API calls such as the onnx checker + shape inference fail when given a model > 2GB, but work if a path to a model >2GB is passed in.

…ssor, so that onnx.load is called twice instead of three times

orttraining/orttraining/python/training/artifacts.py

orttraining/orttraining/python/training/onnxblock/model_accessor.py

orttraining/orttraining/python/training/onnxblock/blocks.py

orttraining/orttraining/python/training/onnxblock/loss/loss.py

orttraining/orttraining/python/training/onnxblock/onnxblock.py

orttraining/orttraining/test/python/orttraining_test_ort_apis_onnxblock.py

…date

orttraining/orttraining/python/training/onnxblock/blocks.py

orttraining/orttraining/test/python/orttraining_test_ort_apis_onnxblock.py

…nerate_artifacts typically will throw an error if unsuccessful

& relative path for external data

…r deletion

carzh added 2 commits June 6, 2024 18:26

added string option to generate artifacts

d5418ca

switched to passing both the modelproto + modelpath to the model acce…

924ceda

…ssor, so that onnx.load is called twice instead of three times

github-advanced-security bot found potential problems Jun 6, 2024

View reviewed changes

orttraining/orttraining/python/training/artifacts.py Fixed Show fixed Hide fixed

carzh added 2 commits June 12, 2024 14:31

wrote unit tests + added check for large files + fixed some bugs

e458492

removed unnecessary helper functions + import from unit tests

8069700

github-advanced-security bot found potential problems Jun 12, 2024

View reviewed changes

orttraining/orttraining/python/training/onnxblock/model_accessor.py Fixed Show fixed Hide fixed

orttraining/orttraining/python/training/onnxblock/model_accessor.py Fixed Show fixed Hide fixed

carzh added 2 commits June 12, 2024 22:48

fine I'll use X|Y type annotations

29ad02a

lintrunner format

a35bc80

carzh requested review from baijumeswani and skottmckay June 13, 2024 06:15

carzh marked this pull request as ready for review June 13, 2024 06:15

skottmckay reviewed Jun 13, 2024

View reviewed changes

carzh added 3 commits June 13, 2024 07:03

removed some print statements + cleaned up some code

72a55c7

updated handling of temp files + added comment on crossentropyloss up…

378afa8

…date

Merge branch 'main' into carzh/generate-artifacts-large-models

c27820e

skottmckay reviewed Jun 18, 2024

View reviewed changes

orttraining/orttraining/python/training/onnxblock/blocks.py Outdated Show resolved Hide resolved

orttraining/orttraining/test/python/orttraining_test_ort_apis_onnxblock.py Show resolved Hide resolved

carzh added 2 commits June 19, 2024 18:31

changed release method to __del__ & added comments to explain that ge…

44acd08

…nerate_artifacts typically will throw an error if unsuccessful

comments

0f93974

baijumeswani previously approved these changes Jun 19, 2024

View reviewed changes

switched to setting absolute path in constructor for temp onnx file path

fff6723

& relative path for external data

carzh dismissed baijumeswani’s stale review via fff6723 June 20, 2024 07:29

updated variable name + finds absolute path for external data file fo…

c5a5cc1

…r deletion

skottmckay previously approved these changes Jun 20, 2024

View reviewed changes

lintrunner sigh

8fcd74c

carzh dismissed skottmckay’s stale review via 8fcd74c June 20, 2024 23:10

baijumeswani approved these changes Jun 20, 2024

View reviewed changes

carzh merged commit 6236707 into microsoft:main Jun 21, 2024
94 of 98 checks passed

carzh mentioned this pull request Jun 21, 2024

[Training] protobuf limit reached when generating training artifacts for a onnx model more than 2GB. #18874

Closed

jkbeavers mentioned this pull request Oct 11, 2024

[Training] >2GB Model Offline Artifacts fail with MSE Loss #22411

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable >2GB models + allow model paths to be passed for generate_artifacts API #20958

Enable >2GB models + allow model paths to be passed for generate_artifacts API #20958

carzh commented Jun 6, 2024

Enable >2GB models + allow model paths to be passed for generate_artifacts API #20958

Enable >2GB models + allow model paths to be passed for generate_artifacts API #20958

Conversation

carzh commented Jun 6, 2024

Description

Motivation and Context