Fully dynamic ETW controlled logging for ORT and QNN logs #20537
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… to WPR profile. +Code cleanup and formatting
…evel via ETW. However, this is commented out / disabled for now due to 3rd party QNN crash seen in QNN 2.19.2 (qnn-sdk-v2.19.2.24021)
ivberg
requested review from
adrianlizarraga,
HectorSVC,
jywu-msft and
vibhorbhattMS
…c_cast and RTTI. Instead use a custom type system on ISink that is safe for use with static_cast
|
+@snnn , @skottmckay who helped review #18882 |
HectorSVC
reviewed
May 7, 2024
… events as ORT verbose. This is to preserve existing expected behavior by some customers. It also unfortunately means the only pratical method to get any QNN logs is to fully set the level to VERBOSE
HectorSVC
reviewed
May 30, 2024
skottmckay
reviewed
May 31, 2024
skottmckay
reviewed
Jun 5, 2024
skottmckay
reviewed
Jun 6, 2024
skottmckay
approved these changes
Jun 6, 2024
HectorSVC
approved these changes
Jun 7, 2024
sophies927
added
release:1.18.1
triage:approved
ivberg
added a commit
that referenced
this pull request
Jun 20, 2024
### Description Under certain conditions with enabling & disabling ETW continuously, we got a crash report. Allows ETW callbacks to be de-registered upon class destructor. Related to #20537 ### Motivation and Context Fixes crash ### Callstack We see it crash in [0x0] onnxruntime!<lambda_967a738fca8512372f170fcaf2d094d4>::operator()+0x34 0x12941ff570 0x7ffa994f0a04 [0x1] onnxruntime!std::_Func_class<void,_GUID const *,unsigned long,unsigned char,unsigned __int64,unsigned __int64,_EVENT_FILTER_DESCRIPTOR *,void *>::operator()+0x54 0x12941ff7b0 0x7ffa994f0d64 [0x2] onnxruntime!onnxruntime::logging::EtwRegistrationManager::InvokeCallbacks+0xcc 0x12941ff7b0 0x7ffa994f0d64 [0x3] onnxruntime!onnxruntime::logging::EtwRegistrationManager::ORT_TL_EtwEnableCallback+0x94 0x12941ff860 0x7ffa98d19628 and seems to us that the this pointer captured in etwRegistrationManager.RegisterInternalCallback( [&etwRegistrationManager, this]( ... is no longer valid when the callback is called.
jywu-msft
removed
triage:approved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Windows - Fully dynamic ETW controlled logging for ORT and QNN logs
The logging support is documented here
Also add support for logging ORT SessionCreation on ETW CaptureState
Motivation and Context
The previous ETW support only worked if you enabled ETW before the session started. There can commonly be long-lived AI inference processes that need to be traced & debugged. This enables logging fully on the fly.
Without this support a dev would have to end up killing a process or stopping a service in order to get tracing. We had to do this for a recent issue with QNN, and it was a bit painful to get the logs and it ruined the repro.
Testing
I tested with the following cases
Related:
#18882
#19428