Fix ETW trace logging crash in multithreading situations #21566

smk2007 · 2024-07-30T18:22:49Z

Description

ETW trace logger is fakely registered as initialized_ is marked as true before the registration is done, causing crashing issue for Lenovo camera application.

A prior attempt to address was made here: #21226
It was reverted here: #21360

Motivation and Context

The problem is that during initialization of TraceLoggingRegisterEx, it will reinvoke the callback and attempt reinitialization, which is not allowed. TraceLoggingRegisterEx however can be initialized concurrently when initialization happens on multiple threads. For these reasons it needs to be protected by a lock, but the lock cannot naively block because the callback's reinvocation will cause a deadlock.

To solve this problem another tracking variable is added : "initializing" which protects against reinitialization during the first initialization.

…completed successfully.

…ceLoggingRegisterEx

onnxruntime/core/platform/windows/logging/etw_sink.cc

…id reinitializing logger.

onnxruntime/core/platform/windows/logging/etw_sink.cc

fdwr

👍

ETW trace logger is fakely registered as initialized_ is marked as true before the registration is done, causing crashing issue for Lenovo camera application. A prior attempt to address was made here: #21226 It was reverted here: #21360 The problem is that during initialization of TraceLoggingRegisterEx, it will reinvoke the callback and attempt reinitialization, which is not allowed. TraceLoggingRegisterEx however can be initialized concurrently when initialization happens on multiple threads. For these reasons it needs to be protected by a lock, but the lock cannot naively block because the callback's reinvocation will cause a deadlock. To solve this problem another tracking variable is added : "initializing" which protects against reinitialization during the first initialization. --------- Co-authored-by: Sheil Kumar <[email protected]>

…1603) ETW trace logger is fakely registered as initialized_ is marked as true before the registration is done, causing crashing issue for Lenovo camera application. A prior attempt to address was made here: #21226 It was reverted here: #21360 The problem is that during initialization of TraceLoggingRegisterEx, it will reinvoke the callback and attempt reinitialization, which is not allowed. TraceLoggingRegisterEx however can be initialized concurrently when initialization happens on multiple threads. For these reasons it needs to be protected by a lock, but the lock cannot naively block because the callback's reinvocation will cause a deadlock. To solve this problem another tracking variable is added : "initializing" which protects against reinitialization during the first initialization. --------- ### Description A few CI pipelines are disabled because they are no longer compatible with this legacy GE branch. ### Motivation and Context  --------- Co-authored-by: Sheil Kumar <[email protected]> Co-authored-by: Xiang Zhang <[email protected]>

### Description ETW trace logger is fakely registered as initialized_ is marked as true before the registration is done, causing crashing issue for Lenovo camera application. A prior attempt to address was made here: #21226 It was reverted here: #21360 ### Motivation and Context The problem is that during initialization of TraceLoggingRegisterEx, it will reinvoke the callback and attempt reinitialization, which is not allowed. TraceLoggingRegisterEx however can be initialized concurrently when initialization happens on multiple threads. For these reasons it needs to be protected by a lock, but the lock cannot naively block because the callback's reinvocation will cause a deadlock. To solve this problem another tracking variable is added : "initializing" which protects against reinitialization during the first initialization. --------- Co-authored-by: Sheil Kumar <[email protected]>

Sheil Kumar added 3 commits July 22, 2024 11:27

Move ETW initialization check assignment to after initialization has …

8b71d89

…completed successfully.

Add initialization check to prevent re-entry into the lock during Tra…

afd6858

…ceLoggingRegisterEx

Update logging to honor initialization checks

dbac3b9

smk2007 requested review from snnn, zhangxiang1993, adrianlizarraga and martinb35 July 30, 2024 18:22

PatriceVignola reviewed Jul 30, 2024

View reviewed changes