Page crashes related to Owin

[nithyaRagupathy]

We are recently seeing multiple page crashes related to Owin and need SME help to investigate the issue.

Stack trace

We are seeing multiple page crashes related to Owin and need SME help to investigate the issue.

In our Code, we are using the packages with these versions-
Microsoft.Owin - 4.2.2
"Microsoft.Identity.ServiceEssentials.Extensions.OwinMiddleware - "1.28.1"
Microsoft.Identity.ServiceEssentials.Extensions.OwinMiddleware.S2S - "1.28.1"
Microsoft.IdentityModel.S2S" - "4.10.0"
Microsoft.IdentityModel.S2S.Tokens" - "4.10.0"
Microsoft.IdentityModel.S2S.Configuration" - "4.10.0"
Microsoft.IdentityModel.S2S.Extensions.Owin" - "4.9.1"
Microsoft.IdentityModel.S2S.Extensions.AspNetCore" - "4.10.0"
Microsoft.IdentityModel.Abstractions" - "8.1.0"
Microsoft.IdentityModel.JsonWebTokens" - "8.1.0"
Microsoft.IdentityModel.Logging" - "8.1.0"
Microsoft.IdentityModel.Protocols" - "8.1.0"
Microsoft.IdentityModel.Protocols.OpenIdConnect" - "8.1.0"
Microsoft.IdentityModel.Protocols.SignedHttpRequest" - "8.1.0"
Microsoft.IdentityModel.Protocols.WsFederation" - "8.1.0"
Microsoft.IdentityModel.Tokens" - "8.1.0"
Microsoft.IdentityModel.Validators" - "8.1.0"
Microsoft.IdentityModel.Xml" - "8.1.0"
System.IdentityModel.Tokens.Jwt" - "8.1.0"
Microsoft.Identity.Abstractions" - "6.0.0"

Given below is the stack trace from Watson dump

msvcrt!__ascii_stricmp+0xd [minkernel\crts\crtw32\string\stricmp.c @ 124]
iiscore!W3_RESPONSE::SetHeader+0x458 [servercommon\inetsrv\iis\iisrearc\iis70\core\w3response.cxx @ 638]
webengine4!MgdSetUnknownHeader+0xa7 [f:\dd\ndp\fx\src\xsp\webengine\mgdexports.cxx @ 854]
system_web!DomainNeutralILStubClass.IL_STUB_PInvoke+0xd6
system_web!System.Web.Hosting.IIS7WorkerRequest.SetUnknownResponseHeader+0xa7 [f:\dd\ndp\fx\src\xsp\system\Web\Hosting\IIS7WorkerRequest.cs @ 2076]
system_web!System.Web.Hosting.IIS7WorkerRequest.SetKnownResponseHeader+0x70 [f:\dd\ndp\fx\src\xsp\system\Web\Hosting\IIS7WorkerRequest.cs @ 2063]
system_web!System.Web.HttpHeaderCollection.SetHeader+0x147 [f:\dd\ndp\fx\src\xsp\system\Web\HttpHeaderCollection.cs @ 129]
microsoft_owin_host_systemweb!Microsoft.Owin.Host.SystemWeb.CallHeaders.AspNetResponseHeaders.Set+0xc9 [//src/Microsoft.Owin.Host.SystemWeb/CallHeaders/AspNetResponseHeaders.cs @ 108]
microsoft_owin!Microsoft.Owin.ResponseCookieCollection.Append+0x35e [//src/Microsoft.Owin/ResponseCookieCollection.cs @ 76]
microsoft_owin_security_openidconnect!Microsoft.Owin.Security.OpenIdConnect.OpenIdConnectAuthenticationHandler.RememberNonce+0x1a7 [/_/src/Microsoft.Owin.Security.OpenIdConnect/OpenidConnectAuthenticationHandler.cs @ 676]
microsoft_owin_security_openidconnect!Microsoft.Owin.Security.OpenIdConnect.OpenIdConnectAuthenticationHandler+ApplyResponseChallengeAsync_d__10 [//src/Microsoft.Owin.Security.OpenIdConnect/OpenidConnectAuthenticationHandler.cs @ 195]
mscorlib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[System.Security.Cryptography.CryptoStream+_WriteAsyncInternal_d__39 [[System.Security.Cryptography.CryptoStream+WriteAsyncInternal_d__39, mscorlib @ 322]
microsoft_owin_security_openidconnect!Microsoft.Owin.Security.OpenIdConnect.OpenIdConnectAuthenticationHandler.ApplyResponseChallengeAsync+0x4e [//src/Microsoft.Owin.Security.OpenIdConnect/OpenidConnectAuthenticationHandler.cs @ 640]
microsoft_owin_security!Microsoft.Owin.Security.Infrastructure.AuthenticationHandler+ApplyResponseCoreAsync_d__40 [//src/Microsoft.Owin.Security/Infrastructure/AuthenticationHandler.cs @ 179]
mscorlib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[System.Security.Cryptography.CryptoStream+_WriteAsyncInternal_d__39 [[System.Security.Cryptography.CryptoStream+WriteAsyncInternal_d__39, mscorlib @ 322]
microsoft_owin_security!Microsoft.Owin.Security.Infrastructure.AuthenticationHandler.ApplyResponseCoreAsync+0x4e [//src/Microsoft.Owin.Security/Infrastructure/AuthenticationHandler.cs @ 207]
mscorlib!System.Threading.LazyInitializer.EnsureInitializedCore[[System.__Canon, mscorlib]]+0x8b [[System.__Canon, mscorlib @ 241]
microsoft_owin_security!Microsoft.Owin.Security.Infrastructure.AuthenticationHandler+ApplyResponseAsync_d__39 [//src/Microsoft.Owin.Security/Infrastructure/AuthenticationHandler.cs @ 157]
mscorlib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[System.Security.Cryptography.CryptoStream+_WriteAsyncInternal_d__39 [[System.Security.Cryptography.CryptoStream+WriteAsyncInternal_d__39, mscorlib @ 322]
microsoft_owin_security!Microsoft.Owin.Security.Infrastructure.AuthenticationHandler.ApplyResponseAsync+0x4e [//src/Microsoft.Owin.Security/Infrastructure/AuthenticationHandler.cs @ 207]
microsoft_owin_security!Microsoft.Owin.Security.Infrastructure.AuthenticationHandler+TeardownAsync_d__34 [//src/Microsoft.Owin.Security/Infrastructure/AuthenticationHandler.cs @ 95]
mscorlib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[System.Security.Cryptography.CryptoStream+_WriteAsyncInternal_d__39 [[System.Security.Cryptography.CryptoStream+WriteAsyncInternal_d__39, mscorlib @ 322]
microsoft_owin_security!Microsoft.Owin.Security.Infrastructure.AuthenticationHandler.TeardownAsync+0x4e [//src/Microsoft.Owin.Security/Infrastructure/AuthenticationHandler.cs @ 128]
microsoft_owin_security!Microsoft.Owin.Security.Infrastructure.AuthenticationMiddleware1+_Invoke_d__5 [[System.__Canon, mscorlib @ 32] mscorlib!System.Threading.ExecutionContext.RunInternal+0x172 [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 980] mscorlib!System.Threading.ExecutionContext.Run+0x15 [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 928] mscorlib!System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner [f:\dd\ndp\clr\src\BCL\system\runtime\compilerservices\AsyncMethodBuilder.cs @ 1070] mscorlib!System.Threading.Tasks.AwaitTaskContinuation.RunCallback+0x6a [f:\dd\ndp\clr\src\BCL\system\threading\Tasks\TaskContinuation.cs @ 759] mscorlib!System.Threading.Tasks.Task.FinishContinuations+0xfe [f:\dd\ndp\clr\src\BCL\system\threading\Tasks\Task.cs @ 3642] mscorlib!System.Threading.Tasks.Task1[[System.__Canon, mscorlib]].TrySetResult+0x9ad45 [[System.__Canon, mscorlib @ 463]
mscorlib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder1[[System.__Canon, mscorlib]].SetResult+0x0 [[System.__Canon, mscorlib @ 590] mscorlib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder1[[System.__Canon, mscorlib]].SetResult+0x0 [[System.__Canon, mscorlib @ 638]
microsoft_owin_security!Microsoft.Owin.Security.Infrastructure.AuthenticationMiddleware`1+_Invoke_d__5 [[System.__Canon, mscorlib @ 30]
mscorlib!System.Threading.ExecutionContext.RunInternal+0x172 [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 980]
mscorlib!System.Threading.ExecutionContext.Run+0x15 [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 928]
mscorlib!System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner [f:\dd\ndp\clr\src\BCL\system\runtime\compilerservices\AsyncMethodBuilder.cs @ 1070]
system_web!System.Web.Util.SynchronizationHelper.SafeWrapCallback+0x76 [f:\dd\ndp\fx\src\xsp\system\Web\Util\SynchronizationHelper.cs @ 159]
mscorlib!System.Threading.Tasks.Task.Execute+0x47 [f:\dd\ndp\clr\src\BCL\system\threading\Tasks\Task.cs @ 2498]
mscorlib!System.Threading.ExecutionContext.RunInternal+0x172 [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 980]
mscorlib!System.Threading.ExecutionContext.Run+0x15 [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 928]
mscorlib!System.Threading.Tasks.Task.ExecuteWithThreadLocal+0x231 [f:\dd\ndp\clr\src\BCL\system\threading\Tasks\Task.cs @ 2827]
mscorlib!System.Threading.Tasks.Task.ExecuteEntry+0xa1 [f:\dd\ndp\clr\src\BCL\system\threading\Tasks\Task.cs @ 2767]
mscorlib!System.Threading.ThreadPoolWorkQueue.Dispatch+0x156 [f:\dd\ndp\clr\src\BCL\system\threading\threadpool.cs @ 820]
clr!CallDescrWorkerInternal+0x83
clr!CallDescrWorkerWithHandler+0x4e
clr!MethodDescCallSite::CallTargetWorker+0xfa
clr!QueueUserWorkItemManagedCallback+0x2a
clr!ManagedThreadBase_DispatchInner+0x40
clr!ManagedThreadBase_DispatchMiddle+0x6c
clr!ManagedThreadBase_DispatchOuter+0x4c
clr!ManagedThreadBase_DispatchInCorrectAD+0x15
clr!Thread::DoADCallBack+0x26b
clr!ManagedThreadBase_DispatchInner+0x2e57
clr!ManagedThreadBase_DispatchMiddle+0x6c
clr!ManagedThreadBase_DispatchOuter+0x4c
clr!ManagedThreadBase_FullTransitionWithAD+0x2f
clr!ManagedPerAppDomainTPCount::DispatchWorkItem+0xa4
clr!ThreadpoolMgr::ExecuteWorkRequest+0x64
clr!ThreadpoolMgr::WorkerThreadStart+0xf6
clr!Thread::intermediateThreadProc+0x8b
kernel32!BaseThreadInitThunk+0x10 [clientcore\base\win32\client\thread.c @ 75]
ntdll!RtlUserThreadStart+0x2b [minkernel\ntdll\rtlstrt.c @ 1152]

[halter73]

Upgrading package versions seems like a plausible cause for issues. We’ve seen this a lot of issues with the Midrosoft.IdentityModel.* package versions getting misaligned.

I notice that the list of NuGet packages that you’ve included in the issue do not include some transitive dependencies. For example, I see the Microsoft.Owin.Security.OpenIdConnect in the stack trace, but I don’t see the package version for it. Using the latest dotnet SDKs, you can run something like the following to get all the transitive packages:

$ dotnet list package --include-transitive
Project 'OwinOidc' has the following package references
   Top-level Package                                      Requested    Resolved
   > Microsoft.AspNet.Mvc                                 5.2.9        5.2.9
   > Microsoft.AspNet.WebPages                            3.2.9        3.2.9
   > Microsoft.IdentityModel.Logging                      7.0.2        7.0.2
   > Microsoft.IdentityModel.Protocols                    7.0.2        7.0.2
   > Microsoft.IdentityModel.Protocols.OpenIdConnect      7.0.2        7.0.2
   > Microsoft.IdentityModel.Tokens                       7.0.2        7.0.2
   > Microsoft.Owin                                       4.2.2        4.2.2
   > Microsoft.Owin.Host.SystemWeb                        4.2.2        4.2.2
   > Microsoft.Owin.Hosting                               4.2.2        4.2.2
   > Microsoft.Owin.Security                              4.2.2        4.2.2
   > Microsoft.Owin.Security.Cookies                      4.2.2        4.2.2
   > Microsoft.Owin.Security.OpenIdConnect                4.2.2        4.2.2
   > Microsoft.Owin.Security.WsFederation                 4.2.2        4.2.2
   > Newtonsoft.Json                                                   3.5.8
   > Owin                                                 1.0.0        1.0.0
   > System.Buffers                                       4.5.1        4.5.1
   > System.IdentityModel.Tokens.Jwt                      7.0.2        7.0.2
   > System.Management.Automation.dll                     10.0.10586   10.0.10586
   > System.Numerics.Vectors                              4.5.0        4.5.0
   > System.Runtime.CompilerServices.Unsafe               6.0.0        6.0.0

   Transitive Package                                    Resolved
   > Microsoft.AspNet.Razor                              3.2.9
   > Microsoft.IdentityModel.Abstractions                7.0.2
   > Microsoft.IdentityModel.JsonWebTokens               7.0.2
   > Microsoft.IdentityModel.Protocols.WsFederation      5.3.0
   > Microsoft.IdentityModel.Tokens.Saml                 5.3.0
   > Microsoft.IdentityModel.Xmll                        5.3.0

This is just an example from a previous similar issue. I don’t know what feed Microsoft.Identity.ServiceEssentials.Extensions.OwinMiddleware and the like are coming from, so I don’t know what your transitive dependencies are, but mixing 7.0.2 Microsoft.IdentityModel.* packages with other transitive 5.3.0 Microsoft.IdentityModel.* packages caused problems in that case. I suspect you’re running into a similar issue.

Otherwise, it’s hard to diagnose what’s going on with just a stack trace. What’s happening at that stack trace? We normally need an exception type and message at a minimum. If it’s an AV, a dump might be helpful, but as far as I know, the Microsoft.Owin packages referenced in the stack trace don’t have any unsafe logic that could reasonably cause an AV, and they have not been updated in a couple years.

[1]: [Feature Request] Warn when different versions of the IdentityModel NuGet packages are used (#2513)