OIDC suddenly stopped working for one user, unclear why

[mikehearn]

A couple of days ago OIDC stopped working for one of my users, although I can still log in fine. We've tried clearing cookies, refreshing, forcing logouts, wiping the on-disk cache and many other things, all to no avail. We use JetBrains Hub as the OIDC provider and in the logs we can see this:

[Thu Jan 26 15:26:35.194749 2023] [auth_openidc:error] [pid 31680:tid 140402740946688] [client 77.56.44.1:51145] oidc_util_http_call: curl_easy_perform() failed on: https://hub.hq.hydraulic.software/hub/api/rest/oauth2/token (Failed writing body (0 != 1291)), referer: https://www.hq.hydraulic.software/
[Thu Jan 26 15:26:35.195368 2023] [auth_openidc:warn] [pid 31680:tid 140402740946688] [client 77.56.44.1:51145] oidc_proto_token_endpoint_request: error when calling the token endpoint (https://hub.hq.hydraulic.software/hub/api/rest/oauth2/token), referer: https://www.hq.hydraulic.software/
[Thu Jan 26 15:26:35.195378 2023] [auth_openidc:error] [pid 31680:tid 140402740946688] [client 77.56.44.1:51145] oidc_proto_resolve_code_and_validate_response: failed to resolve the code, referer: https://www.hq.hydraulic.software/
[Thu Jan 26 15:26:35.519687 2023] [auth_openidc:error] [pid 31680:tid 140401784637184] [client 77.56.44.1:51145] oidc_curl_write: HTTP response larger than maximum allowed size: current size=1048292, additional size=1291, max=1048576, referer: https://www.hq.hydraulic.software/

So, Hub appears to be returning a token that's larger than the max size accepted by the plugin (which is hard coded). I suspect that what's being returned maybe isn't a token at all but some HTML or other junk, as otherwise it's hard to explain the size, but I don't know how to see what's going on here. Does anyone know how to debug this?

I should note that the Hub logs look good - as far as it's concerned, this user is logging in just fine. Plus, other apps that use this SSO are still working for him. It's just Apache that has suddenly got unhappy.

Also, we're using version 2.4.1-1 which is the one that came with the old-ish Ubuntu LTS we're using. Is it maybe too old?

[zandbelt]

either the token or the error returned from the token endpoint is larger than 1Mb; in both cases I think there's something going on that is not right for that user on the OP side; it is probably worth to look at the logs there; I could provide a custom bulid that allows for >1 Mb responses but I'm not sure that that is the way to go (other than for debugging purposes)

[mikehearn]

Wow, thanks! This has to be one of the best open source support experiences I had for a while! If I can donate in some way let me know, or send you a pizza voucher or something.

The deb you sent fixes it, of course. And the logging reveals the mystery. What broke it was ... 🥁 ... he added a profile picture to his Hub account, and JetBrains Hub includes a base64 encoded PNG of the profile picture in the token response. But the actual picture he uploaded is a photo, so base64 encoded PNG is the worst possible way to encode such a thing and that blows the response up past 1mb.

I don't really know much about OIDC. I guess returning a profile picture is a reasonable thing for an SSO system to do. I can see why apps would want to have that. Including it inline in the token seems a bit more questionable but that's what it does. I'd suggest the higher limit should therefore be kept.

[zandbelt]

thanks for reporting back; in OIDC one would return the profile picture from the user info endpoint, not include it in the token

[mikehearn]

I can file it as a bug against Hub, but I wonder if their other apps expect it to work this way. The bigger limit seems safe at any rate, 10mb is not excessive for a transient buffer these days.

[zandbelt]

the issue is not with the transport but the fact that it would end up in the session cache and potentially the session cookie

[mikehearn]

I see, OK. I'll talk to JetBrains about it then and point them at this thread.

[zandbelt]

@mikehearn after thinking this over I still believe that the ID token - since it represents an authentication event - should not contain these claims, but on the other hand:

1. this situation needs to be handled more gracefully by mod_auth_openidc
2. the HTTP response size limit also holds for the userinfo endpoint
3. large claims from the userinfo endpoint would result in the same caching challenges
4. there's an existing OIDCBlackListedClaims primitive that can be used to filter out unwanted/unfit claims from both sources

So I've committed a patch that allows handling this situation in a better way 2865154 and uploaded a build for you to test/confirm here: https://www.mod-auth-openidc.org/downloads/libapache2-mod-auth-openidc_2.4.12.4rc1-1.focal_amd64.deb

In summary: all claims should be processed, and if they present a problem later on in the cache/cookie/storage, they can be black- or white-listed by https://github.com/OpenIDC/mod_auth_openidc/blob/v2.4.12.3/auth_openidc.conf#L955-L963

[mikehearn]

Great. I upgraded and it still works.

Also, I filed a bug against Hub and they are implementing a fix on their side by shipping a profile image thumbnail instead of the actual profile image: https://youtrack.jetbrains.com/issue/HUB-11372