🐛 [BUG]: Unable to disable JupyterHub behind Twitcher

[tlvu]

Summary

Setting JUPYTERHUB_AUTHENTICATOR_AUTHORIZATION_URL="" in env.local is unable to disable JupyterHub behind Twitcher. It breaks the spawn of the personal JupyterLab server completely.

Details

Error in docker logs jupyterhub when JUPYTERHUB_AUTHENTICATOR_AUTHORIZATION_URL="" is set in env.local: Error starting server lvu: Invalid URL '': No scheme supplied. Perhaps you meant https://?

Full error log:

[I 2024-04-04 16:19:18.682 JupyterHub log:191] 302 GET /jupyter/hub/ -> /jupyter/hub/spawn ([email protected]) 89.01ms
[D 2024-04-04 16:19:18.843 JupyterHub scopes:877] Checking access to /jupyter/hub/spawn via scope servers
[D 2024-04-04 16:19:18.843 JupyterHub scopes:690] Argument-based access to /jupyter/hub/spawn via servers
[D 2024-04-04 16:19:18.845 JupyterHub pages:209] Serving options form for lvu
[I 2024-04-04 16:19:18.875 JupyterHub log:191] 200 GET /jupyter/hub/spawn ([email protected]) 41.23ms
[D 2024-04-04 16:19:21.646 JupyterHub scopes:877] Checking access to /jupyter/hub/spawn via scope servers
[D 2024-04-04 16:19:21.647 JupyterHub scopes:690] Argument-based access to /jupyter/hub/spawn via servers
[D 2024-04-04 16:19:21.647 JupyterHub pages:257] Triggering spawn with supplied form options for lvu
[D 2024-04-04 16:19:21.647 JupyterHub base:344] Refreshing auth for lvu
[E 2024-04-04 16:19:21.650 JupyterHub pages:313] Error starting server lvu: Invalid URL '': No scheme supplied. Perhaps you meant https://?
    Traceback (most recent call last):
    NoneType: None
    
[E 2024-04-04 16:19:21.650 JupyterHub pages:266] Failed to spawn single-user server with form
    Traceback (most recent call last):
      File "/usr/local/lib/python3.10/dist-packages/jupyterhub/handlers/pages.py", line 262, in _post
        return await self._wrap_spawn_single_user(
      File "/usr/local/lib/python3.10/dist-packages/jupyterhub/handlers/pages.py", line 317, in _wrap_spawn_single_user
        raise web.HTTPError(
    tornado.web.HTTPError: HTTP 500: Internal Server Error (Unhandled error starting server lvu)
    
[I 2024-04-04 16:19:21.654 JupyterHub log:191] 200 POST /jupyter/hub/spawn?_xsrf=[secret] ([email protected]) 10.75ms
[I 2024-04-04 16:20:05.381 JupyterHub log:191] 200 GET /jupyter/hub/login (@172.18.0.1) 7.34ms

Full log when it works (that switch is not set), starting from the exact same line GET /jupyter/hub/ -> /jupyter/hub/spawn:

[I 2024-04-04 16:24:33.887 JupyterHub log:191] 302 GET /jupyter/hub/ -> /jupyter/hub/spawn ([email protected]) 92.05ms
[D 2024-04-04 16:24:33.959 JupyterHub scopes:877] Checking access to /jupyter/hub/spawn via scope servers
[D 2024-04-04 16:24:33.959 JupyterHub scopes:690] Argument-based access to /jupyter/hub/spawn via servers
[D 2024-04-04 16:24:33.960 JupyterHub pages:209] Serving options form for lvu
[I 2024-04-04 16:24:33.974 JupyterHub log:191] 200 GET /jupyter/hub/spawn ([email protected]) 17.31ms
[D 2024-04-04 16:24:36.638 JupyterHub scopes:877] Checking access to /jupyter/hub/spawn via scope servers
[D 2024-04-04 16:24:36.638 JupyterHub scopes:690] Argument-based access to /jupyter/hub/spawn via servers
[D 2024-04-04 16:24:36.638 JupyterHub pages:257] Triggering spawn with supplied form options for lvu
[D 2024-04-04 16:24:36.639 JupyterHub base:344] Refreshing auth for lvu
[D 2024-04-04 16:24:36.661 JupyterHub base:961] Initiating spawn for lvu
[D 2024-04-04 16:24:36.661 JupyterHub base:965] 0/100 concurrent spawns
[D 2024-04-04 16:24:36.661 JupyterHub base:970] 0 active servers
[I 2024-04-04 16:24:37.012 JupyterHub provider:659] Creating oauth client jupyterhub-user-lvu
[D 2024-04-04 16:24:37.450 JupyterHub user:794] Calling Spawner.start for lvu
[I 2024-04-04 16:24:37.451 JupyterHub dockerspawner:1262] pulling pavics/workflow-tests:py39-230601-1-update240116
[I 2024-04-04 16:24:37.641 JupyterHub log:191] 302 POST /jupyter/hub/spawn?_xsrf=[secret] -> /jupyter/hub/spawn-pending/lvu?_xsrf=[secret] ([email protected]) 1004.86ms
[D 2024-04-04 16:24:37.786 JupyterHub scopes:877] Checking access to /jupyter/hub/spawn-pending/lvu via scope servers
[D 2024-04-04 16:24:37.787 JupyterHub scopes:690] Argument-based access to /jupyter/hub/spawn-pending/lvu via servers
[I 2024-04-04 16:24:37.788 JupyterHub pages:398] lvu is pending spawn
[I 2024-04-04 16:24:37.805 JupyterHub log:191] 200 GET /jupyter/hub/spawn-pending/lvu?_xsrf=[secret] ([email protected]) 23.01ms
[D 2024-04-04 16:24:37.949 JupyterHub dockerspawner:1027] Getting container 'jupyter-lvu'
[I 2024-04-04 16:24:37.952 JupyterHub dockerspawner:1033] Container 'jupyter-lvu' is gone
[D 2024-04-04 16:24:37.959 JupyterHub dockerspawner:1205] Starting host with config: {REMOVED}
[D 2024-04-04 16:24:38.888 JupyterHub scopes:877] Checking access to /jupyter/hub/api/users/lvu/server/progress via scope read:servers
[D 2024-04-04 16:24:38.888 JupyterHub scopes:690] Argument-based access to /jupyter/hub/api/users/lvu/server/progress via read:servers
[I 2024-04-04 16:24:40.576 JupyterHub dockerspawner:1311] Created container jupyter-lvu (id: f68bcca) from image pavics/workflow-tests:py39-230601-1-update240116
[I 2024-04-04 16:24:40.576 JupyterHub dockerspawner:1335] Starting container jupyter-lvu (id: f68bcca)
[D 2024-04-04 16:24:42.706 JupyterHub spawner:1384] Polling subprocess every 30s
[D 2024-04-04 16:24:42.707 JupyterHub dockerspawner:972] Persisting state for lvu: container name=jupyter-lvu, id=f68bcca87532c4d696a28d476337deee7bcde561062012900fe375b1a65c6807

To Reproduce

Steps to reproduce the behavior:

1. birdhouse-deploy version 2.1.2 (but should be broken since 1.36.0 when JupyterHub was put behind Twitcher)
2. set export JUPYTERHUB_AUTHENTICATOR_AUTHORIZATION_URL="" in env.local
3. login to JupyterHub will work
4. but spawning the personal server will fail with the log above server-side

Environment

Information	Value
Server/Platform URL
Version Tag/Commit	2.1.2
Related issues/PR	#358 (tag 1.36.0)
Related components	jupyterhub, twitcher, magpie
Custom configuration

Concerned Organizations

@mishaschwartz @fmigneault

[tlvu]

@mishaschwartz I merged the pending Ouranosinc/jupyterhub#29 so you are clear to do any fixes in that repo.

[mishaschwartz]

@tlvu

Is JUPYTERHUB_CRYPT_KEY set?
If so, then the MagpieAuthenticator.refresh_pre_spawn is set to True and the authorization_url is required.

[mishaschwartz]

@tlvu Ouranosinc/jupyterhub#30 should cover this edge case

[tlvu]

Is JUPYTERHUB_CRYPT_KEY set?
If so, then the MagpieAuthenticator.refresh_pre_spawn is set to True and the authorization_url is required.

@mishaschwartz Oh indeed I also had JUPYTERHUB_CRYPT_KEY set. I was trying out all the new options, didn't know they were mutually exclusive.

Confirmed that removing JUPYTERHUB_CRYPT_KEY while having JUPYTERHUB_AUTHENTICATOR_AUTHORIZATION_URL="", I was able to login and spawn the server fine.

[tlvu]

Ouranosinc/jupyterhub#30 should cover this edge case

So I've read the PR, so the problem is not due to both JUPYTERHUB_CRYPT_KEY and JUPYTERHUB_AUTHENTICATOR_AUTHORIZATION_URL="" are mutually exclusive but due to a previous auth_state left behind?

So both of these options can be set at the same time and it should work?

[mishaschwartz]

So both of these options can be set at the same time and it should work?

Yeah exactly

[tlvu]

So both of these options can be set at the same time and it should work?

Yeah exactly

Just to be clear. Your fix is to make these 2 options not mutually exclusive. Otherwise, in the current state, they are mutually exclusive? Just so I fully understand the problem.

[mishaschwartz]

@tlvu

As of Ouranosinc/jupyterhub#30 the behaviour is as follows:

• JUPYTERHUB_AUTHENTICATOR_AUTHORIZATION_URL is set:
  • only users that have permission to access jupyterhub (as defined by magpie service permissions) can log in to jupyterhub
  • if JUPYTERHUB_CRYPT_KEY is set, jupyterhub will also:
    • check if the user still has permission when they spawn a server
    • check if the user still has permission at most every JUPYTERHUB_AUTHENTICATOR_REFRESH_AGE seconds
    • store the login cookies for the user in the jupyterhub database. A user can access these in their jupyterlab session to make accessing other resources that are protected behind twitcher easier.
• JUPYTERHUB_AUTHENTICATOR_AUTHORIZATION_URL is not set:
  • any user that exists in magpie is allowed to log in to jupyterhub
  • JUPYTERHUB_CRYPT_KEY can be set or not and will have no effect