Can't connect to kernel: stays pending as "connecting"

[rsignell-usgs]

Our ESIP Nebari deployment had been working fine, but this morning when I try to start a notebook it can't connect to any of the kernels. It just stays pending with the screenshot here:

So of course I can't run any cells.

I took a look at k9s and all the pods seem fine.

What should I do next?

[rsignell-usgs]

And now it's working again. Grrrrr.... I tried everything 2 hours ago, logging out, restarting my server, trying different kernels, nothing would connect. But now it's working fine .

Any idea what might have been going on?
And perhaps how to avoid or fix the next time it happens?

[iameskild]

Hi, @rsignell-usgs I'm sorry to hear you've been experiencing these issues, it definitely sounds frustrating! Unfortunately without logs, it's difficult to diagnose the problem. Are there any logs from the hub pod or perhaps the user's jupyterlab pod?

[rsignell-usgs]

The hub is currently spewing out tons of message like:

[I 2023-04-03 17:28:11.408 JupyterHub log:189] 302 GET /user/rsignell-usgs/api/kernels/577bdd17-9577-4435-b2eb-b9e80 │
[W 2023-04-03 17:28:11.495 JupyterHub base:1349] Failing suspected API request to not-running server: /hub/user/rsig │
[E 2023-04-03 17:28:11.495 JupyterHub log:189] 503 GET /hub/user/rsignell-usgs/api/kernels/b5c10fd3-90b6-45be-818a-1 │
[W 2023-04-03 17:28:11.499 JupyterHub base:1349] Failing suspected API request to not-running server: /hub/user/rsig ││ 
[E 2023-04-03 17:28:11.499 JupyterHub log:189] 503 GET /hub/user/rsignell-usgs/api/kernels/577bdd17-9577-4435-b2eb-b ││
[I 2023-04-03 17:28:13.422 JupyterHub log:189] 302 GET /user/rsignell-usgs/api/kernels/72e31cb0-0de2-4c09-ba27-62b61 ││ 
[W 2023-04-03 17:28:13.512 JupyterHub base:1349] Failing suspected API request to not-running server: /hub/user/rsig ││ 
[E 2023-04-03 17:28:13.513 JupyterHub log:189] 503 GET /hub/user/rsignell-usgs/api/kernels/72e31cb0-0de2-4c09-ba27-6 ││ 
[I 2023-04-03 17:28:14.257 JupyterHub log:189] 302 GET /user/dwade0708/api/kernels/edc8ee75-8660-4b0f-92ae-f97572fd2 ││ 
[W 2023-04-03 17:28:14.302 JupyterHub base:1349] Failing suspected API request to not-running server: /hub/user/dwad ││ 
[E 2023-04-03 17:28:14.302 JupyterHub log:189] 503 GET /hub/user/dwade0708/api/kernels/edc8ee75-8660-4b0f-92ae-f9757 ││ 
[I 2023-04-03 17:28:15.447 JupyterHub log:189] 302 GET /user/rsignell-usgs/api/kernels/c044a77e-fce8-4c63-b058-48627 ││ 
[W 2023-04-03 17:28:15.541 JupyterHub base:1349] Failing suspected API request to not-running server: /hub/user/rsig ││ 
[E 2023-04-03 17:28:15.542 JupyterHub log:189] 503 GET /hub/user/rsignell-usgs/api/kernels/c044a77e-fce8-4c63-b058-4 ││ 
[I 2023-04-03 17:28:16.239 JupyterHub log:189] 302 GET /user/salme146/api/kernels/5ddddec1-9da0-4478-8500-912c46e9b2 ││ 
[W 2023-04-03 17:28:16.431 JupyterHub base:1349] Failing suspected API request to not-running server: /hub/user/salm ││ 
[E 2023-04-03 17:28:16.431 JupyterHub log:189] 503 GET /hub/user/salme146/api/kernels/5ddddec1-9da0-4478-8500-912c46 │

but those happen whenever people just leave their Nebari session open without activity, right?

[rsignell-usgs]

My user pod has messages in the logs like this:

 notebook CONDA_PREFIX=/home/conda/users/envs/users-pangeo                                                            
││ notebook [I 2023-04-03 12:32:47.237 SingleUserNotebookApp kernelmanager:222] Starting buffering for cfacf468-758a-4d 
││ notebook [W 2023-04-03 12:32:48.870 SingleUserNotebookApp handlers:612] Kernel does not exist: 18b0e941-d09f-4801-80 
││ notebook [W 2023-04-03 12:32:49.286 SingleUserNotebookApp handlers:612] Kernel does not exist: 72e31cb0-0de2-4c09-ba 
││ notebook [W 2023-04-03 12:32:49.441 SingleUserNotebookApp handlers:612] Kernel does not exist: b5c10fd3-90b6-45be-81 
││ notebook [W 2023-04-03 12:32:50.292 SingleUserNotebookApp handlers:612] Kernel does not exist: 7564241b-bf64-4578-99 
││ notebook [W 2023-04-03 12:32:52.855 SingleUserNotebookApp handlers:612] Kernel does not exist: 577bdd17-9577-4435-b2
 ││ notebook [W 2023-04-03 12:32:53.485 SingleUserNotebookApp handlers:612] Kernel does not exist: 0fa1d3f1-8aa2-47a3-9a 
││ notebook [W 2023-04-03 12:32:54.284 SingleUserNotebookApp handlers:612] Kernel does not exist: c044a77e-fce8-4c63-b0 
││ notebook [W 2023-04-03 12:32:55.337 SingleUserNotebookApp handlers:612] Kernel does not exist: 72e31cb0-0de2-4c09-ba 
││ notebook [W 2023-04-03 12:32:56.765 SingleUserNotebookApp handlers:612] Kernel does not exist: 7564241b-bf64-4578-99 
││ notebook [W 2023-04-03 12:32:58.775 SingleUserNotebookApp handlers:612] Kernel does not exist: 18b0e941-d09f-4801-80 ││

[iameskild]

It's hard to say but it looks like the conda-store environments are not available. If they were deleted, those kernels would be inaccessible from the user's notebook.

[jkellndorfer]

So, I ran in to trouble today along those lines! I worked on a notebook and wanted to switch to a larger server. After shutting down the server I used and firing up a larger server, the kernel stayed in "connecting" status. Tried all the things Rich tried, shutting down, restarting, logging out etc. trying different servers ... nada. The Kernel stayed in "connecting' and I never got my notebook to run again, thus far....

[rsignell-usgs]

@iameskild we continue to have these problems. Eventually we are able to start working, but it takes many minutes, apparently working through all these kernel-does-not-exist errors, as you can see from this screen grab. Restarting the hub did not seem to fix the problem. Do you or @costrouc have other ideas?