TypeError: '<' not supported between instances of 'datetime.datetime' and 'float'

[JaimeMagiera]

Using this on OpenShift 4.5, I'm getting the following:

TypeError: '<' not supported between instances of 'datetime.datetime' and 'float'

It's a quick fix, but wanted to give you a heads up in case you didn't see this.

[GrahamDumpleton]

Need more details, such as the full Python traceback and where you are seeing this.

One cause for this error used to be use of bad version of Tornado. Another is bad version of Python REST API client which would fail with newer Kubernetes versions.

The details you provided isn't enough to know where or what is causing it.

[mosuke5]

I encountered same error. Following is full error log in jupyterhub pod.

[E 2020-09-15 06:41:39.498 JupyterHub user:626] Unhandled error starting 05716774-13c9-4de4-91b1-148b0662e7d6's server: '<' not supported between instances of 'datetime.datetime' and 'float'
[I 2020-09-15 06:41:39.508 JupyterHub log:174] 302 GET /hub/spawn/05716774-13c9-4de4-91b1-148b0662e7d6 -> /hub/spawn-pending/05716774-13c9-4de4-91b1-148b0662e7d6 (05716774-13c9-4de4-91b1-148b0662e7d6@::ffff:10.131.0.9) 31.14ms
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<BaseHandler.spawn_single_user() done, defined at /opt/app-root/lib/python3.6/site-packages/jupyterhub/handlers/base.py:697> exception=TypeError("'<' not supported between instances of 'datetime.datetime' and 'float'",)>
Traceback (most recent call last):
  File "/opt/app-root/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 889, in spawn_single_user
    timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future
  File "/opt/app-root/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 807, in finish_user_spawn
    await spawn_future
  File "/opt/app-root/lib/python3.6/site-packages/jupyterhub/user.py", line 642, in spawn
    raise e
  File "/opt/app-root/lib/python3.6/site-packages/jupyterhub/user.py", line 546, in spawn
    url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
  File "/opt/app-root/lib/python3.6/site-packages/tornado/gen.py", line 748, in run
    yielded = self.gen.send(value)
  File "/opt/app-root/lib/python3.6/site-packages/kubespawner/spawner.py", line 1661, in _start
    events = self.events
  File "/opt/app-root/lib/python3.6/site-packages/kubespawner/spawner.py", line 1512, in events
    for event in self.event_reflector.events:
  File "/opt/app-root/lib/python3.6/site-packages/kubespawner/spawner.py", line 73, in events
    key=lambda x: x.last_timestamp if x.last_timestamp is not None else 0.,
TypeError: '<' not supported between instances of 'datetime.datetime' and 'float'
[E 2020-09-15 06:41:39.619 JupyterHub pages:284] Previous spawn for 05716774-13c9-4de4-91b1-148b0662e7d6 failed: '<' not supported between instances of 'datetime.datetime' and 'float'
[E 2020-09-15 06:41:39.620 JupyterHub log:166] {
      "X-Forwarded-For": "180.235.2.22,::ffff:10.131.0.9",
      "Forwarded": "for=180.235.2.22;host=jupyterhub-mosuke5.apps.cluster-eaa3.eaa3.sandbox81.opentlc.com;proto=https;proto-version=\"\"",
      "X-Forwarded-Proto": "https,http",
      "X-Forwarded-Port": "443,80",
      "X-Forwarded-Host": "jupyterhub-mosuke5.apps.cluster-eaa3.eaa3.sandbox81.opentlc.com",
      "Host": "jupyterhub-mosuke5.apps.cluster-eaa3.eaa3.sandbox81.opentlc.com",
      "Cookie": "jupyterhub-hub-login=[secret]; jupyterhub-session-id=[secret]",
      "Accept-Language": "en-US,en;q=0.9,ja;q=0.8",
      "Accept-Encoding": "gzip, deflate, br",
      "Referer": "https://jupyterhub-mosuke5.apps.cluster-eaa3.eaa3.sandbox81.opentlc.com/hub/spawn-pending/05716774-13c9-4de4-91b1-148b0662e7d6",
      "Sec-Fetch-Dest": "document",
      "Sec-Fetch-User": "?1",
      "Sec-Fetch-Mode": "navigate",
      "Sec-Fetch-Site": "same-origin",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36",
      "Upgrade-Insecure-Requests": "1",
      "Connection": "close"
    }
[E 2020-09-15 06:41:39.620 JupyterHub log:174] 500 GET /hub/spawn-pending/05716774-13c9-4de4-91b1-148b0662e7d6 (05716774-13c9-4de4-91b1-148b0662e7d6@::ffff:10.131.0.9) 4.88ms

And this is a jupyterhub screen shot.

[cloudbustinguk]

jupyterhub/kubespawner#354 (comment)

[velmurugan-r]

Hi Any update on this error, i am getting the same error in openshift 4.5

[kazutnb]

I'm also having trouble with this problem. How can I deal with it?
OpenShift 4.5.35 (Kubernates v1.18.3)

[skvaditya]

Hi,
I am using this on openshift 4 and I am getting trouble while launching python 3 notebook. It gives error as:
A connection to the notebook server could not be established. The notebook will continue trying to reconnect. Check your network connection or notebook server configuration.
It worked fine with Openshift 3. Any guide why the issue is only in openshift 4 but not in 3 and how to fix it for openshift 4 as well.

[neumannrf]

@JaimeMagiera @mosuke5 We are also facing this error in our OpenShift 4.6 cluster. Are you still having this problem?

In the opendatahub-io/jupyterhub-quickstart fork there seems to have been a couple of new relases, but nothing that indicates that the compatibility issues with OpenShift 4 are gone.

[vemonet]

We are also facing this issue with OpenShift 4.6

Is this repository about JupyterHub still active? There has been no commit in more than a year

More in general, it would be interesting to know more about where to find informations about deploying such a service on OpenShift. Because those services (like JupyterHub) are well documented and working on Kubernetes, but usually not documented and facing multiple deep permissions/implementations issues for RedHat OpenShift.

According to @GrahamDumpleton in this post jupyter-on-openshift/jupyter-notebooks#27 the jupyter-on-openshift repos are deprecated (it is not clearly stated in the readme, so it can be quite confusing for potential users)

Note that:

• OpenDataHub is not an option because it requires to install the operator globally (and it does not gives much details for JupyterHub configuration anyway)
• Trying the helm charts that have been developed for Kubernetes might be an option, but those charts are not built to work on OpenShift so it will require to dive into those charts, understanding them and fixing them. Unfortunately even diving into those charts is not always possible!

For example, this documentation for JupyterHub deployment on kubernetes: https://zero-to-jupyterhub.readthedocs.io refers to the official JupyterHub Helm charts GitHub repo https://github.com/jupyterhub/helm-chart where the charts and templates are not available, and it is not clear where we can find them, so the charts cannot easily be tweaked to work OpenShift (not sure what is their motivation for hiding the Helm charts)

It is surprising that the deployment JupyterHub, a quite common data science tool, is so convoluted and poorly documented on RedHat OpenShift, any idea where/how we can find a stable deployment for JupyterHub on OpenShift?

[GrahamDumpleton]

No, these repositories are no longer being actively maintained. People from Red Hat are aware of this and understand it is a problem affecting a number of OpenShift users but although I am happy for them to take the repositories over I have not been approached to assign the repositories to them.

The only officially supported solutions from Red Hat that I know of are still just opendatahub.io and radanalytics.io.

[vemonet]

Thanks @GrahamDumpleton for the feedback!

This is quite concerning for everyone using RedHat products though

The point of Kubernetes is to be able to deploy complex softwares on different systems (self hosted, GCP, AWS), with guarantee of stability. But with OpenShift, RedHat broke this major Kubernetes feature, and OpenShift users cannot re-use Kubernetes packages built by communities, they need to build from scratch the deployment for their OpenShift (which is more complex than kubernetes, and will lead to more security flaws)

Additionally RedHat seems to be maintaining multiple version of JupyterHub/Jupyter: one for opendatahub and one for radanalytics. Wouldn't it possible to factorize efforts by using packaging systems properly?

It seems to be much more efficient and logical to have 1 JupyterHub Helm chart working for OpenShift, and re-use this one Helm chart in OpenDataHub and radanalytics, instead of developing adhoc solution everytime.

[Itohan97]

Hello,
please i keep having this error ''not supported between instances of 'float' and 'complex'''

[Itohan97]

How do i fix this ?