Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unorderable types: tuple() < NoneType() (500 INTERNAL SERVER ERROR) #244

Open
bert2002 opened this issue Jun 6, 2019 · 0 comments
Open

Comments

@bert2002
Copy link

bert2002 commented Jun 6, 2019

Hi,
running into an weird problem. I am collecting data with collectd and push it to graphite-api which has graphite-carbon as backend. Works wonderful for all metrics, except for this one.

Jun 06 15:45:31 monitoring-1 gunicorn3[725]: {"reader": "whisper", "start": 1559806831, "end": 1559807131, "path": "/var/lib/graphite/whisper/collectd/test-node-server-1/processes/ps_state-running.wsp", "metric_path": "collectd.test-node-server-1.processes.ps_state-running", "event": "fetch"}
Jun 06 15:45:31 monitoring-1 gunicorn3[725]: {"event": "Exception on /render [POST]", "exception": "Traceback (most recent call last):\n  File \"/usr/lib/python3/dist-packages/flask/app.py\", line 1982, in wsgi_app\n    response = self.full_dispatch_request()\n  File \"/usr/lib/python3/dist-packages/flask/app.py\", line 1614, in full_dispatch_request\n    rv = self.handle_user_exception(e)\n  File \"/usr/lib/python3/dist-packages/flask/app.py\", line 1517, in handle_user_exception\n    reraise(exc_type, exc_value, tb)\n  File \"/usr/lib/python3/dist-packages/flask/_compat.py\", line 33, in reraise\n    raise value\n  File \"/usr/lib/python3/dist-packages/flask/app.py\", line 1612, in full_dispatch_request\n    rv = self.dispatch_request()\n  File \"/usr/lib/python3/dist-packages/flask/app.py\", line 1598, in dispatch_request\n    return self.view_functions[rule.endpoint](**req.view_args)\n  File \"/usr/lib/python3/dist-packages/graphite_api/app.py\", line 375, in render\n    data_store = fetchData(context, paths)\n  File \"/usr/lib/python3/dist-packages/graphite_api/render/datalib.py\", line 167, in fetchData\n    (node, node.fetch(startTime, endTime)) for node in single_nodes]\n  File \"/usr/lib/python3/dist-packages/graphite_api/render/datalib.py\", line 167, in <listcomp>\n    (node, node.fetch(startTime, endTime)) for node in single_nodes]\n  File \"/usr/lib/python3/dist-packages/graphite_api/node.py\", line 28, in fetch\n    return self.reader.fetch(startTime, endTime)\n  File \"/usr/lib/python3/dist-packages/graphite_api/finders/whisper.py\", line 121, in fetch\n    cached_datapoints = self.carbonlink.query(self.real_metric_path)\n  File \"/usr/lib/python3/dist-packages/graphite_api/carbonlink.py\", line 199, in query\n    results = self.send_request(request)\n  File \"/usr/lib/python3/dist-packages/graphite_api/carbonlink.py\", line 230, in send_request\n    host = self.select_host(metric)\n  File \"/usr/lib/python3/dist-packages/graphite_api/carbonlink.py\", line 160, in select_host\n    for node in self.hash_ring.get_nodes(key):\n  File \"/usr/lib/
Jun 06 15:45:31 monitoring-1 gunicorn3[725]: python3/dist-packages/graphite_api/carbonlink.py\", line 108, in get_nodes\n    index = bisect.bisect_left(self
[ps_state-running.dump.txt](https://github.com/brutasse/graphite-api/files/3260621/ps_state-running.dump.txt)
.ring, search_entry) % self.ring_len\nTypeError: unorderable types: tuple() < NoneType()"}

I already deleted all datasets (stopped all processes before), but the problem seems to be persistent. I attached a dump of the data (from whisper-fetch). I don't see anything wrong and wondering what is happening. because I have 10s of other server which the same configuration and dont run into this problem. Also all other metrics in processes are working as expected. Any idea?

OS: Debian 9.9
graphite-api: 1.1.3-2+deb9u1
graphite-carbon: 0.9.15-1
grafana: 6.2.2

Many thanks,
bert

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant