Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

timeout reached while waiting for fetchSockets response #234

Closed
jimaek opened this issue Nov 16, 2022 · 4 comments
Closed

timeout reached while waiting for fetchSockets response #234

jimaek opened this issue Nov 16, 2022 · 4 comments
Assignees
@MartinKolarik

Comments

@jimaek
Copy link
Member

jimaek commented Nov 16, 2022 

[2022-11-16 15:50:11] [ERROR] [19] [error-handler-mw] timeout reached while waiting for fetchSockets response
  stack -> Error: timeout reached while waiting for fetchSockets response
    at Timeout.<anonymous> (/app/node_modules/@socket.io/redis-adapter/dist/index.js:712:28)
    at /app/node_modules/newrelic/lib/shim/shim.js:1313:22
    at LegacyContextManager.runInContext (/app/node_modules/newrelic/lib/context-manager/legacy-context-manager.js:59:23)
    at Shim.applySegment (/app/node_modules/newrelic/lib/shim/shim.js:1303:25)
    at Timeout.wrappedCallback [as _onTimeout] (/app/node_modules/newrelic/lib/shim/shim.js:1187:21)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)
[2022-11-16 15:50:11] [ERROR] [19] [error-handler-mw] timeout reached while waiting for fetchSockets response
  stack -> Error: timeout reached while waiting for fetchSockets response
    at Timeout.<anonymous> (/app/node_modules/@socket.io/redis-adapter/dist/index.js:712:28)
    at /app/node_modules/newrelic/lib/shim/shim.js:1313:22
    at LegacyContextManager.runInContext (/app/node_modules/newrelic/lib/context-manager/legacy-context-manager.js:59:23)
    at Shim.applySegment (/app/node_modules/newrelic/lib/shim/shim.js:1303:25)
    at Timeout.wrappedCallback [as _onTimeout] (/app/node_modules/newrelic/lib/shim/shim.js:1187:21)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)
[2022-11-16 15:50:12] [DEBUG] [19] [gateway] Probe disconnected. (reason: transport close) [0KGsCTGQlHrJPBVbAAOr][141.164.47.106]
[2022-11-16 15:50:12] [DEBUG] [19] [gateway] Probe disconnected. (reason: transport close) [rYaCgNnyHlcB65KOAAMd][66.90.89.43]
[2022-11-16 15:50:12] [DEBUG] [19] [gateway] Probe disconnected. (reason: transport close) [Z0gBRaecbdm89F6tAANO][192.74.254.48]
[2022-11-16 15:50:14] [ERROR] [19] [error-handler-mw] timeout reached while waiting for fetchSockets response
  stack -> Error: timeout reached while waiting for fetchSockets response
    at Timeout.<anonymous> (/app/node_modules/@socket.io/redis-adapter/dist/index.js:712:28)
    at /app/node_modules/newrelic/lib/shim/shim.js:1313:22
    at LegacyContextManager.runInContext (/app/node_modules/newrelic/lib/context-manager/legacy-context-manager.js:59:23)
    at Shim.applySegment (/app/node_modules/newrelic/lib/shim/shim.js:1303:25)
    at Timeout.wrappedCallback [as _onTimeout] (/app/node_modules/newrelic/lib/shim/shim.js:1187:21)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)
[2022-11-16 15:50:14] [ERROR] [19] [error-handler-mw] timeout reached while waiting for fetchSockets response
  stack -> Error: timeout reached while waiting for fetchSockets response
    at Timeout.<anonymous> (/app/node_modules/@socket.io/redis-adapter/dist/index.js:712:28)
    at /app/node_modules/newrelic/lib/shim/shim.js:1313:22
    at LegacyContextManager.runInContext (/app/node_modules/newrelic/lib/context-manager/legacy-context-manager.js:59:23)
    at Shim.applySegment (/app/node_modules/newrelic/lib/shim/shim.js:1303:25)
    at Timeout.wrappedCallback [as _onTimeout] (/app/node_modules/newrelic/lib/shim/shim.js:1187:21)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)
[2022-11-16 15:50:14] [ERROR] [19] [error-handler-mw] timeout reached while waiting for fetchSockets response
  stack -> Error: timeout reached while waiting for fetchSockets response
    at Timeout.<anonymous> (/app/node_modules/@socket.io/redis-adapter/dist/index.js:712:28)
    at /app/node_modules/newrelic/lib/shim/shim.js:1313:22
    at LegacyContextManager.runInContext (/app/node_modules/newrelic/lib/context-manager/legacy-context-manager.js:59:23)
    at Shim.applySegment (/app/node_modules/newrelic/lib/shim/shim.js:1303:25)
    at Timeout.wrappedCallback [as _onTimeout] (/app/node_modules/newrelic/lib/shim/shim.js:1187:21)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)
[2022-11-16 15:50:14] [ERROR] [19] [error-handler-mw] timeout reached while waiting for fetchSockets response
  stack -> Error: timeout reached while waiting for fetchSockets response
    at Timeout.<anonymous> (/app/node_modules/@socket.io/redis-adapter/dist/index.js:712:28)
    at /app/node_modules/newrelic/lib/shim/shim.js:1313:22
    at LegacyContextManager.runInContext (/app/node_modules/newrelic/lib/context-manager/legacy-context-manager.js:59:23)
    at Shim.applySegment (/app/node_modules/newrelic/lib/shim/shim.js:1303:25)
    at Timeout.wrappedCallback [as _onTimeout] (/app/node_modules/newrelic/lib/shim/shim.js:1187:21)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)
[2022-11-16 15:50:14] [ERROR] [19] [error-handler-mw] timeout reached while waiting for fetchSockets response
  stack -> Error: timeout reached while waiting for fetchSockets response
    at Timeout.<anonymous> (/app/node_modules/@socket.io/redis-adapter/dist/index.js:712:28)
    at /app/node_modules/newrelic/lib/shim/shim.js:1313:22
    at LegacyContextManager.runInContext (/app/node_modules/newrelic/lib/context-manager/legacy-context-manager.js:59:23)
    at Shim.applySegment (/app/node_modules/newrelic/lib/shim/shim.js:1303:25)
    at Timeout.wrappedCallback [as _onTimeout] (/app/node_modules/newrelic/lib/shim/shim.js:1187:21)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)
[2022-11-16 15:50:14] [DEBUG] [19] [gateway] Probe disconnected. (reason: transport error) [3WxT1-Zfim6sWvOwAABq]
[2022-11-16 15:50:14] [DEBUG] [19] [gateway] Probe disconnected. (reason: transport error) [nhMrmuCvsSozCIPiAACe]
[2022-11-16 15:50:14] [DEBUG] [19] [gateway] Probe disconnected. (reason: transport error) [cbx1nEiHokvdIYCuAAEI]
[2022-11-16 15:50:14] [DEBUG] [19] [gateway] Probe disconnected. (reason: transport error) [ZfKU37ou79sQwQUtAACT]
[2022-11-16 15:50:14] [DEBUG] [19] [gateway] Probe disconnected. (reason: transport error) [ZQOTKgq3S6T7mGktAAHr]
[2022-11-16 15:50:18] [ERROR] [19] [error-handler-mw] timeout reached while waiting for fetchSockets response
  stack -> Error: timeout reached while waiting for fetchSockets response
    at Timeout.<anonymous> (/app/node_modules/@socket.io/redis-adapter/dist/index.js:712:28)
    at /app/node_modules/newrelic/lib/shim/shim.js:1313:22
    at LegacyContextManager.runInContext (/app/node_modules/newrelic/lib/context-manager/legacy-context-manager.js:59:23)
    at Shim.applySegment (/app/node_modules/newrelic/lib/shim/shim.js:1303:25)
    at Timeout.wrappedCallback [as _onTimeout] (/app/node_modules/newrelic/lib/shim/shim.js:1187:21)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)
[2022-11-16 15:50:18] [ERROR] [19] [error-handler-mw] timeout reached while waiting for fetchSockets response
  stack -> Error: timeout reached while waiting for fetchSockets response
    at Timeout.<anonymous> (/app/node_modules/@socket.io/redis-adapter/dist/index.js:712:28)
    at /app/node_modules/newrelic/lib/shim/shim.js:1313:22
    at LegacyContextManager.runInContext (/app/node_modules/newrelic/lib/context-manager/legacy-context-manager.js:59:23)
    at Shim.applySegment (/app/node_modules/newrelic/lib/shim/shim.js:1303:25)
    at Timeout.wrappedCallback [as _onTimeout] (/app/node_modules/newrelic/lib/shim/shim.js:1187:21)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)
[2022-11-16 15:50:18] [ERROR] [19] [error-handler-mw] timeout reached while waiting for fetchSockets response
  stack -> Error: timeout reached while waiting for fetchSockets response
    at Timeout.<anonymous> (/app/node_modules/@socket.io/redis-adapter/dist/index.js:712:28)
    at /app/node_modules/newrelic/lib/shim/shim.js:1313:22
    at LegacyContextManager.runInContext (/app/node_modules/newrelic/lib/context-manager/legacy-context-manager.js:59:23)
    at Shim.applySegment (/app/node_modules/newrelic/lib/shim/shim.js:1303:25)
    at Timeout.wrappedCallback [as _onTimeout] (/app/node_modules/newrelic/lib/shim/shim.js:1187:21)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)
[2022-11-16 15:50:18] [ERROR] [19] [error-handler-mw] timeout reached while waiting for fetchSockets response
  stack -> Error: timeout reached while waiting for fetchSockets response
    at Timeout.<anonymous> (/app/node_modules/@socket.io/redis-adapter/dist/index.js:712:28)
    at /app/node_modules/newrelic/lib/shim/shim.js:1313:22
    at LegacyContextManager.runInContext (/app/node_modules/newrelic/lib/context-manager/legacy-context-manager.js:59:23)
    at Shim.applySegment (/app/node_modules/newrelic/lib/shim/shim.js:1303:25)
    at Timeout.wrappedCallback [as _onTimeout] (/app/node_modules/newrelic/lib/shim/shim.js:1187:21)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)
[2022-11-16 15:50:18] [ERROR] [19] [error-handler-mw] timeout reached while waiting for fetchSockets response
  stack -> Error: timeout reached while waiting for fetchSockets response
    at Timeout.<anonymous> (/app/node_modules/@socket.io/redis-adapter/dist/index.js:712:28)
    at /app/node_modules/newrelic/lib/shim/shim.js:1313:22
    at LegacyContextManager.runInContext (/app/node_modules/newrelic/lib/context-manager/legacy-context-manager.js:59:23)
    at Shim.applySegment (/app/node_modules/newrelic/lib/shim/shim.js:1303:25)
    at Timeout.wrappedCallback [as _onTimeout] (/app/node_modules/newrelic/lib/shim/shim.js:1187:21)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)
[2022-11-16 15:50:18] [ERROR] [19] [error-handler-mw] timeout reached while waiting for fetchSockets response
  stack -> Error: timeout reached while waiting for fetchSockets response
    at Timeout.<anonymous> (/app/node_modules/@socket.io/redis-adapter/dist/index.js:712:28)
    at /app/node_modules/newrelic/lib/shim/shim.js:1313:22
    at LegacyContextManager.runInContext (/app/node_modules/newrelic/lib/context-manager/legacy-context-manager.js:59:23)
    at Shim.applySegment (/app/node_modules/newrelic/lib/shim/shim.js:1303:25)
    at Timeout.wrappedCallback [as _onTimeout] (/app/node_modules/newrelic/lib/shim/shim.js:1187:21)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)
[2022-11-16 15:50:18] [ERROR] [19] [error-handler-mw] timeout reached while waiting for fetchSockets response
  stack -> Error: timeout reached while waiting for fetchSockets response
    at Timeout.<anonymous> (/app/node_modules/@socket.io/redis-adapter/dist/index.js:712:28)
    at /app/node_modules/newrelic/lib/shim/shim.js:1313:22
    at LegacyContextManager.runInContext (/app/node_modules/newrelic/lib/context-manager/legacy-context-manager.js:59:23)
    at Shim.applySegment (/app/node_modules/newrelic/lib/shim/shim.js:1303:25)
    at Timeout.wrappedCallback [as _onTimeout] (/app/node_modules/newrelic/lib/shim/shim.js:1187:21)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)
[2022-11-16 15:50:18] [ERROR] [19] [error-handler-mw] timeout reached while waiting for fetchSockets response
  stack -> Error: timeout reached while waiting for fetchSockets response
    at Timeout.<anonymous> (/app/node_modules/@socket.io/redis-adapter/dist/index.js:712:28)
    at /app/node_modules/newrelic/lib/shim/shim.js:1313:22
    at LegacyContextManager.runInContext (/app/node_modules/newrelic/lib/context-manager/legacy-context-manager.js:59:23)
    at Shim.applySegment (/app/node_modules/newrelic/lib/shim/shim.js:1303:25)
    at Timeout.wrappedCallback [as _onTimeout] (/app/node_modules/newrelic/lib/shim/shim.js:1187:21)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)
[2022-11-16 15:50:18] [ERROR] [19] [error-handler-mw] timeout reached while waiting for fetchSockets response
  stack -> Error: timeout reached while waiting for fetchSockets response
    at Timeout.<anonymous> (/app/node_modules/@socket.io/redis-adapter/dist/index.js:712:28)
    at /app/node_modules/newrelic/lib/shim/shim.js:1313:22
    at LegacyContextManager.runInContext (/app/node_modules/newrelic/lib/context-manager/legacy-context-manager.js:59:23)
    at Shim.applySegment (/app/node_modules/newrelic/lib/shim/shim.js:1303:25)
    at Timeout.wrappedCallback [as _onTimeout] (/app/node_modules/newrelic/lib/shim/shim.js:1187:21)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)
[2022-11-16 15:50:18] [ERROR] [19] [error-handler-mw] timeout reached while waiting for fetchSockets response
  stack -> Error: timeout reached while waiting for fetchSockets response
    at Timeout.<anonymous> (/app/node_modules/@socket.io/redis-adapter/dist/index.js:712:28)
    at /app/node_modules/newrelic/lib/shim/shim.js:1313:22
    at LegacyContextManager.runInContext (/app/node_modules/newrelic/lib/context-manager/legacy-context-manager.js:59:23)
    at Shim.applySegment (/app/node_modules/newrelic/lib/shim/shim.js:1303:25)
    at Timeout.wrappedCallback [as _onTimeout] (/app/node_modules/newrelic/lib/shim/shim.js:1187:21)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)
[2022-11-16 15:50:18] [ERROR] [19] [error-handler-mw] timeout reached while waiting for fetchSockets response
  stack -> Error: timeout reached while waiting for fetchSockets response
    at Timeout.<anonymous> (/app/node_modules/@socket.io/redis-adapter/dist/index.js:712:28)
    at /app/node_modules/newrelic/lib/shim/shim.js:1313:22
    at LegacyContextManager.runInContext (/app/node_modules/newrelic/lib/context-manager/legacy-context-manager.js:59:23)
    at Shim.applySegment (/app/node_modules/newrelic/lib/shim/shim.js:1303:25)
    at Timeout.wrappedCallback [as _onTimeout] (/app/node_modules/newrelic/lib/shim/shim.js:1187:21)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)
@patrykcieszkowski
Copy link
Contributor

These two issues are related:
#120

@alexey-yarmosh
Copy link
Member

The error is from redis-adapter module. It has a default 5000 ms timeout for operations. Every fetchSockets operation talks to redis. Under high load, during which redis operations tooks seconds or even minutes, fetchSockets tooks too long and we see "timeout reached while waiting for fetchSockets response" in the logs. 500 Internal Server Error is sent to user.

Possible solutions are increase the adapter timeout or scale/tune the redis. Raising the timeout seems pointless as it is not fixing the root problem (redis perf). And since we are curenlty focusing on stabilizing without infra updates, I am closing the issue.

This was referenced Nov 30, 2022
Closed
Closed
@alexey-yarmosh alexey-yarmosh mentioned this issue Aug 29, 2023
Closed
@alexey-yarmosh
Copy link
Member

90% of the time issue happens because of many parallel probes connecting to the API simultaneously on the API start/restart. jsdelivr/globalping-probe#179 will help here partially, other part of fix is perf tuning. When #418 is finished, we will have an environment to test different approaches like #419 or #307

@MartinKolarik
Copy link
Member

#302

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@MartinKolarik MartinKolarik

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jimaek @MartinKolarik @patrykcieszkowski @alexey-yarmosh