Bad Gateway error when checking crash signatures on bugzilla-dev.allizom.org

[suhaibmujahid]

2023-08-15 09:58:29,586 - ERROR - Rule file_crash_bug
Traceback (most recent call last):
  File "/Users/smujahid/repos/mozilla/bugbot/bugbot/bzcleaner.py", line 844, in run
    self.send_email(date=date)
  File "/Users/smujahid/repos/mozilla/bugbot/bugbot/bzcleaner.py", line 763, in send_email
    data = self.get_email_data(date)
  File "/Users/smujahid/repos/mozilla/bugbot/bugbot/bzcleaner.py", line 681, in get_email_data
    bugs = self.get_bugs(date=date)
  File "/Users/smujahid/repos/mozilla/bugbot/bugbot/rules/file_crash_bug.py", line 88, in get_bugs
    signatures = data_fetcher.analyze()
  File "/Users/smujahid/repos/mozilla/bugbot/bugbot/crash/analyzer.py", line 909, in analyze
    bugs = self.fetch_bugs()
  File "/Users/smujahid/repos/mozilla/bugbot/bugbot/crash/analyzer.py", line 899, in fetch_bugs
    ).wait()
  File "/Users/smujahid/repos/mozilla/bugbot/venv/lib/python3.10/site-packages/libmozdata/bugzilla.py", line 177, in wait
    super(Bugzilla, self).wait()
  File "/Users/smujahid/repos/mozilla/bugbot/venv/lib/python3.10/site-packages/libmozdata/connection.py", line 135, in wait
    r.result()
  File "/opt/homebrew/Cellar/[email protected]/3.10.11/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/opt/homebrew/Cellar/[email protected]/3.10.11/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/opt/homebrew/Cellar/[email protected]/3.10.11/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/smujahid/repos/mozilla/bugbot/venv/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/smujahid/repos/mozilla/bugbot/venv/lib/python3.10/site-packages/requests/sessions.py", line 710, in send
    r = dispatch_hook("response", hooks, r, **kwargs)
  File "/Users/smujahid/repos/mozilla/bugbot/venv/lib/python3.10/site-packages/requests/hooks.py", line 30, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/Users/smujahid/repos/mozilla/bugbot/venv/lib/python3.10/site-packages/libmozdata/connection.py", line 124, in cb
    res.raise_for_status()
  File "/Users/smujahid/repos/mozilla/bugbot/venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://bugzilla-dev.allizom.org/rest/bug?include_fields=cf_crash_signature&f1=OP&j1=OR&f2=cf_crash_signature&o2=regexp&v2=%5C%5B%28%40+%7C%40%29mozilla%3A%3Alayers%3A%3AImageBridgeChild%3A%3AInitForContent%28+%5C%5D%7C%5C%5D%29&f3=cf_crash_signature&o3=regexp&v3=%5C%5B%28%40+%7C%40%29mozilla%3A%3Anet%3A%3AHttpChannelParent%3A%3AContinueVerification%28+%5C%5D%7C%5C%5D%29&f4=cf_crash_signature&o4=regexp&v4=%5C%5B%28%40+%7C%40%29CallQueryInterface%3CT%3E%28+%5C%5D%7C%5C%5D%29&f5=cf_crash_signature&o5=regexp&v5=%5C%5B%28%40+%7C%40%29new%28+%5C%5D%7C%5C%5D%29&f6=cf_crash_signature&o6=regexp&v6=%5C%5B%28%40+%7C%40%29NSSSocketControl%3A%3ACloseSocketAndDestroy%5C+%5C%7C%5C+nsSSLIOLayerClose%28+%5C%5D%7C%5C%5D%29&f7=cf_crash_signature&o7=regexp&v7=%5C%5B%28%40+%7C%40%29mozilla%3A%3AFFmpegVideoDecoder%3CT%3E%3A%3A%5C~FFmpegVideoDecoder%28+%5C%5D%7C%5C%5D%29&f8=cf_crash_signature&o8=regexp&v8=%5C%5B%28%40+%7C%40%29servo_arc%3A%3AArc%3CT%3E%3A%3Ainner%28+%5C%5D%7C%5C%5D%29&f9=cf_crash_signature&o9=regexp&v9=%5C%5B%28%40+%7C%40%29je_free%28+%5C%5D%7C%5C%5D%29&f10=cf_crash_signature&o10=regexp&v10=%5C%5B%28%40+%7C%40%29js%3A%3AAtomizeString%28+%5C%5D%7C%5C%5D%29&f11=cf_crash_signature&o11=regexp&v11=%5C%5B%28%40+%7C%40%29nvwgf2umx%5C.dll%5C+%5C%7C%5C+RtlpAllocateHeapInternal%28+%5C%5D%7C%5C%5D%29&f12=cf_crash_signature&o12=regexp&v12=%5C%5B%28%40+%7C%40%29core%3A%3Aoption%3A%3Aexpect_failed%5C+%5C%7C%5C+webrender%3A%3Aclip%3A%3AClipTreeBuilder%3A%3Abuild_clip_set%28+%5C%5D%7C%5C%5D%29&f13=cf_crash_signature&o13=regexp&v13=%5C%5B%28%40+%7C%40%29mozilla%3A%3Adom%3A%3AXMLHttpRequest_Binding%3A%3ACreateInterfaceObjects%28+%5C%5D%7C%5C%5D%29&f14=cf_crash_signature&o14=regexp&v14=%5C%5B%28%40+%7C%40%29libvulkan%5C.so%5C.1%400x3ca78%28+%5C%5D%7C%5C%5D%29&f15=CP&f16=status_whiteboard&o16=substring&v16=%5Bbugbot-crash-v3%5D

[suhaibmujahid]

From @dlactin on Slack:

I've increased the backend timeout to 60s and added more application pods. If it still fails the bottleneck could be at the database level which is a bit more time consuming to adjust.

It seems to be working fine now!

[suhaibmujahid]

It is happening again:

2023-09-06 17:45:11,877 - ERROR - Rule file_crash_bug
Traceback (most recent call last):
  File "/home/rm_bot/bugbot/bugbot/bzcleaner.py", line 844, in run
    self.send_email(date=date)
  File "/home/rm_bot/bugbot/bugbot/bzcleaner.py", line 763, in send_email
    data = self.get_email_data(date)
  File "/home/rm_bot/bugbot/bugbot/bzcleaner.py", line 681, in get_email_data
    bugs = self.get_bugs(date=date)
  File "/home/rm_bot/bugbot/bugbot/rules/file_crash_bug.py", line 88, in get_bugs
    signatures = data_fetcher.analyze()
  File "/home/rm_bot/bugbot/bugbot/crash/analyzer.py", line 909, in analyze
    bugs = self.fetch_bugs()
  File "/home/rm_bot/bugbot/bugbot/crash/analyzer.py", line 899, in fetch_bugs
    ).wait()
  File "/home/rm_bot/bugbot/venv/lib/python3.10/site-packages/libmozdata/bugzilla.py", line 177, in wait
    super(Bugzilla, self).wait()
  File "/home/rm_bot/bugbot/venv/lib/python3.10/site-packages/libmozdata/connection.py", line 135, in wait
    r.result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/rm_bot/bugbot/venv/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/rm_bot/bugbot/venv/lib/python3.10/site-packages/requests/sessions.py", line 710, in send
    r = dispatch_hook("response", hooks, r, **kwargs)
  File "/home/rm_bot/bugbot/venv/lib/python3.10/site-packages/requests/hooks.py", line 30, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/home/rm_bot/bugbot/venv/lib/python3.10/site-packages/libmozdata/connection.py", line 124, in cb
    res.raise_for_status()
  File "/home/rm_bot/bugbot/venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://bugzilla-dev.allizom.org/rest/bug?include_fields=cf_crash_signature&f1=OP&j1=OR&f2=cf_crash_signature&o2=regexp&v2=%5C%5B%28%40+%7C%40%29igd10um64gen11%5C.dll%5C+%5C%7C%5C+%3Cunknown%5C+in%5C+d3d11%5C.pdb%3E%28+%5C%5D%7C%5C%5D%29&f3=cf_crash_signature&o3=regexp&v3=%5C%5B%28%40+%7C%40%29strlen%5C+%5C%7C%5C+nsCharTraits%3CT%3E%3A%3Alength%28+%5C%5D%7C%5C%5D%29&f4=cf_crash_signature&o4=regexp&v4=%5C%5B%28%40+%7C%40%29mozilla%3A%3Adom%3A%3AMultipartBlobImpl%3A%3AGetAllocationSize%28+%5C%5D%7C%5C%5D%29&f5=cf_crash_signature&o5=regexp&v5=%5C%5B%28%40+%7C%40%29mozilla%3A%3AVideoFrameSurface%3CT%3E%3A%3AReleaseVAAPIData%28+%5C%5D%7C%5C%5D%29&f6=cf_crash_signature&o6=regexp&v6=%5C%5B%28%40+%7C%40%29ffm64%5C.dll%5C+%5C%7C%5C+LdrControlFlowGuardEnforcedWithExportSuppression%28+%5C%5D%7C%5C%5D%29&f7=cf_crash_signature&o7=regexp&v7=%5C%5B%28%40+%7C%40%29arena_run_reg_dalloc%5C+%5C%7C%5C+arena_t%3A%3ADallocSmall%5C+%5C%7C%5C+arena_dalloc%5C+%5C%7C%5C+idalloc%28+%5C%5D%7C%5C%5D%29&f8=cf_crash_signature&o8=regexp&v8=%5C%5B%28%40+%7C%40%29mozilla%3A%3Alayers%3A%3ACanvasEventRingBuffer%3A%3ASetNewBuffer%28+%5C%5D%7C%5C%5D%29&f9=cf_crash_signature&o9=regexp&v9=%5C%5B%28%40+%7C%40%29arena_t%3A%3AArenaRunRegAlloc%5C+%5C%7C%5C+arena_t%3A%3AMallocSmall%5C+%5C%7C%5C+arena_t%3A%3AMalloc%5C+%5C%7C%5C+BaseAllocator%3A%3Amalloc%5C+%5C%7C%5C+Allocator%3CT%3E%3A%3Amalloc%5C+%5C%7C%5C+PageMalloc%28+%5C%5D%7C%5C%5D%29&f10=cf_crash_signature&o10=regexp&v10=%5C%5B%28%40+%7C%40%29nsAppShell%3A%3AInitHiddenWindow%28+%5C%5D%7C%5C%5D%29&f11=cf_crash_signature&o11=regexp&v11=%5C%5B%28%40+%7C%40%29moz_container_wayland_surface_lock%28+%5C%5D%7C%5C%5D%29&f12=cf_crash_signature&o12=regexp&v12=%5C%5B%28%40+%7C%40%29js%3A%3Ajit%3A%3ABailoutFrameInfo%3A%3ABailoutFrameInfo%28+%5C%5D%7C%5C%5D%29&f13=cf_crash_signature&o13=regexp&v13=%5C%5B%28%40+%7C%40%29RtlpFreeHeapInternal%5C+%5C%7C%5C+RtlFreeHeap%5C+%5C%7C%5C+delete%28+%5C%5D%7C%5C%5D%29&f14=cf_crash_signature&o14=regexp&v14=%5C%5B%28%40+%7C%40%29selectors%3A%3Abuilder%3A%3ASelectorFlags%3A%3Ais_empty%28+%5C%5D%7C%5C%5D%29&f15=cf_crash_signature&o15=regexp&v15=%5C%5B%28%40+%7C%40%29mozilla%3A%3ARestyleManager%3A%3AProcessPostTraversal%28+%5C%5D%7C%5C%5D%29&f16=cf_crash_signature&o16=regexp&v16=%5C%5B%28%40+%7C%40%29core%3A%3Aoption%3A%3Aexpect_failed%5C+%5C%7C%5C+hashbrown%3A%3Araw%3A%3AProbeSeq%3A%3Amove_next%28+%5C%5D%7C%5C%5D%29&f17=cf_crash_signature&o17=regexp&v17=%5C%5B%28%40+%7C%40%29core%3A%3Aresult%3A%3Aunwrap_failed%5C+%5C%7C%5C+webrender%3A%3Arenderer%3A%3Aupload%3A%3Acopy_into_staging_buffer%28+%5C%5D%7C%5C%5D%29&f18=cf_crash_signature&o18=regexp&v18=%5C%5B%28%40+%7C%40%29IPCError%5C-browser%5C+%5C%7C%5C+Killed%5C+via%5C+about%3Acrashextensions%28+%5C%5D%7C%5C%5D%29&f19=cf_crash_signature&o19=regexp&v19=%5C%5B%28%40+%7C%40%29JSObject%3A%3AallocKindForTenure%28+%5C%5D%7C%5C%5D%29&f20=cf_crash_signature&o20=regexp&v20=%5C%5B%28%40+%7C%40%29core%3A%3Aresult%3A%3Aunwrap_failed%5C+%5C%7C%5C+wgpu_server_encoder_copy_texture_to_buffer%28+%5C%5D%7C%5C%5D%29&f21=cf_crash_signature&o21=regexp&v21=%5C%5B%28%40+%7C%40%29mozilla%3A%3AJSHolderMap%3A%3AExtract%28+%5C%5D%7C%5C%5D%29&f22=cf_crash_signature&o22=regexp&v22=%5C%5B%28%40+%7C%40%29webrender%3A%3Aintern%3A%3Aimpl%5C%247%3A%3Aindex%28+%5C%5D%7C%5C%5D%29&f23=cf_crash_signature&o23=regexp&v23=%5C%5B%28%40+%7C%40%29libxul%5C.so%400x65a7d4f%5C+%5C%7C%5C+libxul%5C.so%400x3fa1554%5C+%5C%7C%5C+libxul%5C.so%400x3fa1228%5C+%5C%7C%5C+libxul%5C.so%400x65d88be%28+%5C%5D%7C%5C%5D%29&f24=cf_crash_signature&o24=regexp&v24=%5C%5B%28%40+%7C%40%29nsCycleCollectingAutoRefCnt%3A%3Aget%28+%5C%5D%7C%5C%5D%29&f25=cf_crash_signature&o25=regexp&v25=%5C%5B%28%40+%7C%40%29RtlpAllocateHeap%5C+%5C%7C%5C+RtlpAllocateHeapInternal%5C+%5C%7C%5C+opentext64%5C.dll%5C+%5C%7C%5C+RtlFreeHeap%5C+%5C%7C%5C+UserCallWinProcCheckWow%28+%5C%5D%7C%5C%5D%29&f26=cf_crash_signature&o26=regexp&v26=%5C%5B%28%40+%7C%40%29nsIFrame%3A%3AGetOffsetToCrossDoc%28+%5C%5D%7C%5C%5D%29&f27=cf_crash_signature&o27=regexp&v27=%5C%5B%28%40+%7C%40%29IPCError%5C-content%5C+%5C%7C%5C+RecvNotifyVisited%28+%5C%5D%7C%5C%5D%29&f28=cf_crash_signature&o28=regexp&v28=%5C%5B%28%40+%7C%40%29nsIFrame%3A%3AHasAnyStateBits%28+%5C%5D%7C%5C%5D%29&f29=cf_crash_signature&o29=regexp&v29=%5C%5B%28%40+%7C%40%29JS%3A%3ARealm%3A%3AunsafeUnbarrieredMaybeGlobal%28+%5C%5D%7C%5C%5D%29&f30=cf_crash_signature&o30=regexp&v30=%5C%5B%28%40+%7C%40%29mozilla%3A%3AMediaFormatReader%3A%3ARequestVideoData%28+%5C%5D%7C%5C%5D%29&f31=cf_crash_signature&o31=regexp&v31=%5C%5B%28%40+%7C%40%29ssl_SendSavedWriteData%5C+%5C%7C%5C+ssl3_SendRecord%5C+%5C%7C%5C+ssl3_FlushHandshakeMessages%5C+%5C%7C%5C+ssl3_FlushHandshake%5C+%5C%7C%5C+tls13_SendClientSecondFlight%28+%5C%5D%7C%5C%5D%29&f32=CP&f33=status_whiteboard&o33=substring&v33=%5Bbugbot-crash-v3%5D