FR Prod instance down

[audez]

Description

Website completely down.
Then Getting the snail when trying to access any reports at https://coopcircuits.fr/admin/reports

[audez]

this is OK now. The bug lasted 1h-2h, was reported at 9:55 this morning

• snail on reports
• importing orders to the integrated cash software (Pastèque) failed
• Every page was very slow to load (It was already slow yesterday)

It looks like it might be linked to our previous memory issue..

[audez]

ongoing work:
#9398
#9413

[RachL]

@audez the snail did not occur only on reports, right, on all pages?

As on kuma we can see it's a server timeout:

Note we had the same downtime yesterday.

Datadog confirms the first spike occured at 9:49 with some request already starting at 9:39 :

I haven't had time yet to see which request started the chain reaction. Perhaps if someone who has ssh access can access the logs, that would be quicker.

I'll be back on Monday, but if we need to put more metal to it in the meantime, our credit card is stored in our host account. You can reach me on telegram or signal if there is a 2FA problem.

[audez]

thanks!
Yes you're right, the user that reported at 9:55 confirms that everything was down.
For me the snail occured only when trying to access a report - I tested around 11:30. I was able to access orders (but it was veeery slow) and all pages were slow. Yesterday everything was slow too - I think I loaded some reports around 12:30 and got no snail though

[audez]

everything down again around 15:15 --> i did a superadmin enterprise search, might be due to this..
Edit: and down again at 15:30

It's confirmed by datadog spike

[RachL]

@audez did you loggedin as super admin or enterprise user?

[audez]

in a window as super admin and in another enterprise user
just now without being logged in i couldn't reach coopcircuits.fr, down again
Now it seems ok but very slow

[mkllnk]

in a window as super admin and in another enterprise user

If it's in the same browser then you are logged in with the same account.

[audez]

@mkllnk yea thanks :P it was same browser but one private window and one normal so I could test both. But it was actually the same whether you were logged in as superadmin, enterprise, or not logged in..

[mkllnk]

Probably best to plan downtime between 2am and 7am French time. That should cover my workday very well.

[RachL]

Closing for now. Let's see if we need to reopen after openfoodfoundation/ofn-install#825