Migrate CAN server to new provider

[luisramos0]

Description

As a result of the massive growth in use experienced this spring, both the OFN USA server and the OFN CAN server are at capacity, and moving to a bigger server is highly recommended. Both OFN CAN and OFN USA would like to move to servers that are run by more ethical and/or environmentally responsible companies. OFN CAN uses Digital Ocean, which is near the bottom of the list of environmentally responsible practices for web hosting services.

We need to move to a server which can cope with the growth that OFN CAN is experiencing.

Expected Behavior

New server for CAN can handle growth, zapier integrations and be in a vlues aligned provider.

Possible Fix

This is related to
https://community.openfoodnetwork.org/t/switch-to-green-hosting-when-new-projects-provisionning/1616

[tschumilas]

FYI - I'm not sure this came up because of a capacity problem. I'm not aware that OFN-CAN has a capacity problem - but maybe we do? I think this came up because OFN-CAN is going to pay for a new OFN-US server for the time being (until our money runs out anyway), and we thought if we were going to seek a sustainable server for the US, we could seek a more sustainable option for Canada too, given Digital Ocean seems to have a poor record in this regard. However, that said, given capacity issues - we should maybe focus on the US first - OFN CAN is fine for now.

[lauriewayne]

Per https://openfoodnetwork.slack.com/archives/G22SQCKJA/p1593799223371200, Matt thinks both servers are undersized (though US probably more so).

[luisramos0]

Hello,
Looking at both these upgrades again, I am not convinced we need any of them.
US - We can transfer the US server to a new AWS account owned by us. This is easy to do on the AWS console and it's much easier then installing/migrating a new server.
US - I think I am contradicting what @Matt-Yorkley said but I am 100% sure we can upgrade an ec2 instance as much as we like without having to re-install/migrate the server.
CAN - @Matt-Yorkley can you please explain why do you think CAN has performance issues?

[lauriewayne]

Would it make sense to spend a bit more effort and move the USA server to a socially/environmentally ethical host now instead of moving later? I'm not attached, just wondering about the future.

[kirstenalarsen]

This may not be about new provider - just about upgrading capacity. Selecting socially/env better provider unfortunately not prioritised right now, just urgency to get upgraded asap as server falling over

[Matt-Yorkley]

If there's sharing of costs, would it make sense to migrate US to Canada's DO account?

[tschumilas]

Not sure if it would be a cost savings or not? OFN-CAN will be paying for OFN-US hosting for a year - @lauriewayne - what is the best approach here?
Maybe its not worth doing because we'll have to have a separate account for OFN-US in a year anyway.

[lauriewayne]

tl;dr - let's do what's easiest and cheapest

@tschumilas for me it's a balance between how much money we save and how much overhead it takes to maintain/monitor/share that savings. I am looking at https://www.digitalocean.com/pricing/ and I have no idea how big a "droplet" we would need each or together - if the costs are relatively minimal and it would be cleaner, I would just as soon either have the US pay for its own situation from day 1. If it doesn't matter or is easier from an administration standpoint, that would be great to have both North American instances in the same place (and for the US to take on the cost for both at some point in the very near future, thank you Team Canada)!

Maybe this program will make the costs a non-issue, at least for a while? https://www.digitalocean.com/community/pages/hub-for-good

[andrewpbrett]

@lauriewayne Just to clarify, I wouldn't recommend that the two instances share a droplet. The deploy scripts assume that there's only one instance running on each server (or I'm 99% sure of that, anyway) and it'd require some customization to have two on the same server, which is not a road I think we should go down.

Having both on the same account would slightly simplify administering them. They would be both billed to the same card in that case, though, but maybe there's an arrangement to think about where CA pays for both for a year, then US pays for a year, etc.

[kirstenalarsen]

Two separate accounts seems much simpler for long term. They can both bill to same card, but then very easy to switch one to another card etc

…

On Wed, 20 Jan 2021, 8:25 am Andy Brett, ***@***.***> wrote: @lauriewayne <https://github.com/lauriewayne> Just to clarify, I wouldn't recommend that the two instances share a droplet. The deploy scripts assume that there's only one instance running on each server (or I'm 99% sure of that, anyway) and it'd require some customization to have two on the same server, which is not a road I think we should go down. Having both on the same account would slightly simplify administering them. They would be both billed to the same card in that case, though, but maybe there's an arrangement to think about where CA pays for both for a year, then US pays for a year, etc. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#647 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAWGXSCONFZ2KMB7UNP5A3TS2X2DNANCNFSM4PGRIDMQ> .

[Matt-Yorkley]

So in theory the CA upgrade might just involve resizing the current droplet? We did that with UK before and it was maybe 5 minutes downtime and a couple of button clicks. 👌

[andrewpbrett]

Yep, CA should be pretty simple. I just wanted to have a look at some of the metrics. There's plenty of free space on the disk, but it's consistently over 50% CPU usage. We could double to 4xCPU for $40/month and see where that gets us.

As @Matt-Yorkley mentions, that should just be a short downtime and a couple clicks, that could be done in the middle of the night. And then just some basic checks afterwards to make sure everything started up properly again. Was there any need to restart workers when you did the UK Matt?

[Matt-Yorkley]

I don't remember exactly. We might have done a unicorn restart, but definitely nothing more complex than that. I think Maikel tweaked the unicorn service recently to make it more robust on restarts, so it probably won't need it 👍

But yeah it's good to do a few checks after, like see if mails are delivering and delayed job is doing it's thing.

[andrewpbrett]

@Matt-Yorkley thanks for the background. Assuming @tschumilas is good with the increase to $40/month, I think the only thing left to do then is schedule it...would this weekend be out of the question? Like overnight Saturday or Sunday night? That would translate to Sunday/Monday morning for @Matt-Yorkley if he doesn't mind volunteering :) The steps would be:

1. Log into the CA DO account (credentials are in the usual place)
2. Resize to 4xCPU (for $40/month)
3. Wait about 5 minutes
4. Check that the server is up
5. Send a test mail, which might also confirm that delayed job is working? But just in case:
6. Verify that delayed job is running.

If we get a 👍 from @tschumilas on the timing and the cost, and a 👍 from @Matt-Yorkley on the timing, I think we're good to go on that plan.

[Matt-Yorkley]

I should be free, but it would be good to check the 2FA situation with DO and make sure i can log in before we go for it.

[Matt-Yorkley]

I encountered a small email config issue during the upgrade, documented here.

Other than that it went pretty well. It's done 🎉