Migrate US server to new provider

[luisramos0]

Description

As a result of the massive growth in use experienced this spring, both the OFN USA server and the OFN CAN server are at capacity, and moving to a bigger server is highly recommended. Both OFN CAN and OFN USA would like to move to servers that are run by more ethical and/or environmentally responsible companies. OFN CAN uses Digital Ocean, which is near the bottom of the list of environmentally responsible practices for web hosting services.

The USA's access for website and postgres/Zapier updates has become very difficult. We need to move to a server which can cope with the growth that OFN USA is experiencing.

Pre-Migration

• Buy a new server
• Create a temporary subdomain and point it at the server
• Set up some temporary configs for the new server (hosts entry, host_vars, etc)
• Run thesetup playbook on the new server
• Use the new letsencrypt_proxy playbook on the current server, specifying the new server's IP
• Run theprovision playbook on the new server
• Deploy the current release to the new server
• Run the db_integrations playbook on the new server, if it uses integrations like Zapier.

Migration

• Set the current server in maintenance mode, with a nice custom message
• Stop unicorn and delayed_job on the current server
• Transfer the assets to the new server (includes all images, which are hosted locally on the US server) < -- is this using playbooks/transfer_assets.yml?
• Migrate the database to the new server with db_transfer playbook
• Restart delayed_job and unicorn on the new server to ensure changes are picked up
• Check everything is okay on the site
• Run transparent_proxy playbook on the current server, pointing to the new one

Post-Migration

• Switch the domain name to point to the new server's IP
• Leave some time for the DNS to propagate, then shut down the old server
• Clean up the ofn-install inventory entries so only one remains, with the new IP address

Optional Extras

• Remove the temporary subdomain
• Regenerate the certificate on the new server, with the temporary subdomain removed from the list Katuma Server Upgrade openfoodnetwork#5418 (comment)

Expected Behavior

New server for USA can handle growth, zapier integrations and be in a vlues aligned provider.

Possible Fix

This is related to
https://community.openfoodnetwork.org/t/switch-to-green-hosting-when-new-projects-provisionning/1616

[lauriewayne]

Thank you for creating this issue, Luis! What communication is needed with Jim, the holder of the AWS account where the US server currently resides?

[luisramos0]

I dont think we need anything, we do have admin access to the server so we can take whatever we need from there.
He will probably just need to stop the instance once the migration is done. Which is really just to press a button.

[luisramos0]

I just confirmed the DNS is on AUS godaddy account (I have delegate access given by Kirsten to this godaddy account if anyone needs to change something).

[lauriewayne]

Great, thanks! Sometimes it's hard to get ahold of Jim. Will anything on our side break or be problematic if he doesn't press the button to stop the instance? If we will be impacted by his stopping or not stopping the instance, I should start working on that now.

[luisramos0]

no not at all. Stopping the instance means to switch it down when it's all migrated. The only problem of not switching down is that he will have to pay for it afterwards even if we dont use it.
This is something for after the migration.

[lauriewayne]

Perfect thank you @luisramos0! I will let him know after we are happily up and running on the new server.

[Matt-Yorkley]

Server migration template here: openfoodfoundation/openfoodnetwork#5418 😉

[andrewpbrett]

I started a conversation with the current holder of the AWS account to start figuring out steps to transfer to another AWS machine.

For later on, looking at the specs on some of the other instances, most have 4 x CPU and 8GB memory.

• On AWS EC2 that translates to an a1.xlarge reserved instance, which is $47/month (hosted in Oregon)
• On Digital Ocean a standard droplet with 4x CPU and 8GB memory is $40/month
• On Infomaniak an unmanaged cloud server with 4x CPU and 12GB memory is 49€/month (currently about $58)

Infomaniak does have the advantage of being based in Switzerland (privacy wise) and looks more values-aligned, not to mention the slightly higher specs.

[luisramos0]

Hey @andrewpbrett nice start!

On Infomaniak an unmanaged cloud server with 4x CPU and 12GB memory is 49€/month (currently about $58)

Are the servers in CH as well? The users will be in the US, you need a server in some US data center, right?

OVH, the provider for France is cheap

I think they will have servers in the US.

[andrewpbrett]

OVH looks nice! They do have servers in the Eastern US (Virginia) as well as Montreal. Maybe Montreal would be appealing since it's close enough for latency but outside the scope of the Patriot Act etc.? (I don't know how much the physical location of the server makes a difference for that though, as opposed to how and where the provider is organized).

[lauriewayne]

Yay! I definitely prioritize reliability and customer service/reputation but given those, cheaper (and more ethical?) is awesome!!! Thank you @luisramos0 and @andrewpbrett for your sleuthing!

[andrewpbrett]

I'm still waiting to get what we need from our current AWS account holder for this to proceed. I'm hopeful that once we have it, we could first do a quick(er) transfer to an AWS account that we control and then later switch providers (probably to OVH, but need a little more research on it first).

[Matt-Yorkley]

Ok, so we're scheduled for migrating on Monday morning. I'm just doing a few checks now, I think there's a minor thing with the TLS certificates setup that needs a slight adjustment 👌

[Matt-Yorkley]

Alright, I think we're done. That was faaast! 🚀

I ran into a small email issue related to #696, but it's fixed now.

@andrewpbrett you can update the DNS now so the primary domain points to the new server's IP address. Note: we should leave the old server running for a day or so before shutting it down, and make sure nobody provisions the old server until then.

[Matt-Yorkley]

I think the total downtime was around 10 minutes 💪

[lauriewayne]

Oooh hey, email confirmations for new registrations want to go to"demo.openfoodnetwork.org" and the site url was set to that in the general settings. I changed the general settings to "openfoodnetwork.net" but confirmations still want to go to the demo site. Is there anything else that might be impacted by this (for example order confirmations yikes)?

[RachL]

ping @openfoodfoundation/core-devs some configurations are missing ☝️

[andrewpbrett]

Thanks for noticing that and reporting @lauriewayne. I redeployed the latest release after the site URL was changed and the emails now have the correct link. I think I mistakenly assumed that the values in general settings would have been carried over to the new site with the database, but thinking about that more I wouldn't be surprised if some of them are actually environment variables.... probably worth adding a warning/step in the migration "template"?

[andrewpbrett]

All done 🎉