How should we handle SSL expiration going forward?

[ml-evs]

https://providers.optimade.org is currently down due to an expired cert (Materials-Consortia/status#8). I will try to ask around and find who is in control of this...

[CasperWA]

I think this is @gmrigna?

[ml-evs]

I've asked the UCLouvain IT desk, but I'm actually not sure if it's their problem. It seems like https://providers.optimade.org is from the netlify build but https://optimade.org/providers is from GitHub pages. Once they get back to me, I will ask them to remove the record for providers.optimade.org and just use the GH pages build, I think... unless someone has access to the netlify?

[CasperWA]

For Netlify, I think this was possibly setup by @rartino?

[rartino]

@ml-evs, @CasperWA, @gmrigna ; ah - yes - this needs action from the UCLovain IT desk to renew the certificate, and then me to upload the renewed certificate to Netlify. (When we set this up over the vacations last year, I really did intend to work out a renewal protocol so we'd not miss this date. Once we've renewed this time, let us get that protocol sorted for next iteration.)

For very technical reasons that relate both to Netlify and UCLovain policies, we must use UCLovains certificate authority for subdomains hosted on Netlify (even though Letsencrypt works nicely for subdomains hosted on github). Hence, please ask the UCLovain IT desk to renew the 'providers.optimade.org' certificate and then securely transfer to me:

• The 'PEM formatted cetificate'.
• The private key used for the certificate request.
• The CA certificate chain.

And will be able to use these to update the Netlify certificate.

[ml-evs]

@rartino: I've just forwarded you the secure links from UCLouvain, it looks like they have everything you requested.

Do you think we should try to move this hosting to GitHub pages to avoid this again in 1 year? Or can more of us get access to the netlify deployment to spread the responsibility?

[rartino]

@ml-evs Well, we should at least certainly come up with a protocol to make sure the certificate is renewed before it expires next time around.

We went to Netlify because we need URLs that don't end with .json to serve things with a JSON mimetype. When we did this, we said to keep an eye on if GitHub pages enables these features, and if so, swap back, but as far as I know it isn't possible.

The "correct" solution is to set up an organization Netlify account with multiple 'key' people having access. However, last time I looked this costs quite a bit of money. However, with the traction we have right now, we might actually be able to fund that. (But also, when I look now, Netlify pricing now says something about 'teams' also on the free plan, let me look into that.)

Edit: Oh, and I forgot to say: I have upgraded the cert now. Let me know if anything still doesn't work.

[rartino]

After looking around, as far as I can see, Netlify with multiple collaborators will cost at least $15/account/month (but we may need $19/account/month for pro, because at first look it seems there are obstacles to be connected to multiple 'starter' plans). It isn't impossible to fund this for a couple of users, if we think it is crucial that multiple people have access.

On the other hand, even with single-user Netlify hosting, there may be a somewhat safe protocol we can use that doesn't really rely on a "single person of failure".

• We set up a service (GitHub action?) to automatically create a GitHub issue for "certificate renewal" 6 weeks before the cert expires (this also applies to the https://optimade.org cert that also will expire in about a year). This issue will be visible to everyone.
• We expect the current 'key' people to act on this within a couple of weeks and renew the certificate.
• However, our policy is that if the certificate is not renewed within 4 weeks, other people are encouraged to step in and re-route the website to a different Netlify account and install the renewed certificate there.

(This prevents an issue if the person owning the Netlify account go missing. However, we still absolutely rely on that someone at UCLovain can renew the cert. Not sure what to do about that.)

[ml-evs]

Thanks @rartino, everything is back to normal (see Materials-Consortia/status#8). I'll rename this issue so we can continue the discussion about certs.

I don't have a strong preference for how to handle this, though paying for netlify feels like overkill to host some static files - we could switch to a free tier of Heroku, or maybe even Cloudflare free (which I think would let us host on gh pages but define some proxies to circumvent the mime-type issue - not clear if we could set up a shared account though)?

More generally, it would be good to make the whole providers approach more robust, perhaps a few implementations could also "officially" mirror the content of this repository at e.g. providers.odbx.science in a more formal way than just providing external links at optimade.odbx.science/v1/links. We can then maintain a "provider providers" fallback list for clients to use...

[rartino]

@ml-evs We probably want to stick with a solution that has tight integration with GitHub with e.g., preview URLs for PRs. I think Heroku has it (?), but as far as I know there are not many other good alternatives. Am I reading right that Heroku allows teams up to 5 persons for free? That sounds useful for us. I never used it, is the free tier generally considered reliable?

I'm opposed to changing the handling of mime types to Cloudflare trickery; that seems as a hack of the kind that is easily forgotten with time, breaks when people who don't understand the setup tries to make changes, and overall would greatly complicate our configuration.

I like the idea of endorsed mirrors of providers.optimade.org. Nevertheless, they also don't really solve the key issue we discuss here: making sure we have a protocol for renewing certificates on our main server that does not hinge on any one single person not going missing.

We can start with setting up a GitHub action to file an issue on impending certificate expiry. It seems sectigo allows renewals up to 90 days before the expiry, so maybe creating the issue 60 days before the expiry is good? There seems to be actions to do this on the market, e.g., https://github.com/marketplace/actions/domain-and-ssl-cert-watcher.

If we need a Linux command, this seems to be the way to do it:

if ! openssl s_client -servername providers.optimade.org -connect providers.optimade.org:443 2>/dev/null < /dev/null | openssl x509 -checkend $((60*60*24*60)); then
  echo "File issue to renew certificate!"
fi

[ml-evs]

@ml-evs We probably want to stick with a solution that has tight integration with GitHub with e.g., preview URLs for PRs. I think Heroku has it (?), but as far as I know there are not many other good alternatives. Am I reading right that Heroku allows teams up to 5 persons for free? That sounds useful for us. I never used it, is the free tier generally considered reliable?

We use Heroku to do a deployment of the reference server in optimade-python-tools. There are 3 of us attached to the account, so 5 sounds plausible too... My understanding is that Heroku is less targeted towards static sites than Netlify, so we may need to set up a simple web server ourselves to serve the files, with presumably higher latency as the server will be shutdown when idle. I've found it to be very reliable, so it could be an option (especially if we can find the appropriate way of serving static files).

EDIT: @fekad has pointed out that Heroku's free tier does not allow custom domains, so nevermind!

I'm opposed to changing the handling of mime types to Cloudflare trickery; that seems as a hack of the kind that is easily forgotten with time, breaks when people who don't understand the setup tries to make changes, and overall would greatly complicate our configuration.

👍

I like the idea of endorsed mirrors of providers.optimade.org. Nevertheless, they also don't really solve the key issue we discuss here: making sure we have a protocol for renewing certificates on our main server that does not hinge on any one single person not going missing.

We can start with setting up a GitHub action to file an issue on impending certificate expiry. It seems sectigo allows renewals up to 90 days before the expiry, so maybe creating the issue 60 days before the expiry is good? There seems to be actions to do this on the market, e.g., https://github.com/marketplace/actions/domain-and-ssl-cert-watcher.

Sounds good to me

[JPBergsma]

In about a month, the certificate is going to expire again.
Could one of you @rartino or @ml-evs take action to renew the certificate before it expires around the 6th of august?
If I remember correctly, the certificate of https://optimade.org/ ends somewhere in September, so perhaps that certificate could be renewed as well.

[rartino]

@JPBergsma

Right - we're on trying to solve the other expiry that happened today (see Materials-Consortia/materials-consortia.github.io#55), and should try to handle this at the same time.

(For anyone curious about the technical issues behind our troubles with SSL certificates for pages hosted via netlify, here is a link to the support issue I posted way back when we first ran into this: https://answers.netlify.com/t/no-caa-on-netlify-subdomains/19796/2 )

[JPBergsma]

I had not seen the other issue(#55) yet.
From the previous discussion in this thread, it was not clear to me whether an automatic notification had been setup, so I thought it would be good to give a reminder.

[rartino]

Ok, the certificate is renewed until 2023 now.

[JPBergsma]

Thank you for renewing the certificate.
When I checked on https://www.ionos.com/tools/ssl-checker
I still got a warning that the certificates are installed in the wrong order.
I do not know whether this is a problem, my browser does not give any warning when visiting the site.

[rartino]

Thanks for pointing this out. I just assumed that if Netlify accepted the PEM encoded certs, it would serve things correctly, but apparently one needs to be a bit careful. However, as you also found, most browsers are very forgiving with this. It should be fixed now.

For anyone having similar issues, the "Certificate chain" config box in Netlify should be the PEM certs delivered in "Issuing CA certificates only/Intermediate(s):Root only, PEM encoded" BUT with the very last one REMOVED (i.e., the root).