SSL/HTTPS enabled WebServer

[DanielBaulig]

Describe the problem you have/What new integration you would like

I would like to see SSL/HTTPS support for the WebServer component. I am aware that there has been a lengthy discussion around SSL/TLS support centering around mqtt that seems to have made some progress, but not sure where this stands or how much this applies here.
There also seems to be a HTTPS enabled webserver in ESP-IDF, but I am frankly not sure how relevant this is to enable this feature in ESPHome.

Please describe your use case for this integration and alternatives you've tried:

There seem two major use-cases for HTTPS support. Both of them are currently blocked.

1. Many modern web features like Service Worker, Progressive Web Apps (PWA), etc require the site to be served from a secure origin and as such to be served via HTTPS. To integrate such features into the ESP device website requires the website to be served through a secure connection.
2. Sending HTTP (note, not HTTPS) requests from a website being served through HTTPS violates modern browsers Mixed Content policies and is actively being blocked by the browser, making any requests to the ESP device webserver coming from HTTPS enabled websites (like a PWA hosted on a public HTTPS website) impossible.

Additional context
I've been prototyping a PWA to interact with the ESPHome device web server using the web server API (both REST and Server Sent Events) and it's practically impossible to deploy this PWA on device or on a publicly accessible domain due to the lack of SSL/HTTPS.

Note that SSL/TLS support is sadly not going to fix all problems either though. In particular, SSL/HTTPS is tightly coupled to the ability to publicly resolve domain names to allow common CAs issue valid certificates. Since locally hosted services typically do not have publicly resolvable domain names, the ability to issue valid certificates through a common CA will be severely limited to some special use-cases and self-signed certificates will be the more common use-case for such locally hosted services -- including a HTTPS enabled web server in ESPHome. While these technically enable such use-cases, the usability and user friendliness would still be lacking.

Ultimately I think to enable the web platform to be used in the context of locally hosted services like restful IOT devices we will need some meaningful changes to the web platform itself. For example (user confirmed) exceptions to the secure origin or mixed content policies for local ip addresses. But this is obviously a broader discussion and tougher nut to crack.

[nagyrobi]

Having a full blown HTTPS server on an ESPHome node is a bit of overkill, as the webserver component itself as it is now is already a monster. Not to speak about what a security risk is, to have such a node exposed directly to the public internet as a server.

You could use a reverse proxy in front of it, which would be beneficial in many more aspects:

• not only adds the HTTPS layer
• but also allows configuring multiple nodes behind the same proxy using the same port, only different fqdns
• adds a security layer on top of the devices

[DanielBaulig]

Thanks @nagyrobi!

I'd just like to clarify a couple points since I'm not sure I expressed myself well in my opening post on these:

• I am "only" asking for SSL/TLS support in the web server component, i.e. being able to accept and serve HTTPS connections. No other additional features outside of those currently supported in the web server component (i.e. serving of a single, client rendered static site, a ServerSent Events endpoint and a few REST routes). I am clarifying this since I am not sure how you define "fully blown" and if there was a misunderstanding.
• How do you define overkill? From some of the research I've done it appears this is technically possible within the hardware limitations of an ESP32 (and as the above linked ESP-IDF code demonstrates has even been practically done before). As I've outlined above it would enable us to use some very powerful and useful web features, like PWA and Service Worker. Are there still reasons to not considers this, like an already large web server component and code base? Sure. Is the feature itself impossible, unnecessary or not useful? I would argue otherwise.
• In terms of the security implications: it is important to clarify that I have no intention to or am suggesting that anyone should be making the web server accessible from the public internet. Instead I want to make certain modern and powerful web features available inside the browser displaying the ESP32 website which are only available on websites being served via HTTPS, (e.g. Service Worker). Note that these are web platform features, i.e. they are provided by the browser, and live and execute in the browser. The web server, other than serving the site via HTTPS, does not have to be aware of them or needs to provide any specific features or support them. There are also other limitations to web interoperability imposed on sites that are not being served via HTTPS. The fact that a site is only being served locally is (sadly) irrelevant to this and something I've brought up with the W3C web standards body at the same time.
• I mentioned "[deploying this PWA] on a publicly accessible domain" and recognize that this could be misunderstood as making the ESPHome device accessible to the public internet directly. I want to clarify that this is not the use-case or proposal. Instead, the option that I am outlining here is to deploy a website (off device) on a publicly accessible domain that is being served via HTTPS to enable modern web features like Service Worker and to then just communicate locally with the ESPHome device. However, due to aforementioned mixed content security policy, web sites served via HTTPS can only communicate in a very restricted manner with web sites being served from non secure origins. So even in a scenario in which a website is served on a secure origin using an off device web server, as long as the device web server itself isn't made available via a secure connection, the web site will not be able to communicate with the device web server.
• Lastly, outside of the fact that I do not have the intention or desire to make any devices available to the public internet, setting up a reverse proxy in front of the ESPHome device defeats the purpose of my undertaking. As I mentioned above I am prototyping a Progressive Web App. The point here is to provide a low friction, open and user-friendly interface to easily and quickly deploy, configure and control ESPHome devices without the need for any additional local infrastructure. No app, no ESPHome installation, no Home Assistant. Nothing other than a (pre-flashed) ESPHome device and a URL. Web technologies (Service Worker, PWA, Bluetooth, USB/Serial, etc) and projects based on these web technologies (like Improv, ESPHome Web, ESPHome Web Server) make this absolutely doable. But it would require SSL/TLS/HTTPS support on the ESPHome web server.

Hope this clarifies some of my points and helps outline why SSL/TLS support would be useful in general and necessary for some specific web features to be available.

If it is not obvious, I'd also be happy to elaborate how access to these features (PWA, Service Worker, No Mixed Content) will benefit the ESPHome ecosystem as a whole and will allow us to improve especially the creator / third party vendor use-case and to a degree that deploying, configuring and controlling these devices becomes virtually frictionless for both creators and users.

[nagyrobi]

• How do you define overkill?

Well, simply as I stated above, the webserver component itself as it is now is already too big. Yes, it is technically possible to run it within the hardware limitations of an ESP32 as you say, but at what cost? The simple webserver itself is very resource hungry, you already are on the memory edge just by adding a few basic components along with the webserver.

What could you implement in practice? An https webserver and maybe a couple of switches may live along... But as soon as you add a ModBus device or maybe a PID thermostat, not to speak of any of the Bluetooth-related sensors, you'd run out of RAM pretty quickly.

I'm not against it, it's just not worth the effort imho as the practical usage along with many components would be limited.

Https stuff should be offloaded from the ESP imho.

Or use a RPI4 instead...

[DanielBaulig]

These are all very valid points. I'm aware that the web server is taking up a considerable amount of memory (I think the documentation mentions it). I do not have an understanding of how much exactly and also not by how much that would increase by bundling SSL/TLS. Obviously TLS/SSL could be a optional dependency that wouldn't be included in the build unless explicitly enabled.

Will this enable all thinkable use-cases? Of course not. But would it enable some limited use-case? Certainly, yes. If it's worth actually doing however probably depends more on the question of which use-cases people are actually interested in deploying rather then the complexity of the use-case itself. I don't have any good insight into this. I can speak a little more to my own use-case, however.

My particular use-case involves an ESP32 running ESPHome, including the Bluetooth stack, to communicate with BLE enabled studio light, a few binary sensors and an input select to support a dynamic BLE client configuration flow, the light component and web server for control over WIFI. Basically it's meant as a bridge between a particular Bluetooth enabled studio light and the WIFI, for which I am building a web based interface for provisioning, configuration and control of the light/ESP32. From a hardware perspective the ESP32 is part of a small package that can be easily and semi permanently mounted to the particular bluetooth enabled studio light.

Is there a lot going on here? Yes, probably. Is the ESP32 memory exhausted? ROM is getting pretty packed, but at least according to the flash tool RAM still has space to spare. I recognize that the flashing tool probably does not report any dynamically allocated RAM and the actual runtime utilization might be very different. But notwithstanding the SSL/TLS limitation, this is all working so far.

My intention was to build a generic, open-source interface that would allow anyone to control any (pre-flashed) ESPHome device running the web server component to be deployed (with Improv), configured and controlled (using the web server API). It would be a single, web based app that users can easily add their local ESPHome devices to, connect and control them without the need for any additional infrastructure (like Home Assistant or MQTT servers). This generic web app would be built on top of an abstraction layer that could be used by makers and creators to easily deploy a specialized and/or branded version of this PWA for their specific ESPHome devices.

The goal is to allow provisioning, configuration and control of custom creator/maker devices in user home networks without the need of any additional infrastructure being present (like reverse proxies, Home Assistant installations, etc).

I believe this would be a significant boon to the entire ESPHome ecosystem and would enable deployment of (pre-flashed) ESPHome powered devices developed and created by third party makers/creators without the need of specific technical knowledge or infrastructure with their customers / users. This seems to be a space that the ESPHome project is interested in (given the investments into template configurations, Improv and ESPHome Web Tools) and that with projects like e.g. the Everything Presence One seems to have found some broader public and end-user interest, too. Of course, as I am sure you are aware, there are many, many more examples of ESPHome based devices and products that might ultimately benefit from such an interface / API.

I'm not sure if that explanation makes sense, but I hope it sheds some light on why I think this effort might still be worth it.

All of that said, I also recognize that maybe the better path forward here is to convince the right people at W3C to implement some local network specific and user opt-in exceptions into secure origin policies that would enable this specific use-case without having to deploy SSL/TLS to small 320kb RAM MCUs.

[DanielBaulig]

I think there's a solution to the original problem: Private Network Access, in particular the Permission Prompt.

A lengthy but in the end productive discussion on this proposal's Github issue tracker lead to that solution.

There's an upcoming Chrome origin trial for this new API which not only relaxes the mixed content policies, but also other new web security policies which impede private network access from publicly hosted web applications.

Instead of a full SSL implementation, correctly handling the relevant HTTP preflight requests will now be sufficient to enable a web app to connect to the ESPHome devices. I opened up a new issue for this request. I will not close this feature request, since I do still believe having (optional) TLS support in the web server component might be useful. I won't object to closing this issue as wontfix either though :)