docs: (WIP) Self-hosted realtime docs

docs/docs/advanced-use/real-time-flags.md

@@ -12,8 +12,8 @@ You can achieve this by subscribing to real-time flag updates.
 
 Real-time flag updates require an Enterprise subscription.
 
-Real-time flag updates are only available on the public SaaS Flagsmith instance. Self-hosted and private cloud Flagsmith
-installations do not support real-time flag updates.
+If you are self-hosting Flagsmith, real-time flag updates require
+[additional infrastructure](/deployment/hosting/real-time).
 
 ## Setup
 

docs/docs/deployment/hosting/real-time/benchmark.js

@@ -0,0 +1,47 @@
+// Stress test for the Flagsmith SSE service
+// https://docs.flagsmith.com/deployment/hosting/real-time
+
+import { sleep } from 'k6';
+import http from 'k6/http';
+
+export const options = {
+    discardResponseBodies: true,
+    scenarios: {
+        // Gradually ramp up to 20k concurrent subscribers for the same environment
+        subscribe: {
+            exec: 'subscribers',
+            executor: 'ramping-vus',
+            stages: [{ duration: '1m', target: 10000 }],
+        },
+        // Publish an update to the same environment every 10s
+        publish: {
+            duration: '1m',
+            exec: 'publish',
+            executor: 'constant-vus',
+            vus: 1,
+        },
+    },
+};
+
+const env = 'load_test';
+export function subscribers() {
+    http.get(`http://localhost:8088/sse/environments/${env}/queue-change`, '', {
+        headers: {
+            Accept: 'event/stream',
+        },
+        timeout: '3m',
+    });
+}
+
+export function publish() {
+    const body = JSON.stringify({
+        updated_at: new Date().toISOString(),
+    });
+    http.post(`http://localhost:8088/sse/environments/${env}/queue-change`, body, {
+        headers: {
+            'Content-Type': 'application/json',
+            Authorization: 'Token ' + __ENV.SSE_AUTHENTICATION_TOKEN,
+        },
+    });
+    sleep(10);
+}

docs/docs/deployment/hosting/real-time/deployment.md

@@ -0,0 +1,129 @@
+---
+title: Deployment guide
+sidebar_label: Deployment
+sidebar_position: 30
+---
+
+## Redis
+
+First, deploy a Redis-compatible store, such as [Valkey](https://valkey.io/). Many managed options for Redis-compatible
+stores are offered by different cloud providers. Some examples:
+
+- [Amazon ElastiCache (AWS)](https://aws.amazon.com/elasticache/features/)
+- [Memorystore for Valkey (GCP)](https://cloud.google.com/memorystore/docs/valkey/product-overview)
+- [Azure cache for Redis](https://azure.microsoft.com/en-us/products/cache)
+
+Clustering Redis is recommended for high availability, but not required. All
+[current Redis versions](https://redis.io/docs/latest/operate/rs/installing-upgrading/product-lifecycle/) are supported.
+
+### Authentication
+
+Redis must not require a password for the default user. Options for authentication will be added in the future.
+
+## SSE service
+
+Run the `flagsmith/sse` image, setting these environment variables:
+
+| Variable name                   | Description                                                                                                                     | Default      |
+| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------ |
+| `SSE_AUTHENTICATION_TOKEN`      | Shared secret for authentication on the `/queue-change` endpoint                                                                | **Required** |
+| `REDIS_HOST`                    | Hostname of the Redis load balancer                                                                                             | `localhost`  |
+| `REDIS_PORT`                    | Port number to use when connecting to Redis                                                                                     | 6379         |
+| `REDIS_SCAN_COUNT`              | Number of Redis keys to [`SCAN`](https://redis.io/docs/latest/commands/scan/) at once when updating the in-memory cache         | 500          |
+| `CACHE_UPDATE_INTERVAL_SECONDS` | How long (in seconds) to wait between each `SCAN` to update the in-memory cache                                                 | 1            |
+| `USE_CLUSTER_MODE`              | Whether to connect to Redis with [Cluster Mode](https://redis.io/docs/latest/operate/oss_and_stack/management/scaling/) enabled | `False`      |
+| `REDIS_USE_SSL`                 | Whether to connect to Redis with SSL                                                                                            | `False`      |
+| `MAX_STREAM_AGE`                | How long (in seconds) to keep SSE connections alive for. If negative, connections are kept open indefinitely                    | 30           |
+| `STREAM_DELAY`                  | How long (in seconds) to wait before checking the internal cache for updates                                                    | 1            |
+
+The SSE service will expose its endpoints to `0.0.0.0:8000`.
+
+## API and task processor
+
+The Flagsmith API and task processor need to know about the SSE service. On both the API and task processor, set these
+environment variables:
+
+- `SSE_SERVER_BASE_URL` points to the SSE service load balancer. For example: `http://my-sse-service:8000`
+- `SSE_AUTHENTICATION_TOKEN` can be set to any non-empty string, as long as the SSE service and task processor share the
+  same value.
+
+## Flagsmith configuration
+
+Make sure the Flagsmith projects you are updating have real-time updates enabled. If not, no tasks will be queued when
+its environments are updated.
+
+Lastly, client applications should set their Flagsmith SDK's realtime endpoint URL to the load balancer for the SSE
+service.
+
+## Example: Docker Compose
+
+The following Docker Compose file defines a simple Flagsmith deployment. The highlighted lines are required to support
+real-time flag updates.
+
+```yaml title="compose.yaml"
+services:
+ # highlight-start
+ valkey:
+  image: valkey/valkey:latest
+  # highlight-end
+
+  # highlight-start
+ sse:
+  image: flagsmith/sse:3.3.0
+  environment:
+   SSE_AUTHENTICATION_TOKEN: changeme
+   REDIS_HOST: valkey
+  depends_on:
+   - valkey
+ # highlight-end
+
+ flagsmith:
+  # highlight-start
+  image: flagsmith/flagsmith-private-cloud:latest
+  # highlight-end
+  environment:
+   # highlight-start
+   SSE_AUTHENTICATION_TOKEN: changeme
+   SSE_SERVER_BASE_URL: 'http://sse:8000'
+   # highlight-end
+   DATABASE_URL: postgresql://postgres:password@postgres:5432/flagsmith
+   USE_POSTGRES_FOR_ANALYTICS: 'true'
+   ENVIRONMENT: production
+   DJANGO_ALLOWED_HOSTS: '*'
+   ALLOW_ADMIN_INITIATION_VIA_CLI: 'true'
+   FLAGSMITH_DOMAIN: 'localhost:8000'
+   DJANGO_SECRET_KEY: secret
+   ENABLE_ADMIN_ACCESS_USER_PASS: 'true'
+   TASK_RUN_METHOD: TASK_PROCESSOR
+  ports:
+   - '8000:8000'
+  depends_on:
+   - postgres
+
+ # The flagsmith_processor service is only needed if TASK_RUN_METHOD set to TASK_PROCESSOR
+ # in the application environment
+ flagsmith_processor:
+  image: flagsmith/flagsmith:latest
+  environment:
+   # highlight-start
+   SSE_AUTHENTICATION_TOKEN: changeme
+   SSE_SERVER_BASE_URL: 'http://sse:8000'
+   # highlight-end
+   DATABASE_URL: postgresql://postgres:password@postgres:5432/flagsmith
+   USE_POSTGRES_FOR_ANALYTICS: 'true'
+  depends_on:
+   - flagsmith
+  command: run-task-processor
+
+ postgres:
+  image: postgres:15.5-alpine
+  environment:
+   POSTGRES_PASSWORD: password
+   POSTGRES_DB: flagsmith
+  container_name: flagsmith_postgres
+  volumes:
+   - pgdata:/var/lib/postgresql/data
+
+volumes:
+ pgdata:
+```

docs/docs/deployment/hosting/real-time/index.md

@@ -0,0 +1,106 @@
+---
+title: Self-hosting real-time updates
+sidebar_label: Real-time flags
+sidebar_position: 30
+---
+
+If you are self-hosting Flagsmith, using [real-time flag updates](/advanced-use/real-time-flags.md) requires you to
+deploy additional infrastructure. Start here for an overview of how this infrastructure delivers this capability in your
+self-hosted Flagsmith.
+
+You might also be interested in:
+
+- [How to deploy the infrastructure for real-time flag updates](deployment)
+- [How to operate a real-time flag updates system](operations)
+
+## Prerequisites
+
+Real-time flag updates require an Enterprise subscription.
+
+We assume you already have the [Flagsmith API](/deployment/hosting/locally-api.md) running on your infrastructure.
+
+## Infrastructure
+
+The **real-time flag updates system** is supported by additional infrastructure that your existing Flagsmith deployment
+integrates with:
+
+- **Server-sent events (SSE) service containers**, running the private
+  [`flagsmith/sse`](https://hub.docker.com/repository/docker/flagsmith/sse) Docker image. These serve the real-time
+  endpoint that Flagsmith clients can connect to, and receive updates from the
+  [task processor](/deployment/configuration/task-processor).
+- A Redis-compatible key-value store that the [task processor](/deployment/configuration/task-processor.md) and SSE
+  service can connect to.
+
+This diagram shows how all the components initiate connections to each other:
+
+```mermaid
+graph LR
+    api[Task processor]
+    sse[SSE service]
+    nats[Redis]
+    client1[Client]
+    client2[Client]
+
+    api -->|Publish| sse
+    sse -->|Fetch| nats
+    client1 -->|Connect| sse
+    client2 -->|Connect| sse
+```
+
+## How it works
+
+Real-time flags use a fully distributed and horizontally scalable architecture. Any SSE service instance can respond to
+any client's request. All components can be scaled out or in as needed. Stateful or sticky sessions are not used.
+
+The following sequence diagram shows how a Flagsmith client application connects to the real-time updates stream and
+receives messages when the environment is updated.
+
+```mermaid
+sequenceDiagram
+    loop Every second, background
+      SSE service->>Redis: Fetch update timestamps for all environments
+      SSE service-->SSE service: Update in-memory cache
+    end
+    Client->>SSE service: Subscribe to environment updates
+    SSE service->>Client: Send latest update timestamp
+    loop Every second, per subscriber
+      SSE service-->SSE service: Read latest update from in-memory cache
+      SSE service->>Client: Send latest update timestamp, if changed
+      Client-->Client: Store latest update timestamp
+    end
+    Task processor->>SSE service: Notify environment updated
+    SSE service->>Redis: Store latest update timestamp
+```
+
+### SSE service
+
+The **server-sent events (SSE)** service provides the real-time API endpoints that Flagsmith clients connect to. Clients
+connect to any service instance and hold an HTTP connection open for as long as they want to receive updates over SSE.
+
+This service also accepts HTTP requests from the Flagsmith task processor to get notified of environment updates. Redis
+is used as the storage layer to distribute these updates to clients.
+
+[HTTP/2 is recommended](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events)
+for client connections, especially if the clients are web browsers.
+
+### Redis
+
+Redis stores a key for each environment containing a timestamp of when it was last updated. SSE service instances will
+periodically fetch all environment's keys and cache them in memory. This cache is used to publish update notifications
+to connected clients.
+
+If Redis or its stored data are unavailable, clients will not be able to receive updates.
+
+## How to use it
+
+Refer to the [deployment guide](deployment) for instructions on setting up the required infrastructure.
+
+The `flagsmith/sse` service provides the following HTTP endpoints:
+
+| Method | Route                                          | Called by           | Description                                                    | Authentication                   |
+| ------ | ---------------------------------------------- | ------------------- | -------------------------------------------------------------- | -------------------------------- |
+| GET    | `/sse/environments/{environment}/stream`       | Client applications | Subscribe to an SSE stream for the given environment.          | None                             |
+| POST   | `/sse/environments/{environment}/queue-change` | Task processor      | Notify the SSE service that the given environment was updated. | `Token SSE_AUTHENTICATION_TOKEN` |
+
+The stream protocol is described in the
+[documentation for real-time flag updates](/advanced-use/real-time-flags#implementation-details).