From f3e45b98752e9927ec42e60a44d6dfaa6ba7ecac Mon Sep 17 00:00:00 2001 From: Karl-Aksel Puulmann Date: Thu, 3 Mar 2022 11:30:10 +0200 Subject: [PATCH 01/13] Document some more env vars --- contents/docs/self-host/configure/environment-variables.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/contents/docs/self-host/configure/environment-variables.md b/contents/docs/self-host/configure/environment-variables.md index c40476a130fe..1679b8d144e2 100644 --- a/contents/docs/self-host/configure/environment-variables.md +++ b/contents/docs/self-host/configure/environment-variables.md @@ -33,11 +33,14 @@ Some variables here are default Django variables. This [Django Docs page](https: | `DATABASE_URL`| [Database URL](https://github.com/jacobian/dj-database-url#url-schema) pointing to your PostgreSQL instance. | `postgres://localhost:5432/posthog` if PostHog is running in DEBUG or TEST mode, must be specified otherwise. | `DEBUG_QUERIES`| Whether debugging queries (ClickHouse) is enabled in the Command Palette.| `False` | `DEBUG` | Determines if PostHog should run in [DEBUG mode](https://docs.djangoproject.com/en/2.2/ref/settings/#std:setting-DEBUG). You can set this to a truthy value when developing, but disable this in production! | `False` | +| `CLICKHOUSE_DISABLE_EXTERNAL_SCHEMAS` | If set, disables using ProtoBuf schemas for kafka communication. Needs to be set when using an external ClickHouse service provider during initial deploy. | `False` | `DISABLE_PAID_FEATURE_SHOWCASING`| Whether any showcasing of a paid feature should be disabled. Useful if running a free open source version of PostHog and are not interested in premium functionality. | `False` | `DISABLE_SECURE_SSL_REDIRECT` | Disables automatic redirect from port 80 (HTTP) to port 443 (HTTPS). | `False` | `GITHUB_TOKEN`| GitHub personal access token, used to prevent rate limiting when using plugins and to allow installation of plugins from private repos | `None` | `GITLAB_TOKEN`| GitLab personal access token, used to prevent rate limiting when using plugins and to allow installation of plugins from private repos | `None` | `JS_URL` | URL used by Webpack for loading external resources like images and files. | `http://localhost:8234` if PostHog is running in DEBUG mode, must be specified otherwise. +| `KAFKA_URL` | Address used by the application to contact kafka | `kafka://kafka` +| `KAFKA_URL_FOR_CLICKHOUSE` | Address used by ClickHouse to read from kafka. Falls back to `KAFKA_URL` | `None` | `MATERIALIZE_COLUMNS_ANALYSIS_PERIOD_HOURS` | Diagnostic for what columns to materialize | `168` | `MATERIALIZE_COLUMNS_BACKFILL_PERIOD_DAYS` | How far back backfill materialized columns | `90` | `MATERIALIZE_COLUMNS_MAX_AT_ONCE` | How many columns to materialize at once | `10` From 1603303a07f69435d58f57e6c6480e6460b52005 Mon Sep 17 00:00:00 2001 From: Karl-Aksel Puulmann Date: Thu, 3 Mar 2022 11:31:35 +0200 Subject: [PATCH 02/13] Add initial instructions on how to deploy Altinity Cloud --- .../configure/using-altinity-cloud.md | 137 ++++++++++++++++++ .../docs/self-host/deploy/configuration.md | 5 + src/sidebars/sidebars.json | 4 + 3 files changed, 146 insertions(+) create mode 100644 contents/docs/self-host/configure/using-altinity-cloud.md diff --git a/contents/docs/self-host/configure/using-altinity-cloud.md b/contents/docs/self-host/configure/using-altinity-cloud.md new file mode 100644 index 000000000000..542462b94b49 --- /dev/null +++ b/contents/docs/self-host/configure/using-altinity-cloud.md @@ -0,0 +1,137 @@ +--- +title: Deploying ClickHouse using Altinity.Cloud +sidebar: Docs +showTitle: true +--- + +This document outlines how to deploy PostHog using Altinity Cloud ClickHouse clusters. + +## Prerequisites + +- Altinity.Cloud ClickHouse cluster: + - Minimum ClickHouse version: 21.8.13 + - Single shard and no data replication + - No dashes (`-`) in cluster name +- PostHog helm chart version >= 16.1.1 +- PostHog version >= 1.33.0 + +## Deployment instructions + +PostHog uses Kafka to send data from the app to ClickHouse. For that reason, Kafka needs to be accessible for ClickHouse during deployment. + +### Deploying using external Kafka + +```yaml +env: + - name: CLICKHOUSE_DISABLE_EXTERNAL_SCHEMAS + value: "1" + +kafka: + enabled: false + +externalKafka: + brokers: + - "broker-1.posthog.kafka.us-east-1.amazonaws.com:9094" + - "broker-2.posthog.kafka.us-east-1.amazonaws.com:9094" + - "broker-3.posthog.kafka.us-east-1.amazonaws.com:9094" + +clickhouse: + enabled: false + +externalClickhouse: + host: "somecluster.demo.altinity.cloud" + user: "admin" + password: "password" + cluster: "clustername" + secure: true +``` + +Read more on configuring external Kafka in the chart [here](https://posthog.com/docs/self-host/deploy/configuration#kafka) + +### Using internal Kafka + +Deployment using Kafka managed by Posthog Helm chart requires three steps: + +1. [Deploy helm](/docs/self-host) initially with the following values.yaml: + +```yaml +kafka: + enabled: true + externalAccess: + enabled: true + service: + type: LoadBalancer + ports: + external: 9094 + autoDiscovery: + enabled: true + serviceAccount: + create: true + rbac: + create: true + + +clickhouse: + enabled: false + +redis: + enabled: false + +postgresql: + enabled: false + +pgbouncer: + enabled: false + +plugins: + enabled: false + +worker: + enabled: false + +web: + enabled: false + +events: + enabled: false + +migrate: + enabled: false +``` + +2. Get the external kafka IP via `kubectl get svc -n posthog | grep kafka-0-external` + +3. [Deploy posthog using helm](/docs/self-host) with new values.yaml (fill in placeholder values) + +```yaml +env: + - name: KAFKA_URL_FOR_CLICKHOUSE + value: "kafka://KAFKA_IP:9094" + - name: CLICKHOUSE_DISABLE_EXTERNAL_SCHEMAS + value: "1" + +clickhouse: + enabled: false + +externalClickhouse: + host: "somecluster.demo.altinity.cloud" + user: "admin" + password: "password" + cluster: "clustername" + secure: true + +kafka: + enabled: true + externalAccess: + enabled: true + service: + type: LoadBalancer + ports: + external: 9094 + autoDiscovery: + enabled: true + serviceAccount: + create: true + rbac: + create: true +``` diff --git a/contents/docs/self-host/deploy/configuration.md b/contents/docs/self-host/deploy/configuration.md index e294c2155098..a3afadb130d9 100644 --- a/contents/docs/self-host/deploy/configuration.md +++ b/contents/docs/self-host/deploy/configuration.md @@ -207,6 +207,11 @@ ClickHouse is the datastore system that does the bulk of heavy lifting with rega By default, ClickHouse is installed as a part of the chart, powered by [clickhouse-operator](https://github.com/Altinity/clickhouse-operator/). We are currently working to add the possibility to use an external ClickHouse service (see [issue #279](https://github.com/PostHog/charts-clickhouse/issues/279) for more info). +#### Use an external service +To use an external PostgreSQL service, please set `postgresql.enabled` to `false` and then configure the `externalPostgresql` values. + +Read more on how to deploy using Altinity Cloud [here](/docs/self-host/configure/using-altinity-cloud). + #### Custom settings It's possible to pass custom settings to ClickHouse. This might be needed to e.g. set query time limits or increase max memory usable by clickhouse. diff --git a/src/sidebars/sidebars.json b/src/sidebars/sidebars.json index 17b1e59c66b7..0a0d18180721 100644 --- a/src/sidebars/sidebars.json +++ b/src/sidebars/sidebars.json @@ -536,6 +536,10 @@ { "name": "Helm chart configuration", "url": "/docs/self-host/deploy/configuration" + }, + { + "name": "Deploying ClickHouse using Altinity.Cloud", + "url": "/docs/self-host/configure/using-altinity-cloud" } ] }, From e454c0ab62d9f0fc17d6df0c8997a6a34bff32c2 Mon Sep 17 00:00:00 2001 From: Karl-Aksel Puulmann Date: Thu, 3 Mar 2022 11:35:53 +0200 Subject: [PATCH 03/13] Fix typos --- contents/docs/self-host/deploy/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contents/docs/self-host/deploy/configuration.md b/contents/docs/self-host/deploy/configuration.md index a3afadb130d9..2a689498b152 100644 --- a/contents/docs/self-host/deploy/configuration.md +++ b/contents/docs/self-host/deploy/configuration.md @@ -208,7 +208,7 @@ By default, ClickHouse is installed as a part of the chart, powered by [clickhou #### Use an external service -To use an external PostgreSQL service, please set `postgresql.enabled` to `false` and then configure the `externalPostgresql` values. +To use an external ClickHouse service, please set `clickhouse.enabled` to `false` and then configure the `externalClickhouse` values. Read more on how to deploy using Altinity Cloud [here](/docs/self-host/configure/using-altinity-cloud). From 7de33266059c5d2d01033b6323d24dac496890af Mon Sep 17 00:00:00 2001 From: Karl-Aksel Puulmann Date: Thu, 3 Mar 2022 12:02:21 +0200 Subject: [PATCH 04/13] Update marketplace docs --- contents/marketplace/altinity.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/contents/marketplace/altinity.md b/contents/marketplace/altinity.md index ac3feba57a0a..a1b87c005b32 100644 --- a/contents/marketplace/altinity.md +++ b/contents/marketplace/altinity.md @@ -8,21 +8,21 @@ hideLastUpdated: true Altinity Logo -[Altinity](https://altinity.com) helps enterprises deliver real-time analytics based on ClickHouse anywhere and for any business purpose. Our offerings cover everything needed from project inception to prodution operation. +[Altinity](https://altinity.com) helps enterprises deliver real-time analytics based on ClickHouse anywhere and for any business purpose. Our offerings cover everything needed from project inception to prodution operation. 1. Altinity.Cloud platform offering fully supported and managed ClickHouse clusters in AWS and GCP 2. 24/7 enterprise support for ClickHouse in any environment 3. Training for analytic developers and administrators 4. Altinity Stable builds for ClickHouse -Altinity customers range from startups to Fortune 10 enterprises. +Altinity customers range from startups to Fortune 10 enterprises. ## Services offered ### Support -- Coming soon (likely from PostHog 1.33.0): Altinity.Cloud ClickHouse backends for PostHog users who prefer to use ClickHouse as a hands-off service - get in touch now if you are interested - Enterprise support for users who prefer to operate ClickHouse in self-managed environments +- Altinity.Cloud ClickHouse backends for PostHog users who prefer to use ClickHouse as a hands-off service. Read more [here](/docs/self-host/configure/using-altinity-cloud). ## Contact -[Speak to Altinity](mailto:marketplace+altinity@posthog.com) \ No newline at end of file +[Speak to Altinity](mailto:marketplace+altinity@posthog.com) From bc239c9de680855bcb62f9254aaf0f12437df797 Mon Sep 17 00:00:00 2001 From: Karl-Aksel Puulmann Date: Thu, 3 Mar 2022 12:28:00 +0200 Subject: [PATCH 05/13] Update contents/docs/self-host/configure/using-altinity-cloud.md Co-authored-by: Joe Martin <84011561+joethreepwood@users.noreply.github.com> --- contents/docs/self-host/configure/using-altinity-cloud.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contents/docs/self-host/configure/using-altinity-cloud.md b/contents/docs/self-host/configure/using-altinity-cloud.md index 542462b94b49..0f726f6d9483 100644 --- a/contents/docs/self-host/configure/using-altinity-cloud.md +++ b/contents/docs/self-host/configure/using-altinity-cloud.md @@ -17,7 +17,7 @@ This document outlines how to deploy PostHog using Altinity Cloud ClickHouse clu ## Deployment instructions -PostHog uses Kafka to send data from the app to ClickHouse. For that reason, Kafka needs to be accessible for ClickHouse during deployment. +PostHog uses Kafka to send data from the app to ClickHouse. For that reason, Kafka needs to be accessible to ClickHouse during deployment. ### Deploying using external Kafka From bde891b055bb8ff7e3ccfa12a11e1305e3155ada Mon Sep 17 00:00:00 2001 From: Karl-Aksel Puulmann Date: Thu, 3 Mar 2022 12:28:06 +0200 Subject: [PATCH 06/13] Update contents/docs/self-host/configure/using-altinity-cloud.md Co-authored-by: Joe Martin <84011561+joethreepwood@users.noreply.github.com> --- contents/docs/self-host/configure/using-altinity-cloud.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contents/docs/self-host/configure/using-altinity-cloud.md b/contents/docs/self-host/configure/using-altinity-cloud.md index 0f726f6d9483..6a8109df661c 100644 --- a/contents/docs/self-host/configure/using-altinity-cloud.md +++ b/contents/docs/self-host/configure/using-altinity-cloud.md @@ -46,7 +46,7 @@ externalClickhouse: secure: true ``` -Read more on configuring external Kafka in the chart [here](https://posthog.com/docs/self-host/deploy/configuration#kafka) +Read more about how to configure external Kafka in the chart [in our deployment documentation](https://posthog.com/docs/self-host/deploy/configuration#kafka). ### Using internal Kafka From a171efa878a8daa20554084df69d539d704a3801 Mon Sep 17 00:00:00 2001 From: Karl-Aksel Puulmann Date: Thu, 3 Mar 2022 12:28:15 +0200 Subject: [PATCH 07/13] Update contents/docs/self-host/configure/using-altinity-cloud.md Co-authored-by: Joe Martin <84011561+joethreepwood@users.noreply.github.com> --- contents/docs/self-host/configure/using-altinity-cloud.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contents/docs/self-host/configure/using-altinity-cloud.md b/contents/docs/self-host/configure/using-altinity-cloud.md index 6a8109df661c..638a58e1b2de 100644 --- a/contents/docs/self-host/configure/using-altinity-cloud.md +++ b/contents/docs/self-host/configure/using-altinity-cloud.md @@ -50,7 +50,7 @@ Read more about how to configure external Kafka in the chart [in our deployment ### Using internal Kafka -Deployment using Kafka managed by Posthog Helm chart requires three steps: +To deploy using a version of Kafka managed by the PostHog Helm chart, follow these three steps: 1. [Deploy helm](/docs/self-host) initially with the following values.yaml: From dbc2f4b3d02b41964258631c6a65a04eb450a4d1 Mon Sep 17 00:00:00 2001 From: Karl-Aksel Puulmann Date: Thu, 3 Mar 2022 12:30:50 +0200 Subject: [PATCH 08/13] Update contents/docs/self-host/configure/using-altinity-cloud.md Co-authored-by: Joe Martin <84011561+joethreepwood@users.noreply.github.com> --- contents/docs/self-host/configure/using-altinity-cloud.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contents/docs/self-host/configure/using-altinity-cloud.md b/contents/docs/self-host/configure/using-altinity-cloud.md index 638a58e1b2de..a5fbe86e04b0 100644 --- a/contents/docs/self-host/configure/using-altinity-cloud.md +++ b/contents/docs/self-host/configure/using-altinity-cloud.md @@ -52,7 +52,7 @@ Read more about how to configure external Kafka in the chart [in our deployment To deploy using a version of Kafka managed by the PostHog Helm chart, follow these three steps: -1. [Deploy helm](/docs/self-host) initially with the following values.yaml: +1. [Deploy your Helm chart](/docs/self-host) initially with the following values.yaml: ```yaml kafka: From 8f2cc46d910aca300e0b5f4ab98a1ab6cde4be45 Mon Sep 17 00:00:00 2001 From: Karl-Aksel Puulmann Date: Thu, 3 Mar 2022 12:30:56 +0200 Subject: [PATCH 09/13] Update contents/docs/self-host/configure/using-altinity-cloud.md Co-authored-by: Joe Martin <84011561+joethreepwood@users.noreply.github.com> --- contents/docs/self-host/configure/using-altinity-cloud.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contents/docs/self-host/configure/using-altinity-cloud.md b/contents/docs/self-host/configure/using-altinity-cloud.md index a5fbe86e04b0..d2d743cb533f 100644 --- a/contents/docs/self-host/configure/using-altinity-cloud.md +++ b/contents/docs/self-host/configure/using-altinity-cloud.md @@ -99,7 +99,7 @@ migrate: enabled: false ``` -2. Get the external kafka IP via `kubectl get svc -n posthog | grep kafka-0-external` +2. Get the external Kafka IP via `kubectl get svc -n posthog | grep kafka-0-external` 3. [Deploy posthog using helm](/docs/self-host) with new values.yaml (fill in placeholder values) From 5ba43e49edc02ddf71c9f1fd04cb4bf5130d4c6e Mon Sep 17 00:00:00 2001 From: Karl-Aksel Puulmann Date: Thu, 3 Mar 2022 12:31:01 +0200 Subject: [PATCH 10/13] Update contents/docs/self-host/configure/using-altinity-cloud.md Co-authored-by: Joe Martin <84011561+joethreepwood@users.noreply.github.com> --- contents/docs/self-host/configure/using-altinity-cloud.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contents/docs/self-host/configure/using-altinity-cloud.md b/contents/docs/self-host/configure/using-altinity-cloud.md index d2d743cb533f..ffe969bb348e 100644 --- a/contents/docs/self-host/configure/using-altinity-cloud.md +++ b/contents/docs/self-host/configure/using-altinity-cloud.md @@ -101,7 +101,7 @@ migrate: 2. Get the external Kafka IP via `kubectl get svc -n posthog | grep kafka-0-external` -3. [Deploy posthog using helm](/docs/self-host) with new values.yaml (fill in placeholder values) +3. [Deploy PostHog using helm](/docs/self-host) with new values.yaml (fill in placeholder values) ```yaml env: From 12664616a60c65349c0bf9960fedbc10df412667 Mon Sep 17 00:00:00 2001 From: Joe Martin <84011561+joethreepwood@users.noreply.github.com> Date: Thu, 3 Mar 2022 10:33:23 +0000 Subject: [PATCH 11/13] Update contents/marketplace/altinity.md --- contents/marketplace/altinity.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contents/marketplace/altinity.md b/contents/marketplace/altinity.md index a1b87c005b32..943354202d33 100644 --- a/contents/marketplace/altinity.md +++ b/contents/marketplace/altinity.md @@ -8,7 +8,7 @@ hideLastUpdated: true Altinity Logo -[Altinity](https://altinity.com) helps enterprises deliver real-time analytics based on ClickHouse anywhere and for any business purpose. Our offerings cover everything needed from project inception to prodution operation. +[Altinity](https://altinity.com) helps enterprises deliver real-time analytics based on ClickHouse anywhere and for any business purpose. Altinity covers everything needed from project inception to production operation. 1. Altinity.Cloud platform offering fully supported and managed ClickHouse clusters in AWS and GCP 2. 24/7 enterprise support for ClickHouse in any environment From e8aa90b0ca9cd910f598837b2d3ce5685d39847b Mon Sep 17 00:00:00 2001 From: Joe Martin <84011561+joethreepwood@users.noreply.github.com> Date: Thu, 3 Mar 2022 10:33:28 +0000 Subject: [PATCH 12/13] Update contents/marketplace/altinity.md --- contents/marketplace/altinity.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contents/marketplace/altinity.md b/contents/marketplace/altinity.md index 943354202d33..9f1537eed431 100644 --- a/contents/marketplace/altinity.md +++ b/contents/marketplace/altinity.md @@ -21,7 +21,7 @@ Altinity customers range from startups to Fortune 10 enterprises. ### Support - Enterprise support for users who prefer to operate ClickHouse in self-managed environments -- Altinity.Cloud ClickHouse backends for PostHog users who prefer to use ClickHouse as a hands-off service. Read more [here](/docs/self-host/configure/using-altinity-cloud). +- Altinity.Cloud ClickHouse backends for PostHog users who prefer to use ClickHouse as a hands-off service. Find out more about using Altinity.Cloud to manage your self-hosted instance in [our deployment configuration docs.](/docs/self-host/configure/using-altinity-cloud). ## Contact From cbe9e4a6af5be0ab4cc6eab0c60a9752000b8408 Mon Sep 17 00:00:00 2001 From: Joe Martin <84011561+joethreepwood@users.noreply.github.com> Date: Thu, 3 Mar 2022 10:33:34 +0000 Subject: [PATCH 13/13] Update contents/docs/self-host/deploy/configuration.md --- contents/docs/self-host/deploy/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contents/docs/self-host/deploy/configuration.md b/contents/docs/self-host/deploy/configuration.md index 2a689498b152..38d3da229f3c 100644 --- a/contents/docs/self-host/deploy/configuration.md +++ b/contents/docs/self-host/deploy/configuration.md @@ -210,7 +210,7 @@ By default, ClickHouse is installed as a part of the chart, powered by [clickhou #### Use an external service To use an external ClickHouse service, please set `clickhouse.enabled` to `false` and then configure the `externalClickhouse` values. -Read more on how to deploy using Altinity Cloud [here](/docs/self-host/configure/using-altinity-cloud). +Find out how to deploy PostHog using Altinity Cloud [in our deployment configuration docs](/docs/self-host/configure/using-altinity-cloud). #### Custom settings