From 9346ae9dd3791e05b3a3a544dbffa9deba176113 Mon Sep 17 00:00:00 2001 From: Brett Hoerner Date: Tue, 16 Jan 2024 08:41:17 -0700 Subject: [PATCH] chore(plugin-server): README tweaks (#19785) --- plugin-server/README.md | 31 +++++++++++-------------------- 1 file changed, 11 insertions(+), 20 deletions(-) diff --git a/plugin-server/README.md b/plugin-server/README.md index 16b8fb740a776..6dff75c60702b 100644 --- a/plugin-server/README.md +++ b/plugin-server/README.md @@ -13,11 +13,18 @@ Let's get you developing the plugin server in no time: 1. Install dependencies and prepare for takeoff by running command `pnpm i`. -1. Start a development instance of [PostHog](/PostHog/posthog) - [instructions here](https://posthog.com/docs/developing-locally). After all, this is the _PostHog_ Plugin Server, and it works in conjuction with the main server. +1. Start a development instance of [PostHog](/PostHog/posthog) - [instructions + here](https://posthog.com/docs/developing-locally). After all, this is the _PostHog_ Plugin + Server, and it works in conjuction with the main server. -1. Make sure that the plugin server is configured correctly (see [Configuration](#Configuration)). The following settings need to be the same for the plugin server and the main server: `DATABASE_URL`, `REDIS_URL`, `KAFKA_HOSTS`, `CLICKHOUSE_HOST`, `CLICKHOUSE_DATABASE`, `CLICKHOUSE_USER`, and `CLICKHOUSE_PASSWORD`. Their default values should work just fine in local development though. +1. Make sure that the plugin server is configured correctly (see [Configuration](#Configuration)). + The following settings need to be the same for the plugin server and the main server: + `DATABASE_URL`, `REDIS_URL`, `KAFKA_HOSTS`, `CLICKHOUSE_HOST`, `CLICKHOUSE_DATABASE`, + `CLICKHOUSE_USER`, and `CLICKHOUSE_PASSWORD`. Their default values should work just fine in local + development though. -1. Start the plugin server in autoreload mode with `pnpm start:dev`, or in compiled mode with `pnpm build && pnpm start:dist`, and develop away! +1. Start the plugin server in autoreload mode with `pnpm start:dev`, or in compiled mode with `pnpm +build && pnpm start:dist`, and develop away! 1. Prepare for running tests with `pnpm setup:test`, which will run the necessary migrations. Run the tests themselves with `pnpm test:{1,2}`. @@ -160,7 +167,7 @@ Let's talk about the main thread first. This has: 1. `hub` – Handler of connections to required DBs and queues (ClickHouse, Kafka, Postgres, Redis), holds loaded plugins. Created via `hub.ts -> createHub`. Every thread has its own instance. -1. `piscina` – Manager of tasks delegated to threads. `makePiscina` creates the manager, while `createWorker` creates the worker threads. +1. `piscina` – This used to be a manager of tasks that were delegated to threads. It is now a shim over normal JS function calls that will be removed in the future. 1. `pluginScheduleControl` – Controller of scheduled jobs. Responsible for adding Piscina tasks for scheduled jobs, when the time comes. The schedule information makes it into the controller when plugin VMs are created. @@ -176,18 +183,6 @@ Let's talk about the main thread first. This has: 1. `mmdbServer` – TCP server, which works as an interface between the GeoIP MMDB data reader located in main thread memory and plugins ran in worker threads of the same plugin server instance. This way the GeoIP reader is only loaded in one thread and can be used in all. Additionally this mechanism ensures that `mmdbServer` is ready before ingestion is started (database downloaded from [http-mmdb](https://github.com/PostHog/http-mmdb) and read), and keeps the database up to date in the background. -### Worker threads - -This begins with `worker.ts` and `createWorker()`. - -`hub` is the same setup as in the main thread. - -New functions called here are: - -1. `setupPlugins` – Loads plugins and prepares them for lazy VM initialization. - -2. `createTaskRunner` – Creates a Piscina task runner that allows to operate on plugin VMs. - > Note: > An `organization_id` is tied to a _company_ and its _installed plugins_, a `team_id` is tied to a _project_ and its _plugin configs_ (enabled/disabled+extra config). @@ -215,7 +210,3 @@ We carry a node-rdkafka patch that adds cooperative rebalancing. To generate thi # in the plugin-server directory, target the temporary directory from the previous command pnpm patch-commit /private/var/folders/b7/bmmghlpx5qdd6gpyvmz1k1_m0000gn/T/6082767a6879b3b4e11182f944f5cca3 - -## Questions? - -### [Join our Slack community. 🦔](https://posthog.com/slack)