Skip to content

Commit

Permalink
chore(plugin-server): README tweaks (#19785)
Browse files Browse the repository at this point in the history
  • Loading branch information
bretthoerner authored Jan 16, 2024
1 parent 3bfd752 commit 9346ae9
Showing 1 changed file with 11 additions and 20 deletions.
31 changes: 11 additions & 20 deletions plugin-server/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,18 @@ Let's get you developing the plugin server in no time:

1. Install dependencies and prepare for takeoff by running command `pnpm i`.

1. Start a development instance of [PostHog](/PostHog/posthog) - [instructions here](https://posthog.com/docs/developing-locally). After all, this is the _PostHog_ Plugin Server, and it works in conjuction with the main server.
1. Start a development instance of [PostHog](/PostHog/posthog) - [instructions
here](https://posthog.com/docs/developing-locally). After all, this is the _PostHog_ Plugin
Server, and it works in conjuction with the main server.

1. Make sure that the plugin server is configured correctly (see [Configuration](#Configuration)). The following settings need to be the same for the plugin server and the main server: `DATABASE_URL`, `REDIS_URL`, `KAFKA_HOSTS`, `CLICKHOUSE_HOST`, `CLICKHOUSE_DATABASE`, `CLICKHOUSE_USER`, and `CLICKHOUSE_PASSWORD`. Their default values should work just fine in local development though.
1. Make sure that the plugin server is configured correctly (see [Configuration](#Configuration)).
The following settings need to be the same for the plugin server and the main server:
`DATABASE_URL`, `REDIS_URL`, `KAFKA_HOSTS`, `CLICKHOUSE_HOST`, `CLICKHOUSE_DATABASE`,
`CLICKHOUSE_USER`, and `CLICKHOUSE_PASSWORD`. Their default values should work just fine in local
development though.

1. Start the plugin server in autoreload mode with `pnpm start:dev`, or in compiled mode with `pnpm build && pnpm start:dist`, and develop away!
1. Start the plugin server in autoreload mode with `pnpm start:dev`, or in compiled mode with `pnpm
build && pnpm start:dist`, and develop away!

1. Prepare for running tests with `pnpm setup:test`, which will run the
necessary migrations. Run the tests themselves with `pnpm test:{1,2}`.
Expand Down Expand Up @@ -160,7 +167,7 @@ Let's talk about the main thread first. This has:
1. `hub` – Handler of connections to required DBs and queues (ClickHouse, Kafka, Postgres, Redis), holds loaded plugins.
Created via `hub.ts -> createHub`. Every thread has its own instance.
1. `piscina` – Manager of tasks delegated to threads. `makePiscina` creates the manager, while `createWorker` creates the worker threads.
1. `piscina` – This used to be a manager of tasks that were delegated to threads. It is now a shim over normal JS function calls that will be removed in the future.
1. `pluginScheduleControl` – Controller of scheduled jobs. Responsible for adding Piscina tasks for scheduled jobs, when the time comes. The schedule information makes it into the controller when plugin VMs are created.
Expand All @@ -176,18 +183,6 @@ Let's talk about the main thread first. This has:
1. `mmdbServer` – TCP server, which works as an interface between the GeoIP MMDB data reader located in main thread memory and plugins ran in worker threads of the same plugin server instance. This way the GeoIP reader is only loaded in one thread and can be used in all. Additionally this mechanism ensures that `mmdbServer` is ready before ingestion is started (database downloaded from [http-mmdb](https://github.com/PostHog/http-mmdb) and read), and keeps the database up to date in the background.
### Worker threads
This begins with `worker.ts` and `createWorker()`.
`hub` is the same setup as in the main thread.
New functions called here are:
1. `setupPlugins` – Loads plugins and prepares them for lazy VM initialization.
2. `createTaskRunner` – Creates a Piscina task runner that allows to operate on plugin VMs.
> Note:
> An `organization_id` is tied to a _company_ and its _installed plugins_, a `team_id` is tied to a _project_ and its _plugin configs_ (enabled/disabled+extra config).
Expand Down Expand Up @@ -215,7 +210,3 @@ We carry a node-rdkafka patch that adds cooperative rebalancing. To generate thi
# in the plugin-server directory, target the temporary directory from the previous command
pnpm patch-commit /private/var/folders/b7/bmmghlpx5qdd6gpyvmz1k1_m0000gn/T/6082767a6879b3b4e11182f944f5cca3
## Questions?
### [Join our Slack community. 🦔](https://posthog.com/slack)

0 comments on commit 9346ae9

Please sign in to comment.