How plane.watch receives data from feeders.
Designed to be horizontally scalable, sat behind TCP load balancer(s).
- plane.watch Border Control
- Overview
- Configuring the environment
- Statistics
- NATS Detail
pw_bordercontrol.pingpw_bordercontrol.stunnel.reloadcertkeypw_bordercontrol.feedinimage.rebuildpw_bordercontrol.feeder.kickpw_bordercontrol.feeders.metricspw_bordercontrol.feeder.metricspw_bordercontrol.feeder.metrics.beastpw_bordercontrol.feeder.metrics.mlatpw_bordercontrol.feeder.connected.beastpw_bordercontrol.feeder.connected.mlat
- Terminates incoming stunnel'd BEAST and MLAT connections from feeders
- bordercontrol expects the SSL SNI sent from the client to be the feeder's "API key" (a UUID)
- SNIs in incoming connections are checked against a list of valid UUIDs pulled from ATC.
- Connections with invalid SNIs are rejected.
- Builds and starts regional MLAT servers (muxes)
- Builds and manages "feed-in" container image
- When feeders connect:
- A "feed-in" container is automatically created and started for the specific feeder, which:
- Runs
pw_ingestwhich receives and decodesBEASTand puts the position data into NATS
- Runs
- BEAST data is sent to the "feed-in" container
- MLAT connections are proxied to
mlat-serverinstances
- A "feed-in" container is automatically created and started for the specific feeder, which:
In the root of the repository, create a .env file containing the following:
| Environment Variable | CLI Flag Equiv. | R/O | Description | Example |
|---|---|---|---|---|
ATC_UPDATE_FREQ |
--atcupdatefreq |
O | Frequency (in minutes) for valid feeder updates from ATC | 1 |
ATC_URL |
--atcurl |
R | URL to ATC API | http://10.0.6.2:3000 |
ATC_USER |
--atcuser |
R | Username/email for ATC API | [email protected] |
ATC_PASS |
--atcpass |
R | Password for ATC API | REDACTED |
BC_CERT_FILE |
--cert |
O | Path (within the bordercontrol container) of the X509 cert | /etc/ssl/private/push.plane.watch/fullchain.pem |
BC_KEY_FILE |
--key |
O | Path (within the bordercontrol container) of the X509 key | /etc/ssl/private/push.plane.watch/privkey.pem |
BC_LISTEN_API |
--listenapi |
O | Address and TCP port server will listen on for API, stats & Prometheus metrics | :8080 |
BC_LISTEN_BEAST |
--listenbeast |
O | Address and TCP port to listen on for BEAST connections | 0.0.0.0:12345 |
BC_LISTEN_MLAT |
--listenmlat |
O | Address and TCP port to listen on for MLAT connections | 0.0.0.0:12346 |
BC_VERBOSE |
--verbose |
O | Change default log level from Info to Debug | |
FEED_IN_BUILD_CONTEXT |
--feedinimagecontext |
O | Feed-in-image build context | /opt/feed-in/ |
FEED_IN_BUILD_DOCKERFILE |
--feedinimagedockerfile |
O | Feed-in image build Dockerfile (relative to context) | Dockerfile.feeder |
FEED_IN_CONTAINER_PREFIX |
--feedincontainerprefix |
O | Feed-in container prefix | feed-in- |
FEED_IN_CONTAINER_NETWORK |
--feedincontainernetwork |
O | Feed-in container network | bordercontrol_feeder |
FEED_IN_IMAGE |
--feedinimage |
O | Feed-in image name | feed-in |
PW_INGEST_SINK |
--pwingestpublish |
R | URL passed through to pw_ingest in feed-in containers |
nats://nats-ingest.plane.watch:4222 |
NATS |
--natsurl |
O | NATS URL for stats/control | nats://nats-ingest.plane.watch:4222 |
NATS_INSTANCE |
--natsinstance |
O | NATS instance identifier for this instance of Bordercontrol | prod-bordercontrol-01 |
Note: O = Optional, R = Required.
Example:
# ATC API connection details
ATC_URL=http://atc.yourdomain.tld:3000
ATC_USER=REDACTED
ATC_PASS=REDACTED
# SSL Key/Cert
BC_CERT_FILE=/path/to/fullchain.pem
BC_KEY_FILE=/path/to/privkey.pem
# pw_ingest
PW_INGEST_SINK=nats://nats-ingest.youdomain.tld:4222
NATS=nats://nats-ingest.youdomain.tld:4222Create a the docker-compose-local.yml file (see the example), under services: -> bordercontrol:, ensure the path holding SSL certs/keys is mapped as a volume.
From the root of the repository, run docker compose up -d --build.
This will build the images, containers and launch bordercontrol.
From the root of the repository, run docker compose down.
This will stop and remove multiplexer and bordercontrol containers. After approximately 10 minutes, the feed-in containers will stop and be removed.
From the root of the repository, run docker compose build --pull && docker compose up -d.
Any updated containers will be rebuilt and recreated.
This can be done via rebuilding the feed-in-builder container, or a NATS request to pw_bordercontrol.feedinimage.rebuild with the body containing the NATS instance or wildcard (*) for all instances:
- From the root of the repository, run:
docker compose build --pull feed-in-builder; or - via NATS:
nats req --timeout=2m pw_bordercontrol.feedinimage.rebuild "*"
For the feed-in containers, bordercontrol checks these every 5 minutes. Containers are updated one at a time. After updating a container, bordercontrol waits 30 seconds, to hopefully prevent any visible impact on the web UI. These timeouts/sleeps can be skipped by sending a SIGUSR1 signal to bordercontrol. This is shown in the logs:
Mon Nov 6 20:51:43 UTC 2023 INF caught signal, proceeding immediately goroutine=checkFeederContainers signal="user defined signal 1"
Mon Nov 6 20:51:43 UTC 2023 INF out of date container being killed for recreation container=feed-in-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx goroutine=checkFeederContainers
Bordercontrol will non-disruptively re-read SSL certificate(s) if it receives a SIGHUP, or a NATS request to pw_bordercontrol.stunnel.reloadcertkey with the body containing the NATS instance or wildcard (*) for all instances.
- via
SIGHUP:docker exec bordercontrol pkill -HUP bordercontrol - via NATS:
nats req pw_bordercontrol.stunnel.reloadcertkey "*"
If the need arises to kick a feeder, this can be done via NATS:
| Subject | Header | Body | Description |
|---|---|---|---|
pw_bordercontrol.feeder.kick |
Ignored | Feeder API Key | Immediately removes the feed-in container associated with the feeder. |
Example:
# nats req pw_bordercontrol.feeder.kick <Feeder API Key>
13:33:35 Sending request on "pw_bordercontrol.feeder.kick"
13:33:35 Received with rtt 254.043895ms
+ACK
Healthchecks are run within the feed-in containers.
The healthcheck will check the following:
- Checks BEAST connection inbound from bordercontrol
- Checks NATS connection outbound
If any of the above checks fail, the container is marked as unhealthy.
If the feed-in container has been unhealthy for longer than 10 minutes, the container will terminate.
As the containers are launched via bordercontrol with "autoremove" enabled, the stopped container will be removed automatically.
This ensures that only containers of active feeders are started and running.
Bordercontrol will report statistics via NATS.
On all replies, the bordercontrol instance is returned as a header, so the instance the feeder is connected to can be identified if needed.
| Subject | Header | Body | Description |
|---|---|---|---|
pw_bordercontrol.feeder.connected.beast |
Ignored | Feeder API Key | Returns true if feeder matching API Key has a BEAST protocol connection. |
pw_bordercontrol.feeder.connected.mlat |
Ignored | Feeder API Key | Returns true if feeder matching API Key has a MLAT protocol connection. |
pw_bordercontrol.feeder.metrics |
Ignored | Feeder API Key | Returns JSON containing metrics for all connections. Eg: {"feeder_code":"YPPH-0003","label":"Bayswater2","beast_connected":true,"beast_bytes_in":10120,"beast_bytes_out":0,"beast_connection_time":"2024-01-04T15:14:03.892056108Z","mlat_connected":true,"mlat_bytes_in":140,"mlat_bytes_out":458,"mlat_connection_time":"2024-01-04T15:13:44.397506876Z"} |
pw_bordercontrol.feeder.metrics.beast |
Ignored | Feeder API Key | Returns JSON containing metrics for BEAST connection. Eg: {"bytes_in":305177,"bytes_out":0,"connection_time":"2024-01-04T14:37:47.87329718Z"} |
pw_bordercontrol.feeder.metrics.mlat |
Ignored | Feeder API Key | Returns JSON containing metrics for MLAT connection. Eg: {"bytes_in":29172,"bytes_out":744,"connection_time":"2024-01-04T14:37:27.133693249Z"} |
If the requested feeder is not connected to bordercontrol, bordercontrol will silently ignore the message. This ensures that in scale-out environments, you can simply use the first (and hopefully only) reply. It is suggested to use a short timeout too (100ms should be plenty).
| Subject | Header | Body | Description |
|---|---|---|---|
pw_bordercontrol.feeders.metrics |
Ignored | Ignored | Returns JSON containing metrics for all feeders, all connections. Eg: {"<feeder api key redacted>":{"feeder_code":"YPPH-0003","label":"Bayswater2","beast_connected":true,"beast_bytes_in":258,"beast_bytes_out":0,"beast_connection_time":"2024-01-04T15:28:28.697372348Z","mlat_connected":true,"mlat_bytes_in":140,"mlat_bytes_out":458,"mlat_connection_time":"2024-01-04T15:28:10.295874658Z"}} |
Bordercontrol will display a simple HTML table of feeders with statistics at http://dockerhost:8080
Bordercontrol supports the following API calls to http://dockerhost:8080:
| Query Type | URL Path | Returns |
|---|---|---|
GET |
/api/v1/feeder/<UUID> |
Statistics for single feeder by UUID |
GET |
/api/v1/feeders/ |
Statistics for all feeders |
These queries return a JSON object with keys Data and Error. If Error == "", then the Data key should contain the requested data. If Error != "", then the error details are contained within Error and any data in Data should be discarded.
Prometheus metrics are published at http://dockerhost:8080/metrics.
Type: request
| Subject | Header | Body | Description |
|---|---|---|---|
pw_bordercontrol.ping |
Ignored | Ignored | Each instance of bordercontrol will reply with basic status information. |
Bordercontrol will respond with:
- Header
instancewith the instance identifier defined by--natsinstance/NATS_INSTANCE - Header
uptimewith the time this instance has been running - Header
versionwith the version of this instance - Body of
pong
When submitting this request, set replies=0 so the NATS client listens until timeout is reached. This will ensure responses are received from all running instances.
Example:
# nats req --replies=0 pw_bordercontrol.ping -
01:09:41 Sending request on "pw_bordercontrol.ping"
01:09:41 Received with rtt 1.324181ms
01:09:41 instance: 29146a332fdf
01:09:41 uptime: 10h42m40.528380969s
01:09:41 version: 0.0.1 (39376d7), 2024-01-05T14:25:14Z
pong
Type: request
| Subject | Header | Body | Description |
|---|---|---|---|
pw_bordercontrol.stunnel.reloadcertkey |
Ignored | Bordercontrol instance or wildcard (*) for all instances. |
Bordercontrol will reload SSL/TLS cert/key. |
Bordercontrol will acknowledge the message when certs/keys are reloaded.
When submitting this request using a wildcard, set replies=0 so the NATS client listens until timeout is reached. This will ensure responses are received from all running instances.
Example with wildcard:
# nats req --replies=0 pw_bordercontrol.stunnel.reloadcertkey "*"
01:24:13 Sending request on "pw_bordercontrol.stunnel.reloadcertkey"
01:24:13 Received with rtt 1.586432ms
+ACK
Example specifying instance:
# nats req pw_bordercontrol.stunnel.reloadcertkey 29146a332fdf
01:27:04 Sending request on "pw_bordercontrol.stunnel.reloadcertkey"
01:27:04 Received with rtt 1.997614ms
+ACK
Type: request
| Subject | Header | Body | Description |
|---|---|---|---|
pw_bordercontrol.feedinimage.rebuild |
Ignored | Bordercontrol instance or wildcard (*) for all instances. |
Bordercontrol will rebuild the feed-in image. |
Bordercontrol will build the feed-in containers with the context and Dockerfile defined by:
| Environment Variable | Command Line Flag | Description |
|---|---|---|
FEED_IN_BUILD_CONTEXT |
--feedinimagecontext |
Feed-in-image build context |
FEED_IN_BUILD_DOCKERFILE |
--feedinimagedockerfile |
Feed-in image build Dockerfile (relative to context) |
Bordercontrol reply with the outcome of the build. More detailed build logs are logged to the bordercontrol log.
When submitting this request:
- If using a wildcard, set
replies=0so the NATS client listens until timeout is reached. This will ensure responses are received from all running instances. - Set
timeoutto an appropriate value to enable the build to complete before the timeout is reached.
Example:
# nats req --timeout=2m pw_bordercontrol.feedinimage.rebuild 29146a332fdf
01:34:06 Sending request on "pw_bordercontrol.feedinimage.rebuild"
01:34:42 Received with rtt 35.981288461s
01:34:42 instance: 29146a332fdf
01:34:42 result: ok
{"stream":"Successfully tagged feed-in:latest\n"}
Relevant logs from bordercontrol:
Sat Jan 6 01:37:18 UTC 2024 DBG build output stream="Step 1/15 : FROM ghcr.io/plane-watch/pw-pipeline:pw_ingest as pipeline"
Sat Jan 6 01:37:19 UTC 2024 DBG build output stream="Step 2/15 : FROM debian:bookworm-20231218-slim"
Sat Jan 6 01:37:22 UTC 2024 DBG build output stream="Step 3/15 : ARG S6_OVERLAY_VERSION=3.1.6.0"
Sat Jan 6 01:37:22 UTC 2024 DBG build output stream="Step 4/15 : SHELL [\"/bin/bash\", \"-o\", \"pipefail\", \"-c\"]"
Sat Jan 6 01:37:22 UTC 2024 DBG build output stream="Step 5/15 : RUN set -x && TEMP_PACKAGES=() && KEPT_PACKAGES=() && TEMP_PACKAGES+=(curl) && TEMP_PACKAGES+=(git) && TEMP_PACKAGES+=(file) && KEPT_PACKAGES+=(ca-certificates) && KEPT_PACKAGES+=(procps) && KEPT_PACKAGES+=(iproute2) && KEPT_PACKAGES+=(xz-utils) && apt-get update && apt-get install -y --no-install-recommends \"${KEPT_PACKAGES[@]}\" \"${TEMP_PACKAGES[@]}\" && git clone --depth=1 https://github.com/mikenye/docker-healthchecks-framework.git /opt/healthchecks-framework && rm -rf /opt/healthchecks-framework/.git* /opt/healthchecks-framework/*.md /opt/healthchecks-framework/tests && apt-get remove -y \"${TEMP_PACKAGES[@]}\" && apt-get autoremove -y && rm -rf /src/* /tmp/* /var/lib/apt/lists/*"
Sat Jan 6 01:37:22 UTC 2024 DBG build output stream="Step 6/15 : COPY --from=pipeline /app/pw_ingest /usr/local/bin/pw_ingest"
Sat Jan 6 01:37:22 UTC 2024 DBG build output stream="Step 7/15 : COPY rootfs/ /"
Sat Jan 6 01:37:22 UTC 2024 DBG build output stream="Step 8/15 : ADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-noarch.tar.xz /tmp"
Sat Jan 6 01:37:23 UTC 2024 DBG build output stream="Step 9/15 : RUN tar -C / -Jxpf /tmp/s6-overlay-noarch.tar.xz"
Sat Jan 6 01:37:23 UTC 2024 DBG build output stream="Step 10/15 : ADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-x86_64.tar.xz /tmp"
Sat Jan 6 01:37:25 UTC 2024 DBG build output stream="Step 11/15 : RUN tar -C / -Jxpf /tmp/s6-overlay-x86_64.tar.xz"
Sat Jan 6 01:37:25 UTC 2024 DBG build output stream="Step 12/15 : ENV S6_LOGGING=0 S6_VERBOSITY=1"
Sat Jan 6 01:37:25 UTC 2024 DBG build output stream="Step 13/15 : ENV PW_INGEST_INPUT_MODE=listen PW_INGEST_INPUT_PROTO=beast PW_INGEST_INPUT_ADDR=0.0.0.0 PW_INGEST_INPUT_PORT=12345"
Sat Jan 6 01:37:25 UTC 2024 DBG build output stream="Step 14/15 : ENTRYPOINT [ \"/init\" ]"
Sat Jan 6 01:37:25 UTC 2024 DBG build output stream="Step 15/15 : HEALTHCHECK --interval=300s --timeout=40s --start-period=600s --retries=5 CMD /scripts/healthcheck.sh"
Sat Jan 6 01:37:25 UTC 2024 DBG build output stream="Successfully built 8e3321e421fc"
Sat Jan 6 01:37:25 UTC 2024 DBG build output stream="Successfully tagged feed-in:latest"
Type: request
| Subject | Header | Body | Description |
|---|---|---|---|
pw_bordercontrol.feeder.kick |
Ignored | API Key of feeder | Bordercontrol will immediately kill and remove the feeder's feed-in container. |
Bordercontrol will acknowledge when the feeder is kicked.
Only the instance(s) with connections from the specified feeder will respond.
Example:
# nats req pw_bordercontrol.feeder.kick <API Key Redacted>
01:42:24 Sending request on "pw_bordercontrol.feeder.kick"
01:42:25 Received with rtt 263.201835ms
+ACK
Relevant logs from bordercontrol:
Sat Jan 6 01:42:24 UTC 2024 INF killing feed-in container container=feed-in-<API Key Redacted>
Sat Jan 6 01:42:24 UTC 2024 ERR error writing to server error="write tcp 172.18.0.13:42626->172.18.0.15:12345: write: broken pipe" code=YPPH-0003 connNum=5 connections=1/1 dst=feed-in-<API Key Redacted> func=["feeder_conn.go","proxyClientConnection"] label=Bayswater2 mux=mux-wa port=12345 proto=BEAST proxy=ClientToServer src=<IP redacted> uuid=<API Key Redacted>
Type: request
| Subject | Header | Body | Description |
|---|---|---|---|
pw_bordercontrol.feeders.metrics |
Ignored | Ignored | Bordercontrol will send statistics of all connected feeders. |
Bordercontrol reply with all connected feeder metrics in JSON format.
When submitting this request:
- It is recommended to use
replies=0so the NATS client listens until timeout is reached. This will ensure responses are received from all running instances.
Example:
# nats req --replies=0 pw_bordercontrol.feeders.metrics -
09:02:31 Sending request on "pw_bordercontrol.feeders.metrics"
09:02:31 Received with rtt 957.873µs
09:02:31 instance: 29146a332fdf
{"<API Key Redacted>":{"feeder_code":"YPPH-0003","label":"Bayswater2","beast_connected":true,"beast_bytes_in":40845491,"beast_bytes_out":0,"beast_connection_time":"2024-01-06T01:42:30.572867107Z","mlat_connected":true,"mlat_bytes_in":7489647,"mlat_bytes_out":53888,"mlat_connection_time":"2024-01-05T14:27:08.508839791Z"}}
Type: request
| Subject | Header | Body | Description |
|---|---|---|---|
pw_bordercontrol.feeder.metrics |
Ignored | API Key of feeder | Bordercontrol will send statistics of individual feeder. |
The instance of bordercontrol with the connected feeder reply with the feeder's metrics in JSON format. If feeder is not connected, no response will be received.
Example:
# nats req pw_bordercontrol.feeder.metrics <API Key Redacted>
09:06:36 Sending request on "pw_bordercontrol.feeder.metrics"
09:06:36 Received with rtt 1.011305ms
09:06:36 instance: 29146a332fdf
{"feeder_code":"YPPH-0003","label":"Bayswater2","beast_connected":true,"beast_bytes_in":41143080,"beast_bytes_out":0,"beast_connection_time":"2024-01-06T01:42:30.572867107Z","mlat_connected":true,"mlat_bytes_in":7524410,"mlat_bytes_out":54133,"mlat_connection_time":"2024-01-05T14:27:08.508839791Z"}
Type: request
| Subject | Header | Body | Description |
|---|---|---|---|
pw_bordercontrol.feeder.metrics.beast |
Ignored | API Key of feeder | Bordercontrol will send BEAST protocol statistics of individual feeder. |
The instance of bordercontrol with the connected feeder reply with the feeder's BEAST metrics in JSON format. If feeder is not connected, no response will be received.
Example:
# nats req pw_bordercontrol.feeder.metrics.beast <API Key Redacted>
09:18:18 Sending request on "pw_bordercontrol.feeder.metrics.beast"
09:18:18 Received with rtt 1.307498ms
09:18:18 instance: 29146a332fdf
{"feeder_code":"YPPH-0003","label":"Bayswater2","bytes_in":42261164,"bytes_out":0,"connection_time":"2024-01-06T01:42:30.572867107Z"}
Type: request
| Subject | Header | Body | Description |
|---|---|---|---|
pw_bordercontrol.feeder.metrics.mlat |
Ignored | API Key of feeder | Bordercontrol will send MLAT protocol statistics of individual feeder. |
The instance of bordercontrol with the connected feeder reply with the feeder's MLAT metrics in JSON format. If feeder is not connected, no response will be received.
Example:
# nats req pw_bordercontrol.feeder.metrics.mlat <API Key Redacted>
09:14:48 Sending request on "pw_bordercontrol.feeder.metrics.mlat"
09:14:48 Received with rtt 3.270292ms
09:14:48 instance: 29146a332fdf
{"feeder_code":"YPPH-0003","label":"Bayswater2","bytes_in":7608127,"bytes_out":54571,"connection_time":"2024-01-05T14:27:08.508839791Z"}
Type: request
| Subject | Header | Body | Description |
|---|---|---|---|
pw_bordercontrol.feeder.connected.beast |
Ignored | API Key of feeder | Bordercontrol will return true if feeder has a BEAST connection. |
The instance of bordercontrol with a BEAST connection for the specified feeder will return true. If feeder has no BEAST connection, no response will be received.
Example:
# nats req pw_bordercontrol.feeder.connected.beast <API Key Redacted>
09:21:45 Sending request on "pw_bordercontrol.feeder.connected.beast"
09:21:45 Received with rtt 965.63µs
09:21:45 instance: 29146a332fdf
true
Type: request
| Subject | Header | Body | Description |
|---|---|---|---|
pw_bordercontrol.feeder.connected.mlat |
Ignored | API Key of feeder | Bordercontrol will return true if feeder has a MLAT connection. |
The instance of bordercontrol with a MLAT connection for the specified feeder will return true. If feeder has no MLAT connection, no response will be received.
Example:
# nats req pw_bordercontrol.feeder.connected.mlat <API Key Redacted>
09:24:32 Sending request on "pw_bordercontrol.feeder.connected.mlat"
09:24:32 Received with rtt 1.698948ms
09:24:32 instance: 29146a332fdf
true