v24.5.2

24.5.2 (2024-09-30)

Features

• add hash to bootstrap (#1168) (4b0be5f)
• bolt: add workflow commands to bolt (#1131) (a47b296)
• builds: add prewarm ats for builds (#1176) (0a269b4)

Bug Fixes

• allocation sizes for nomad (#1127) (2e3217e)
• bolt: add forwarded and persistent db shells (#1130) (f9b6707)
• builds: allow null tags (#1177) (1c75e3b)
• builds: fix exclusive tags query (#1173) (b8c323a)
• clusters: allow dns deletion when draining and tainted (#1132) (c808d08)
• cluster: skip pruning servers without provider server id (#1133) (ca43432)
• ds, mm: hard code disk per core (#1134) (5ee2809)
• ds: add back runc cleanup (#1172) (8e08889)
• fix build tags (#1190) (6e2d214)
• fix documentation link for errors (#1174) (eb7fdaf)
• job-run: fix dupe allocs, re-enable drain all (#1128) (d019e01)
• mm, ds: fix dupe alloc killing (#1124) (dcdb06a)
• more accurate job-run cpu metrics (#1122) (312958e)
• reduce scheduler skew on distributed clusters (#1175) (2794e09)
• worfklows: add silence ts (#1129) (06d965b)
• workflows: add error message for max sql retries (#1125) (80a33f0)
• workflows: add retry delay for txn errors (#1138) (614846b)
• workflows: use unions instead of OR (#1170) (1ca8ab6)

Chores

• add back node exporter metrics (#1136) (1eeedcb)
• enable batch ssh commands (#1119) (505c09c)
• increase install timeout (#1139) (38584c9)
• increase nomad heartbeat ttl (#1140) (437494a)
• linode: pin kernel version (#1123) (48686f7)
• release 24.5.2 (90318ca)
• remove bolt templates (#1135) (f0925f0)
• revert new node exporter metrics (#1118) (07b6095)

v24.5.1

24.5.1 (2024-09-04)

Features

• workflows: clean up dispatching syntax (#1079) (233efcc)

Bug Fixes

• clusters: add drain padding to nomad (#1100) (01ee21b)
• clusters: fix list lost op (#1110) (8ae85d2)
• clusters: gracefully handle node not found (#1099) (b460374)
• clusters: remove nomad drain complete signal (#1101) (c117224)
• clusters: switch from drain to ineligible system (#1102) (09f5143)
• ds: change nomad prefix (#1113) (705a470)
• ds: implement nomad monitors with signals (#1105) (238a8e9)
• fix signal history divergence (#1115) (3cbfc1b)
• job-run: delete second allocation immediately (#1104) (78b73fd)
• nomad: readd allocation metrics (#1109) (600d4fb)
• update api endpoint names (#1080) (33e780d)
• workflows: add retry to internal sql queries (#1112) (ef010d0)
• workflows: implement backoff for timeouts (#1111) (6659b34)

Chores

• main: release 24.5.0 (#1103) (7652421)
• release 24.5.0 (1657c7c)
• release 24.5.1 (12f7ee9)
• update all uses of workflows to new syntax (#1108) (0079be8)
• workflows: clean up internal contexts (#1107) (2148f9e)

v24.5.0

24.5.0 (2024-08-27)

Features

• add json cache (#939) (7c2897a)
• add ready_ts to servers endpoint (#1006) (8b44a7c)
• add server logs endpoint (#1005) (a23073b)
• better_uptime: allow disabling notifications (#923) (7eb12b0)
• bolt: add k9s to nix-shell (#903) (7668942)
• bolt: add lost servers list and prune commands (#1096) (0480702)
• bolt: build svcs as docker containers locally (#945) (11f4258)
• bolt: run tests in containers (#947) (08a53e3)
• clusters: add toggle for prebakes (#932) (09890e5)
• clusters: convert clusters to new workflow system (#974) (0c5558b)
• clusters: gg monitor for better uptime (#921) (152c55b)
• combine ops and workers into one svc type (#957) (774da5c)
• ds: add datacenters endpoint (#1065) (32d448e)
• ds: add server create failed message (#1068) (82daf2d)
• ds: rewrite dynamic servers on workflows (#1060) (c9b5578)
• infra: auto-create dev tunnel & public ip (#979) (0d82155)
• infra: enable configuring min & max cockroach pool conns (#922) (e8e7255)
• runtime: switch from json to logfmt (#984) (10a0e6c)
• svc: add servers create endpoint (#740) (77f1b3f)
• update billing to use tiers (#900) (918038a)
• workflows, clusters: add workflow backfill service (#1000) (e69b767)
• workflows: add api ctx for workflows (#865) (1a468d3)
• workflows: add loops (#1001) (272a09d)
• workflows: add message and signal history (#987) (0003acc)
• workflows: add messages (#977) (38c1171)
• workflows: add metrics (#1008) (a4837e2)
• workflows: add nats worker wake (#1039) (1fc72f1)
• workflows: add observe workflows fn (#901) (22a1ebd)
• workflows: add operations service type (#898) (0a0d377)
• workflows: add sleep fn (#1077) (c477ba9)
• workflows: add tags (#956) (36494eb)
• workflows: allow changing tags in workflow (#962) (01ecf86)
• workflows: implement retry backoff for activity errors (#999) (6e8560e)

Bug Fixes

• add ip whitelist to tunnels (#930) (88ce4b3)
• add players and servers db indexes (#960) (53dc398)
• add priority class to nats (#1019) (954d864)
• api: move cors verification to endpoint level (#1094) (4a4b4fe)
• backfill script, crdb usage type (#1089) (ad0a260)
• better_uptime: handle null verify_ssl (#950) (e9d8edb)
• bolt: correctly hash untracked files (#1047) (2b885e5)
• bolt: exclude volumes when using native docker builder (#969) (8ac0a55)
• bolt: explicitly handle no nomad leader error (#971) (20822fc)
• bolt: update opengb -> backend env var name (#1058) (4250808)
• bolt: validate hub regex in ns config (#1093) (b2d5cca)
• cache: mixed values in Cache::fetch_all (#927) (d69a072)
• captcha: sanitize form body (#1098) (9b56efc)
• chirp: write message tail when history is disabled (#997) (9f377ba)
• cloud: add clean timeout for matchmaker logs (#942) (a395e3f)
• cluster: dc-get column mismatch (#958) (53e276a)
• cluster: dns creation (#1066) (1ef72e6)
• clusters: add network_out metrics for hardware (#1016) (30d15c3)
• clusters: backfill json columns ([...

v24.4.1

24.4.1 (2024-06-06)

Features

• add compat layer between old ctx and new workflows (#788) (787971b)
• add ray ids to workflows, clean up types (#787) (3072bdc)
• add workflows (#783) (378d528)
• global error raw variant (#784) (4b11578)
• run sub workflows in the same process (#789) (717e096)
• workflows: add retries and timeouts (#860) (cc0b893)
• workflows: add worker instance failover (#854) (c5a32a3)

Bug Fixes

• cast workflow errors to raw global errors (#785) (c90d939)
• draining and tainted server grafana chart (#855) (d0cdb38)
• mm: add index for run_proxied_ports (#868) (e0785e9)
• mm: call mm-lobby-cleanup from mm-gc even for preemptive lobbies without sql row (#856) (5315a9a)
• mm: correctly handle lobby not found error if joining direclty to lobby id that doesn't exist (#867) (af3513a)
• mm: require specifying matchmaker config for new game versions (#895) (92d86fd)
• tls: provision cloudflare cert pack if opengb enabled (#869) (1dafa9e)

Chores

• infra: move cdn to api-traefik-provider (#857) (9370e9e)
• release 24.4.1 (30cc822)

v24.4.0

24.4.0 (2024-06-04)

⚠ BREAKING CHANGES

• Cleanup API definitions, module imports (#534)

Features

• add 1password integration docs (#595) (29045ea)
• Add cluster admin cli (#644) (5b1de57)
• add crdb data source to grafana (#732) (f22694f)
• add env update error (#814) (48a5883)
• add hacky secondary ingress route for game lobbies (#567) (8bb6bd6)
• add internal api monolith (#641) (f25ffe4)
• Add managed OpenGB (#535) (9085d51)
• add opengb to bootstrap (#844) (ebd3c7b)
• add operation to list all clusters (#717) (1f4b169)
• add patch method to router (#744) (ed6596c)
• add pool filter to cluster dashboard (#830) (5436461)
• add provider api token to all linode calls (#613) (3882047)
• add provisioning dashboard (#733) (a1f9dcc)
• add ray id to error body (#833) (c115d6f)
• add region list/resolve per game (#633) (92275d8)
• Add script to run cargo clean (#700) (0f653e2)
• add toggle for load tests (#583) (a78d682)
• add vector http source (#800) (f4f2734)
• api-admin: add server destroy endpoint (#838) (4ff616b)
• bolt: list datacenter CLI command (#728) (c4a88de)
• bolt: update datacenters from CLI (#727) (083cd19)
• configurable drain ts per pool (#684) (f88c457)
• dynamic TLS generation (#635) (66e49dd)
• grafana: rivet logs dashboard (#724) (9a43f3a)
• infra: add ability to provision dev tunnel (#692) (659f8a1)
• Infra: Loops welcome email (b2e4006)
• nix: skip building bolt in nix with NIX_SKIP_BOLT (#664) (8e16a94)
• svc: resolve cluster name id op (#751) (58200ec)

Bug Fixes

• add last upload id (#745) (d10d917)
• add min count to autoscaler (#826) (9fe12a1)
• add patch to CORS (#848) (09f3ddc)
• add region to dns for path routing (#574) (e10ad25)
• add transacitons (#689) (f55b7e6)
• add transactions and locks (#696) (477ade5)
• api admin hub endpoint is incorrect (#660) (0aff347)
• api-status: auto-delete lobby after testing connection (#770) (9803f39)
• ats: don't send requests to ats nodes without install_complete_ts (#807) (618a429)
• bolt: copy & install git in docker for cluster build.rs (#769) (12bf1d4)
• bolt: correctly check for existing env var (#705) (ca4e48d)
• bolt: dont fully parse config when pulling (#816) (d22b08b)
• bolt: uncomment provisioning check (#749) (f25bead)
• bolt: update rust test package_id parsing (#622) (3d987ab)
• Change sdks linguist-vendored to linguist-generated (#662) (602749f)
• change test relative path (#754) (daf1d07)
• check for draining before installing/creating dns (#773) (cbe450b)
• chirp: add bypass for recursive messages (#708) (566088f)
• CI regression (#713) (636f0d3)
• claims (#672) (d61e290)
• clean up nomad jobs per test (#596) (6d7f0ee)
• Cleanup API definitions, module imports (#534) (0e0660a)
• cluster: delete dns record after failure to create (#827) (35fc6fe)
• cluster: don't taint servers that have cloud_destroy_ts (#839) (e5256f1)
• cluster: gg dns records leak if server destroyed before install comp...

v24.3.0

24.3.0 (2024-03-01)

Features

• bolt: add region filter to ssh command (#537) (af274a8)
• expose nomad dashboard via cloudflare tunnels (#543) (3a574c0)
• Main: Added Devcontainer files (9bb97db)
• mm: add config to opt-in individual games for host networking & root containers (#549) (be9ddd6)

Bug Fixes

• add checksum annotations to cloudflared deployment (#542) (f2d847b)
• bolt: clarify 1password service token warning (#541) (eb2e7d5)
• correct hcaptcha length (#548) (748aaa8)
• inaccessible admin routes (#555) (9896b09)
• revert to redis-rs v0.23.3 with panic patch (#552) (3780eaa)
• updated docs error url (#544) (7099658)

v24.2.0

Added

• Infra Added Better Uptime monitor
• Bolt Add Docker RUN cache to distributed deploys to improve deploy speeds
• Infra Prometheus VPA
• Infra Apache Traffic Server VPA
• api-cloud Admins can view all teams & games in a cluster
• Added automatic deploy CI for staging
• Infra Added compactor and GC to Loki
• api-status Test individual Game Guard nodes to ensure all nodes have the correct configuration
• Generate separate SDKs for runtime (lightweight, essentials for running a game) and full (heavy, includes cloud APIs)
• Metrics for cache operations as well as a Grafana dashboard
• Bolt Added namespace config and secrets sync with bolt config pull and bolt config push via 1Password
• GROUP_DEACTIVATED error now shows reasons for deactivation. Added docs for deactivation reasons
• /health/essential endpoint to test connectivity to all essential services
• Added error when trying to deploy a distributed cluster on a non-linux-x86 machine (not supported)

Changed

• api-status More comprehensive status check that both creates a lobby & connects to it
• More details in CLAIMS_MISSING_ENTITLEMENT error
• API Added 120s timeout to reading request body and writing response to all requests going through Traefik
• Infra Update Promtail logs to match k8s semantics
• Infra Added Cache-Control: no-cache to 400 responses from CDN
• [BREAKING] Infra Removed config-less hCaptcha. You are now required to provide a site key and
  secret key for the hCaptcha config in your game version matchmaker config for all future versions (old
  version will remain operational using our own hCaptcha site key).
• Internal Updated source hash calculation to use git diff and git rev-parse HEAD
• API Removed x-fern-* headers from generated TypeScript clients
• Implemented liveness probe to check connectivity to essential services
• Remove public-facing health check endpoints
• API Removed ability to choose a name id when creating a game. One will be generated based on the given display name

Fixed

• Bolt Prompt prod won't prompt if does not have user control
• Bolt Exclude copying bloat from infra/tf/ to distributed Docker builds
• Invalid JWT tokens now return explicit TOKEN_INVALID error instead of 500
• Infra Remove debug logging from traefik-tunnel
• Game lobby logs now ship even when the lobby fails immediately
• Fixed CLAIMS_MISSING_ENTITLEMENT not formatting correctly (reason given was ?)
• Added role ARN to exec commands in k8s-cluster-aws tf provider to properly authenticate
• Change email attached to Stripe on group ownership change
• Enable keep-alive on redis crate
• Update redis crate to mitigate panic on connection failure during AUTH
• Wrong grace period for GG config to update after mm::msg::lobby_ready

Security

• Resolve RUSTSEC-2024-0003

v24.1.0

Added

• Infra New job-runner crate responsible for managing the OCI bundle runtime & log shipping on the machine
• Infra Jobs now log an explicit rate message when logs are rate limited & truncated
• Infra infra-artifacts Terraform plan & S3 bucket used for automating building & uploading internal binaries, etc.
• Infra Aiven Redis provider
• Bolt bolt secret set <path> <value> command
• Bolt bolt.confirm_commands to namespace to confirm before running commands on a namespace
• watch-requests load test
• mm-sustain load test

Changed

• Matchmaker Allow excluding matchmaker.regions in order to enable all regions
• Matchmaker Lowered internal overhead of log shipping for lobbies
• Matchmaker Game mode names are now more lenient to include capital letters & underscores
• API Return API_REQUEST_TIMEOUT error after 50s (see docs/infrastructure/API_TIMEOUTS.md for context)
• API Move generated client APIs to sdks/
• Lower long poll timeout from 60s -> 40s
• Bolt Moved additional project roots to Bolt.toml
• types Support multiple project roots for reusing Protobuf types
• Infra Switch from AWS ELB to NLB to work around surge queue length limitation
• Infra Loki resources are now configurable
• pools Allow infinite Redis reconnection attempts
• pools Set Redis client names
• pools Ping Redis every 15 seconds
• pools Enable test_before_acquire on SQLx
• pools Decrease SQLx idle_timeout to 3 minutes
• pools Set ClickHouse idle_timeout to 15 seconds
• api-helper Box path futures for faster compile times
• Upgrade async-nats
• test-mm-lobby-echo now handles SIGTERM and exits immediately, allows for less resource consumption while testing lobbies
• mm Dynamically sleep based on lobby's create_ts for Treafik config to update
• Infra Update Traefik tunnel client & server to v3.0.0-beta5
• Infra Update Traefik load balancer to v2.10.7

Security

• Resolve RUSTSEC-2023-0044

Fixed

• Infra runc rootfs is now a writable file system
• Matchmaker Logs not shipping if lobby exits immediately
• Matchmaker Returning lnd-atl instead of dev-lcl as the mocked mocked region ID in the region list
• API 520 error when long polling
• api-cloud Returning wrong domain for domains.cdn
• Infra Fix Prometheus storage retention conversion between mebibytes and megabytes
• Infra Fix typo in Game Guard Traefik config not exposing API endpoint
• Infra Kill signal for servers was SIGINT instead of SIGTERM
• Infra NATS cluster not getting enabled
• Infra Redis Kubernetes error when using non-Kubernetes provider
• api-helper Remove excess logging
• user_identity.identities not getting purged on create & delete
• Bolt Error when applying Terraform when a plan is no longer required
• api-helper Instrument path futures
• Infra CNI ports not being removed from the nat iptable, therefore occasionally causing failed connections
• Infra Disable nativeLB for Traefik tunnel
• Infra Update default Nomad storage to 64Gi
• Infra Tunnel now exposes each Nomad server individually so the Nomad client can handle failover natively instead of relying on Traefik
• Infra Traefik tunnel not respecting configured replicas
• Bolt ClickHouse password generation now includes required special characters

v23.2.0-rc1

Added

• Infra Lobby tagging system for filtering lobbies in /find
• Infra Dynamically configurable max player count in /find and /create
• Bolt Added bolt admin login to allow for logging in without an email provider setup. Automatically turns the user into an admin for immediate access to the developer dashboard.
• Bolt Fixed bolt db migrate create
• Infra Added user-admin-set service for creating an admin user

Changed

• Bolt Removed bolt admin team-dev create. You can use bolt admin login and the hub to create a new dev team
• Infra Turnstile CAPTCHA_CAPTCHA_REQUIRED responses now include a site key
• Infra Turnstile is no longer configurable by domain (instead configured by Turnstile itself)
• Infra Job log aggregating to use Vector under the hood to insert directly into ClickHouse
• Matchmaker Players automatically remove after extended periods of time to account for network failures

Fixed

• Infra Job logs occasionally returning duplicate log lines
• Matchmaker /list returning no lobbies unless include_state query parameter is true
• Matchmaker Players remove correctly when the player fails to be inserted into the Cockroach database and only exists in Redis

v23.1.0

Fixed

• prm-job configuration now gets applied correctly again