Cyclotron Megaissue

[oliverb123]

Cyclotron Megaissue

MVP: #24228

---

TODOs

MVP

Data/schema

• Cyclotron needs to remove unicode nulls from passed strings The only String left is queue name, which isn't a user provided / dynamic field
• Swap metadata, params, vm_state and the new "blob" column to Bytes. metadata and params will ~always be serialised objects in json format, but there's not need to force that to be the case (protobufs? capnproto? who knows what the future holds) chore(cyclotron): change columns to bytes, add blob column #24563
• Expose "vm_state" and "blob" as byte arrays that are optionally returned during dequeue, and retrievable later chore(cyclotron): change columns to bytes, add blob column #24563
• Impose (generous) size limits on vm_state and blob (1MB? 10MB? vm_state will generally include event data, and so will the body passed to fetch as part of a request. we impose a 1MB size limit on the body returned from fetch, but no size limit on what we send to external APIs). chore(cyclotron): change columns to bytes, add blob column #24563
• Expose a bulk createJob API

Safety/fetch

• Cyclotron fetch must not make requests to private DNS addresses
• Fetch+Janitor need to write to app_metrics/log_entires chore(cyclotron): write to app_metrics2 CH table via kafka #24625
• Error handling sanity pass on fetch - any failure to deser from the job queue is a user error, and should not lead to a dead letter (feat(cyclotron): Change dead-letter strategy, adopt in fetch and janitor #24569)
• Fetch needs to be updated to make use of the new "blob" column chore(cyclotron): change columns to bytes, add blob column #24563
• Fetch needs to be updated to use the shard-level DLQ approach (feat(cyclotron): Change dead-letter strategy, adopt in fetch and janitor #24569)

Safety/shards

• Backpressure
• Dead letter queues should be shard-wide, not worker specific - add worker functions to have a shared way of dead lettering things (that also adds an entry to an "exceptions" (or DLQ metadata, or some other better name) table recording which worker dead-lettered a job (or the janitor, in the case of poison pills) and why). A job is DLQ'd by setting queue_name, and setting it to available. (feat(cyclotron): Change dead-letter strategy, adopt in fetch and janitor #24569)
• Janitor should emit a gauge of DLQ depth (feat(cyclotron): Change dead-letter strategy, adopt in fetch and janitor #24569)

Monitoring

• Janitor gauges to report queue depth (# of available jobs waiting on a worker), as well as completed/failed/reset/poisoned jobs
• Fetch worker reporting job dequeue and processing time, as well as saturation
• Some grafana dashboards based on the above
• Workers, janitors and maybe managers should expose PgPool utilisation (fetch and janitor should tie this into their metrics)

Infra

• Container images for fetch/janitor
• helm charts for fetch/janitor
• RDS clusters in dev/eu/us

CDP/node consumer

• Get CDP consumer using cyclotron to queue jobs
• Get function callbacks dequeuing returned fetch jobs

Later

• Transparently de/compress vm_state and blob on read/write (maybe)
• Node library should support multiple worker/manager instances, eventually.
• Control commands