How it works

Who talks to whom

There is no nginx in front of RoadRunner: its HTTP plugin serves the pages and static files itself on localhost:8080. Its middleware run left to right: http_metrics, headers (security headers), gzip, static (files from public/ with an ETag, so a repeated request gets a 304) and rate_limiter (a token bucket per IP, only for requests that make it to PHP). RoadRunner is not a reverse proxy, so Centrifugo and Grafana are exposed on their own ports: WebSocket on :8000, the dashboard on :3000. The page gets these addresses from the server, built from the host it was opened by. Everything else lives in the internal docker compose network.

The main load is not HTTP but RPC over WebSocket, and the HTTP limiter never sees it. That is why RPCService takes a token from the player's bucket before any method: the bucket lives in the KV store process (RoadRunner's memory) and is changed under the lock rate:{group}:{player}, because KV has no atomic update. Fast clicking gets through, a stream from a script gets, after burst, a rate limited; the moderator is exempt from the limit.

RoadRunner and Centrifugo are linked by two gRPC channels going in opposite directions. Centrifugo proxies client events (connect, rpc) to the centrifuge plugin at app:10001, and the application publishes messages through the Centrifugo gRPC API at centrifugo:10000.

RoadRunner has three pools of PHP workers, two php app.php processes each: http serves the pages, centrifuge answers connect and RPC, jobs runs the transition jobs from the transitions queue (driver memory). RoadRunner hands requests and jobs to a worker through pipes, the process's stdin/stdout. Back to the kv, lock, jobs, metrics and centrifuge plugins the worker goes over goridge RPC on 127.0.0.1:6001.

Flyers, characters and the top shooters are moved by time, not by requests. That is what the service plugin is for: RoadRunner keeps one long-lived process php app.php hellowin:ticker outside the worker pools and restarts it if it crashes. The ticker reaches the plugins over the same RPC as the workers do, and through the Centrifugo gRPC API it also asks for the channel's presence. A second copy (overlap during a restart) is kept aside by the lock ticker, a lease that its owner renews once a second.

The game state lives in Valkey: the ritual and the story, the players and the top shooters, flyers, characters. It is read and written by RoadRunner's KV plugin: the store hellowin with the redis driver on VALKEY_ADDR, which PHP sees as the PSR-16 cache hellowin. The transition idempotency keys hellowin:amo:* and the stored dice responses hellowin:lease:* are written by the workers to the same Valkey directly, via predis (VALKEY_DSN): the spiral/idempotency package needs an atomic SET NX EX, which the KV plugin lacks. The locks (pumpkin:{id}, scene, player:{id}, flyer:{id}, walkers and others) and the queue live in the memory of the RoadRunner process, which is enough as long as there is a single app.

Page load and connection

GET / is plain HTTP. HomeController::index() takes a state snapshot from SnapshotFactory: RitualStore::snapshot() reads in a single kv.MGet hellowin:round, hellowin:scene, the story hellowin:story and hellowin:pumpkin:{id} of every pumpkin, and one more request fetches the characters hellowin:walkers. All of it without locks: the page may lag a click behind, and that is fine. The layout from app/config/hellowin.php and the policy numbers are added, and the template gets the snapshot (Snapshot):

• round, scene — the current round and scene;
• sceneStages — the number of the final scene (the length of HELLOWIN_SCENE_STAGE_MAX), stageMax — a pumpkin's last stage in the current scene;
• threshold — a pumpkin's threshold, the pumpkin limit (“Limits”); limits — the current scene's limits with the number of players players and the multiplier k;
• elements — the pumpkins: id, type, x, y, size, stage and clicks since the move to the current stage; bushes — the bushes the flyers take off from;
• walkers — the players' characters on the field with their paths;
• the story: ready — the scene is ready and waits for the moderator, finale — the finale step (from the reveal step on, also reveal), paused — progress is paused;
• now — server time in ms: the client uses it to compute its clock offset for the flyers and characters.

Then the browser opens a WebSocket and passes the player's secret from localStorage in the connection's data: {playerSecret}. Centrifugo itself knows nothing about users and asks the application on connect (connect proxy). ConnectService answers with user, the player's public id derived from the secret (no secret means anonymous), a list channels: ['hellowin'], which is a server-side subscription, and the connection's meta (the moderator role, see “Moderator, characters, flyers”). From then on Centrifugo puts user into every RPC itself: the client does not send its id and cannot pose as someone else. The client never calls subscribe: client-side subscriptions and publications are disabled in config.json, and channel messages arrive in centrifuge.on('publication').

The RPC state returns the same snapshot. Every channel event carries round: if it does not match the client's round, the client has missed a reset and requests the snapshot again; the same happens after a reconnect, when the events sent during the outage are lost.

What happens on a click

A click is not an HTTP request but an RPC over the same WebSocket: centrifuge.rpc('click', {element: 'pumpkin-3'}). Centrifugo proxies it to RoadRunner, which hands the call to a free worker from the centrifuge pool, where RPCService passes it on to ClickHandler::click().

1. The element is checked against ElementRegistry (ids from app/config/hellowin.php).
2. RitualMetrics adds 1 to hellowin_clicks_total.
3. RitualStore::click(), under the lock pumpkin:{id}, increments the pumpkin's clicks in KV (and accepted, the round's accepted clicks, for the reveal in the finale) and right away, under the same lock, reads the round and the scene. The lock covers only this pumpkin: clicks on different pumpkins do not wait for each other.
4. PlayerService, under the lock player:{id}, adds the click to the stats of the player, the connection's user; an anonymous click skips the stats. A stats failure is logged and does not break the click.
5. ChannelPublisher publishes a click event to the channel hellowin (with player unless the click is anonymous), to all subscribers at once.
6. LockstepPolicy::pumpkinTransition() decides whether the pumpkin should move on: clicks is at least the pumpkin's threshold (the pumpkin limit) and the stage is below the current scene's stageMax. If so, TransitionQueue queues a TransitionPumpkin job, and hellowin_transition_jobs_pushed_total{kind="pumpkin"} grows by 1.
7. The RPC response is always data: {ok: true} or {ok: false, error: 'unknown element' | 'busy' | 'internal error'}; busy means the pumpkin's lock could not be taken within 5 seconds.

The click itself does not change the stage, it only queues a job. Until the job is done, clicks stays above the threshold, and every next click queues one more identical job. The duplicates are intended: idempotency cuts them off, and the clicks above the threshold are burnt. Details are on the tab “Idempotent transition”.

The browser does not draw its own click locally. The effect appears only when a publication comes from the channel, for the one who clicked just like for everyone else. That is why the screen shows the honest round trip delay browser → server → browser.

Channel messages are JSON strings. The click event:

{"type":"click","round":…,"element":"pumpkin-3","emoji":"🕸️","clicks":…,"threshold":…,"stage":…}

Scenes, pumpkin stages and locks

The engine knows only numbers: round, scene and, for each pumpkin, {stage, clicks}. What it looks like is up to the frontend. When things move is up to the policy — the RitualPolicy interface, implemented by LockstepPolicy: one spot in the code that can be replaced without touching the engine.

• Scenes are numbered from 0 to the final N — the length of the HELLOWIN_SCENE_STAGE_MAX list.
• Scene s has a stage cap stageMax(s) — element s of the same list, cumulative; the finale has the same cap as the scene before it.
• If the list starts with 0, scene 0 is the bar: there are no pumpkins (clicking them returns locked, RitualPolicy::bar()); the scene is ready when the crowd has drunk the bar target or the moderator gives the go-ahead, and the moderator leads everyone out to the pumpkin patch. Details in “The bar”.
• A pumpkin moves to stage + 1 once it collects the click threshold — the pumpkin limit (“Limits”) — while stage < stageMax(scene); the transition resets clicks to zero.
• When all pumpkins reach stageMax, the scene is ready but does not switch by itself: a ready event, the owl says “all set”, and the scene waits for the moderator. The “Next scene” button on the control panel (RPC scene.next) starts the next one; if the crowd is stuck, the moderator can switch an unready scene too (force). Details in “Scene transition”.
• In the final scene pumpkins do not transition; clicks are only broadcast. The moderator walks through the finale steps, and after the reveal the endless “cursed world” begins.

One player on the field, multiplier 1: the threshold is 2 clicks per stage, and this is how a pumpkin goes through a scene:

0. 0round start
1. 1after 2 clicks
2. 22 more
3. …2 each
4. stageMaxwaits for the rest

Limits: base, players, multiplier

A pumpkin threshold is not one number for every case: one player clicks through it in a minute, forty in a second. So every number in the ritual — clicks per stage of a pumpkin, the bonfire and the circle, the bar target, both cursed-world targets, FrankenPHP's health — is computed by one formula from the number of players on the field:

limit = max(1, ⌈k × (BASE + PER_PLAYER × (n − 1))⌉) + bonus

• BASE and PER_PLAYER are the element's env vars HELLOWIN_<X>_BASE and HELLOWIN_<X>_PER_PLAYER, X being PUMPKIN, BONFIRE, CIRCLE, BAR, CURSED_DRINKS, CURSED_ZOMBIES, BOSS_HP: how much one player needs and how much each additional player adds. The formula leaves the number of stages alone — only clicks per stage.
• n — characters on the field, at least one. LimitsService counts them with a ticker once a second and accepts a new n once it has held for LimitsService::DEBOUNCE: someone joins and leaves — the limits don't twitch.
• k — the moderator's multiplier (RPC thresholds), one for all limits: fits the ritual to the crowd on the fly.
• Bonus — manual, from the control panel: the bar target, the cursed-world targets, the boss's health. It lasts until the end of the round; click thresholds have none.

When any input changes, LimitsService::settle compares the current scene's limits with the last announced ones and, if they differ, publishes a limits {players, k, burst, limits} event: browsers update their thresholds, the control panel shows the resulting numbers. A limit that drops below the accumulated count takes no progress away: the server itself queues the stage transition (the same idempotent TransitionPumpkin as after a click) or marks the bar and the cursed world ready, without waiting for a click. FrankenPHP keeps its damage: hp = max(1, max − damage).

The same player count grows the flyer wave the moderator launches with the “Launch flyers” button (RPC flyers.burst): ⌈HELLOWIN_FLYER_BURST_BASE + HELLOWIN_FLYER_BURST_PER_PLAYER × (n − 1)⌉ flyers of the current set at once. No cooldown, no cap: as many presses, as many waves.

Why lock a pumpkin

Clicks are handled by several workers in parallel (the centrifuge pool has two), and many people click the same pumpkin. Incrementing the counter is read-modify-write: read {stage, clicks}, +1, write; the KV plugin has no atomic increment. Without a lock two workers read the same value, and one click is lost.

Locks are RoadRunner's lock plugin; each operation has its own set. TTL is 5 seconds — the lock expires by itself if a worker dies; waiting for a held one — up to 5 seconds, then RitualBusyException.

The bar: the last mug and pouring in batches

The evening starts in the bar (scene 0): no pumpkins there, but a counter where the owl bartender pours beer. The bar shows two more mechanics — the same lock and KV, but on mugs.

• The last mug goes to whoever comes first. A click on the counter (RPC bar.take) only sends the character there and writes the arrival time into its record. When that moment comes, the ticker, under the bar lock, checks in one step whether a mug is in hellowin:bar and takes it off — like “is the flyer alive” for a shot. Two arrive almost at once — the lock lines them up: the first leaves with a beer (take with ok: true), the second finds an empty counter (ok: false, the owl: “all gone”). There is no queue — the latecomer clicks again.
• Pouring in batches. While the counter has free places (one per character on the field, up to HELLOWIN_BAR_MUGS), the owl pours the next batch for HELLOWIN_BAR_POUR_MS ms: a mug for every HELLOWIN_BAR_POUR_PER free places, at least one — a big crowd doesn't wait for beer one mug at a time. The batch is decided when the pour starts: KV holds pouring {since, until, batch} by server time, a RoadRunner ticker every 200 ms sets down the poured mugs (mugs, no more than the places) and starts the next batch (pour). Everyone sees the same foam, and restarting the workers or the ticker itself loses nothing: the state lives in Valkey, not in process memory.
• You can drink only from a mug that has clinked (drink, otherwise not clinked); drinks are counted per round, and once the crowd has drunk the bar target (the moderator adjusts it), the bar is ready and the owl calls everyone to the pumpkin patch. Leaving the bar is a plain scene.next: the mugs vanish, the characters move to the lawn in one scene event, without walking.

The rapier: the magic word, curing, one boss for the whole crowd

The end of the night shows three more mechanics — again a lock, KV and a server that decides on its own.

• Only the server checks the word. The client sends what the player typed (RPC word) and learns only the verdict. The server normalizes the input — lowercase, Cyrillic to Latin, only a-z — and compares it with the answer, forgiving typos (prefix, Levenshtein distance, metaphone). Neither the answer nor the rules are in the browser. Wrong — the character turns into a zombie, and the server's ticker walks it around.
• Cure first. A click on a zombie player (RPC cure) under the walkers lock lifts the spell — the same “check and change in one step” as a shot at a flyer: the second clicker sees the zombie is already gone.
• One boss for the whole crowd. All rapier strikes (RPC strike) go to one record hellowin:boss under one boss lock, like clicks on the bonfire: the crowd competes for one resource, and exactly one strike takes the last point of health. That strike queues the transition to dawn with the idempotency key {round}:finale:7.

Round and reset

A reset starts the ritual over: scene 0 (the bar), every pumpkin at stage 0 and clicks 0, round + 1, the story from the start (the progress pause is lifted, the multiplier k stays) and a reset event to everyone. This is done by ResetService::reset(?int $fromRound), called by the “Restart” button on the moderator's control panel (RPC reset {round} — the round the moderator sees) and by the CLI command hellowin:reset [--round=N]. The service takes the scene lock and the locks of all pumpkins, so neither a click nor a transition sees a half-reset ritual, and under them compares the current round with the one being finished. A second reset of the same round sees the new round already and does nothing — idempotency by reset:{round} without a separate key: the state is already under these locks, and a key in Valkey would be a second source of truth about the round.

Idempotent transition

A pumpkin hit its threshold — exactly one transition to the next stage is needed. But clicks don't stop: while the job sits in the queue waiting for a worker, the pumpkin gets clicked n more times, and each click queues TransitionPumpkin again. This is on purpose. A click doesn't need to know whether someone queued the job before it: no “transition in flight” flag, and no resetting it if the job got lost (the queue is in memory; on an app restart jobs vanish — the next click simply queues a new one). The price of simplicity is k + n jobs per transition, and only one must run.

Duplicates are cut off by the spiral/idempotency package. The job class carries an attribute, and IdempotencyInterceptor in the queue's interceptors.consume chain (app/config/queue.php) takes the key from the payload:

#[Idempotent(storage: 'transitions', key: 'payload.key')]
final class TransitionPumpkin extends JobHandler

Who makes up the key

The server builds the key from the meaning of the operation: TransitionQueue::pumpkinKey() — {round}:{pumpkin}:{toStage}, e.g. 1:pumpkin-3:2 — “in round 1 pumpkin 3 moves to stage 2”. All clicks that queue this transition get the same key independently of each other. This differs from the familiar Stripe-style Idempotency-Key:

In Valkey the key gets the store prefix and a scope — the handler name, so that keys of different jobs don't collide: hellowin:amo:App\Application\Hellowin\Job\TransitionPumpkin::handle:1:pumpkin-3:2.

How a duplicate is cut off

The transitions store in app/config/idempotency.php is RedisAtMostOnceConfig with the prefix hellowin:amo: and retentionTtl: 300. Before calling the handler, the driver claims the key with one atomic command SET {key} M:{token} NX EX 300. Written — it's the first delivery, the handler runs. The key already existed — a duplicate: the handler isn't called at all, the interceptor returns null, and the job is acknowledged (ACK) — it doesn't fail and doesn't go back to the queue. The marker is deleted neither on success nor on error; it expires by TTL on its own.

Why “at most once”

The package has three guarantees, and they are about different things:

ExactlyOnce is out of reach here: the effect is a KV write through a RoadRunner plugin plus a publication to Centrifugo, and they share no transaction with the marker. Of the remaining two, a double transition is worse: the pumpkin would skip a stage, and the pumpkin event would go out twice. Losing a transition is cheap — the pumpkin stays at the threshold, and people click it on through. Hence AtMostOnce.

The price: if a worker dies between the marker and the stage write (or the pumpkin lock stays held longer than 5 seconds), the transition is lost, and new jobs with the same key are cut off while the marker lives — the pumpkin “gets stuck” at the threshold. So the window is short: retentionTtl is 300 seconds, not the default of a day. After it the next click starts the transition again, and a double transition is prevented by the domain check under the lock — stage == toStage − 1: if the transition did in fact go through, the retry changes nothing.

Scene transition

A scene doesn't switch by itself: the game isn't timed, the moderator runs it. The transition is split into two steps, and each is protected from double execution in its own way.

1. Readiness. After each pumpkin transition TransitionHandler asks the policy whether all pumpkins have reached stageMax (RitualPolicy::sceneReady()). The last pumpkins can arrive almost at once, and then several workers hear “yes”. The mark is set by RitualStore::markReady() under the scene lock: if in the story hellowin:story the ready field isn't this scene yet, it writes it and returns true; if it already is, false. So the ready event and the owl's lines last-pumpkin (with the player who finished off the last pumpkin) and scene-ready go out exactly once. The scene itself stays the same.
2. Readiness with the bonfire. From scene 2 on, ready pumpkins are not enough: the middle of the patch also has a bonfire, one element for the whole crowd. While the scene's pumpkins aren't ready, clicking it answers locked and goes nowhere; instead of readiness, the last pumpkin opens the bonfire (a mark in the story under the same scene lock, owl line bonfire-open). Then everyone hits one bonfire lock — hence the busy answers — and past the bonfire threshold (the bonfire limit) they queue a handful of duplicates of one TransitionPumpkin job with the key {round}:bonfire:{toStage}; only one runs, as kind="bonfire" shows in the metrics. The scene becomes ready when the bonfire moves to its last stage of the scene (HELLOWIN_BONFIRE_STAGE_MAX) — the same markReady(), lines bonfire-done and scene-ready. Likewise, in the final scene the crowd draws the magic circle (lock circle, key {round}:circle:{toStage}): a closed circle makes the finale ready, and only then does “Finale step” raise FrankenPHP (without readiness — only with force).
3. Transition. The moderator presses “Next scene” — RPC scene.next {scene} with the number of the scene they see. StoryService::nextScene() checks against a snapshot that the scene is current, not final and ready (or force came), and queues TransitionScene. A double click, two control panel tabs or two moderators queue two jobs, and that is the same as duplicate clicks on a pumpkin: the key {round}:scene:{toScene} (TransitionQueue::sceneKey()), the same #[Idempotent] with the AtMostOnce guarantee, and inside — the scene lock, the check scene == toScene − 1 and one scene event with the new scene's stageMax and the characters' item. A press that arrives after the transition sees the scene has changed and queues nothing — it just answers with the current one.

Why the key includes the round

After a reset the ritual goes through the same transitions again: pumpkin 3 heads to stage 2 once more. Without the round the key would match the old one, and while the old marker lives, the new transition would be cut off as a duplicate. With the round it is a new operation: 2:pumpkin-3:2. And jobs from the previous round left in the queue change nothing, even if they get past the marker: the domain check sees the round has changed.

What you see in Grafana

The dashboard's “Transitions: queue and idempotency” section has the panels “Jobs pushed vs transitions applied” (per minute and total, by kind), “Duplicates dropped” = pushed − transitions and “Pumpkin stages”. The harder people click a pumpkin at the threshold, the bigger the gap. You can also see the markers in Valkey itself, along with the remaining TTL:

docker compose exec valkey valkey-cli --scan --pattern 'hellowin:amo:*'
docker compose exec valkey valkey-cli ttl 'hellowin:amo:App\Application\Hellowin\Job\TransitionPumpkin::handle:1:pumpkin-3:2'

Dice: idempotency with a stored response

Before the uprising in the finale, the crowd rolls three dice until someone gets 6-6-6, a PHP version that never existed. The server does the rolling: the client sends the RPC roll {attempt} with its attempt number, and the dice come back in the reply and to everyone in the roll event. The problem is the same as with a pumpkin transition: a repeated request must not cause a second effect. But a duplicate transition can simply be dropped, while a repeated roll needs an answer: the same dice. So the attempt's response is stored, and the retry gets it.

A fair roll

The dice are truly random: DiceRoller calls Randomizer::getInt(1, 6) three times on top of the system CSPRNG, same as random_int. Nobody can predict which attempt will hit 6-6-6, the server included. But the same dice can't be computed a second time either: a roll is drawn once, and the result has to be remembered.

The key and the stored response

The attempt key is {round}:roll:{player}:{attempt}: the round, the player's public id and the attempt number. DiceService::roll runs the roll through the dice store of the spiral/idempotency package, a RedisLeaseConfig lease with the AtLeastOnce guarantee. The first call with a key rolls and stores the response {attempt, dice} for HELLOWIN_DICE_IDEMPOTENCY_TTL s. A repeat of the same key (a double click, a reconnect, a retry after a lost reply) gets the stored response: no new roll, no event, no count. A refusal (busy, not now, bad attempt) is not a response: FailurePolicy::Release frees the key, and the attempt can be rolled once the refusal no longer applies.

The faces of a player's last attempt are also kept in hellowin:dice under the dice lock, in one record together with the attempt number and the time of the next roll. This is a safety net: AtLeastOnce allows a second run if the worker died before storing the response or the response expired. A run under the lock then finds its attempt in the record and returns those faces without rolling again.

• Why a retry doesn't reroll. A retry doesn't roll: it gets the stored response, and if that is gone, its attempt's faces from the record. Only the next attempt brings new dice, and no more often than the HELLOWIN_DICE_COOLDOWN_MS pause: nobody can cycle through attempts faster than the rest. Compare with the pumpkin transition: there it is AtMostOnce (hellowin:amo:), a SET NX marker cuts off the duplicate, which needs no response; here it is an AtLeastOnce lease (hellowin:lease:), and the duplicate gets the same response as the first call.
• The first 6-6-6 is under the lock. Every roll in the room goes through one record, hellowin:dice, under one dice lock, like the strikes on FrankenPHP. The winner is recorded by whoever takes the lock first; a second 6-6-6 sees it and gets not now, like a second shot at a flyer that is already down. The win makes the finale step ready (markReady), and the owl names the winner.
• The owl. The moderator's fallback button dice.owl: the first two rolls are random, and a 6-6-6 that comes up is rerolled; the third is a guaranteed 6-6-6. There is no idempotency key: each press by the moderator is a new roll. These don't count toward the room's attempts (reveal.rolls).

Moderator, characters, flyers

Around the ritual runs a live evening: the moderator drives the story from the control panel, players' characters walk the lawn, birds and spooks fly across the sky, and in the finale the owl lays out the real numbers. Each of these mechanics is one more technique: a role in the connection meta, channel presence, a shared server clock, “first under the lock”, a count derived from state without extra writes.

The moderator role

The moderator opens the page with ?moderator=<secret>; the page puts the secret into localStorage, removes it from the address and passes it in the connection data next to the player secret. ConnectService compares it with HELLOWIN_MODERATOR_SECRET (hash_equals) and on a match puts {moderator: true} into the connection meta. Centrifugo keeps the meta, the client can't see it, and Centrifugo adds it to every RPC proxy call itself, as configured by rpc.proxy.include_connection_meta in config.json. The client sends only data in an RPC and can't grant itself a role by request. With an empty secret in env, nobody has the role.

RPCService reads the role from meta, and for control panel methods (scene.next, finale.step, owl.say, flyer.special, flyers.burst, pause, thresholds, reset) answers forbidden without it, before parsing the data. thresholds {k} multiplies all limits by k (“Limits”) to fit the ritual to the room on the fly; k lives in the story and survives a reset. pause {on} pauses progress: the scene's goal freezes while the room carries on. RPCs that advance the goal (click on pumpkins, the bonfire and the circle, drink, strike, roll) are not refused: the action goes through and the event reaches everyone, but with counted: false: clicks and stages hold, mugs don't count toward the goal, the boss's health stays the same, 6-6-6 doesn't win, and the scene never becomes ready. Players don't see the pause; it shows only on the control panel. Flyers, zombies and shots are not affected by the pause. Lifting the pause catches up through the same recount as a lowered limit, and a reset lifts it by itself. The role is not in user: the moderator can also be a player with their own secret, and user is the player's public id in events and presence. Every control panel method is safe to repeat: StoryService changes the story hellowin:story under the scene lock, compares “before” and “after”, and publishes the event (and an owl line) only if something changed.

Characters on the lawn: presence and the ticker

A player who has built a character on the entry screen (RPC register with a name and look) is a character on the lawn. The server doesn't track who is on the page by itself: the hellowin channel has Centrifugo presence enabled ("presence": true in config.json, closed to clients). Every connection is subscribed to the channel by the server, so presence is every open tab with user = the player's public id, and an entry disappears on its own as soon as the connection closes.

Once a second, the ticker (WalkService::sync()) asks for presence over the Centrifugo gRPC API (CentrifugoPresence), reduces the entries to a set of players and, under the walkers lock, compares it with the hellowin:walkers key: present players get their seenAt updated, players with a look but no character step onto the lawn (the walker event), and characters whose players have been gone longer than HELLOWIN_WALK_LEAVE_AFTER ms leave (leave). Several tabs mean several entries for one user, but there is one character, and it leaves when the last tab closes. The leave delay keeps a page reload from dropping the character or moving it to the edge.

Why not the connect proxy: at connect time the client isn't subscribed to the channel yet and would miss its own walker event. Why not the refresh proxy with connection renewal: a departure would show up only when the term expired, and every renewal is an extra call and a KV write. Presence is a ready-made record of connections that Centrifugo cleans up itself on disconnect. A failed presence poll is logged, and the characters stay: an unavailable presence doesn't mean everyone left.

The server plans the character's path

A click on the ground is only an intent: the RPC move {x, y}. Nobody sends coordinates frame by frame. WalkService::move() clamps the target into the zone, the lawn rectangle walk.area minus the patch ellipse walk.patch from app/config/hellowin.php (a click inside the patch goes to its edge), starts from where the character is right now according to its current path and the server clock, and builds a polyline: straight if the segment misses the patch, otherwise along a tangent to the ellipse enlarged by 3 %, along its arc and along a tangent to the target, on whichever side is shorter. The duration is the polyline length divided by HELLOWIN_WALK_SPEED. Under the walkers lock the path is written to hellowin:walkers, and the channel gets a move event with Walk {path, startedAt, arrivesAt}.

The server sets the animation, not the clicking player's client. From now in the snapshot, each browser knows its clock offset from the server's and places the character at the point on the path where it should be at that moment of server time. So everyone sees the same movement, and a tab opened mid-walk immediately draws the character halfway along. A new move during a walk starts from the current point.

Clinking mugs on click in the cursed world is the same path plus a deferred check: the RPC clink walks the character to the target and puts next to it in hellowin:walkers a record “waiting to meet by dueAt”, and every 200 ms the ticker, under the walkers lock, takes the records that are due and, if the target hasn't left, publishes exactly one clink event with a toast and the meeting time.

Flyers and zombies: points go to the first

The ticker launches flyers: at random once per HELLOWIN_FLYER_INTERVAL ms, the kind taken from the current scene's set HELLOWIN_FLYER_KINDS, occasionally a rare one (HELLOWIN_FLYER_RARE); the moderator can also release a special one by button or a whole wave (“Launch flyers”, “Limits”). The server sends the trajectory ready-made: a flyer event with points, depth, score and the launchedAt / expiresAt times, and in KV a hellowin:flyer:{id} key with a TTL plus an index of flyers in the air. Zombies in the cursed world are flyers too (kind: zombie, with their own interval HELLOWIN_ZOMBIE_INTERVAL), they just walk across the lawn from the edge of the patch outward, so the shooting, points and metrics are shared. A progress pause doesn't stop the ticker: flyers and zombies keep moving and a hit is credited to the player, only a zombie downed during the pause doesn't count toward the cursed world's goal.

A click on a flyer is the RPC shoot {flyer}. Under the flyer:{id} lock: if the key still exists, the flyer can be shot and its flight isn't over, the key is deleted and it's a hit: points to the player, one shot event, a metric. The next shot won't find the key: a miss. On every tick the ticker accounts for escaped flyers the same way: if the key still exists, delete it and count it as escaped. Exactly one of them deletes the key, so a flyer is never both shot and escaped. The server accepts a shot until expiresAt + 500 ms, and the client doesn't draw a flyer whose time has run out by the server clock.

Points add up to the evening's top shooters: hellowin:top under the top lock (TopService). A hit doesn't write “+N” there but the player's absolute score returned by their record, taking max(old, new): a retry or a race can't count twice. The shot itself publishes nothing: the ticker sends the board as a top {board} event when it has changed, and no more often than HELLOWIN_TOP_INTERVAL ms. This is coalescing: with a full room, the full list to everyone on every hit is too much, while once per interval it is one. The board lives all evening, a reset doesn't clear it, and it is also in the page snapshot.

The reveal: the evening's numbers

At the finale step where the owl reveals the truth, the finale event and the snapshot carry reveal, the round's real numbers. There are no separate per-click counters for this: everything follows from the state in Valkey, which is written under locks anyway. RevealBuilder recomputes them on every build.

A round reset zeroes these numbers without dedicated code: accepted, stage, scene and the story are reset along with the ritual, and the shot count is tied to the round.

Metrics

When a worker starts, HellowinBootloader declares the collectors in the metrics plugin and sets the gauges from the state in Valkey, which survives a RoadRunner restart. After that, RitualMetrics calls metrics.Add and metrics.Set over RPC from the click, job push, transition and reset, and PlayMetrics from the shot and the ticker (flyers, characters, players). A metrics error is only logged: a click never fails because of it. RoadRunner serves everything on :2112/metrics along with its own rr_http_* (the http_metrics middleware in .rr.yaml; also there is gzip, which compresses both PHP responses and static files; it sits before static because RoadRunner 3 runs middleware left to right).

Every 5 seconds Prometheus scrapes app:2112 and centrifugo:9000/metrics (Centrifugo's internal port: only the WebSocket on 8000 is exposed). Grafana gets the datasource and the “Hellowin” dashboard through provisioning and lets visitors in anonymously as Viewer: anyone can look, nobody can edit.

Counters live in the RoadRunner process and reset when it restarts; gauges are restored from Valkey. Dashboard sections: Game: clicks, scene, round, online; Transitions: queue and idempotency: jobs versus executed transitions, dropped duplicates, pumpkin stages; Centrifugo: connections, messages, proxy calls; RoadRunner: HTTP and workers.

Queries from the dashboard, for example:

# clicks per second, total and per pumpkin
sum(rate(hellowin_clicks_total[1m]))
sum by (element) (rate(hellowin_clicks_total[1m]))

# dropped duplicates: jobs in the queue minus executed transitions
sum by (kind) (hellowin_transition_jobs_pushed_total) - sum by (kind) (hellowin_transitions_total)

# p95 latency of Centrifugo → RoadRunner proxy calls
histogram_quantile(0.95, sum by (le, type) (rate(centrifugo_proxy_duration_seconds_histogram_bucket[1m])))

Open the dashboard in Grafana · Back to the pumpkins