The moderator that voted against you

One of the apps we build lets friends make bets and hands the judging to AI moderators — a few LLM “personalities” that read the chat, check the real-world outcome, and vote on who won. When enough of them agree, the bet resolves and the stakes move. It worked well, until we noticed it occasionally paid out the wrong side.

Not randomly, and not a hallucination. The model's written reasoning, in the logs, was correct every single time — it knew who won. It voted for the loser anyway, because it did exactly what we asked. That's the part worth sitting with, because it's the thing about building on LLMs that bites hardest: the model is a literalist. It doesn't reverse-engineer what you meant. It executes the frame you gave it — and if that frame carries an assumption you forgot you'd made, the model carries it too, confidently, without a flicker of doubt. Ours was sitting quietly, correct-looking, waiting for one product change to detonate it.

What a wrong vote actually costs

Worth being precise about the blast radius, because it's what makes this more than a display glitch. A bet has two sides. Each AI moderator casts a vote; when a majority agree on a side, the bet auto-resolves and balances settle — losers owe winners, for real. A moderator that votes for the wrong side isn't cosmetic. It moves money to the people who lost.

The frame with an assumption in it

Here's the original shape of the vote. We handed the model the two sides and asked which one won — but we framed them by role. Side A was “the admin's side” (the person who created the bet); side B was “the challenger's.” The vote came back as admin or challenger, and we stored it by mapping those to positions: admin → Sides[0], challenger → Sides[1].

That mapping is correct — as long as the admin is always on Sides[0]. And for a long time they were; the creator always took the first side. So the assumption was invisible, load-bearing, and untested.

Then we shipped a feature that let a bet's creator pick either side. The moment an admin sat on Sides[1], the role-to-position map was off by exactly one side. The model would correctly decide the challenger had won; we'd faithfully store that as challenger → Sides[1] — but the admin was on Sides[1], so “the challenger's side” and “position 1” were now the same side. Every vote in that configuration inverted.

The model never once questioned whether “the admin's side” was a safe stand-in for a slot in an array. Why would it? We told it to reason about the admin, so it reasoned about the admin — precisely, obediently, and into a wall. It was speaking a language about people; our database was listening in a language about positions; and a product change had quietly swapped the dictionary between them.

The fix: leave nothing to be literal about

The instinct is to call this a mapping bug and patch the mapping. But the mapping was only ever a symptom. The real defect is that we gave the model a convenient label — “the admin's side” — that it could satisfy literally without ever engaging with the question we actually cared about. So we took the label away. Strip the roles out entirely and there's nothing left to be literal about except the content itself:

// SideA/SideB are POSITIONAL (Sides[0]/Sides[1]) — NOT the admin's vs
// challenger's side. The admin can be on either side now, so the old
// "admin's side = Sides[0]" framing made the model vote the opposite side
// when the admin was on Sides[1]. The model picks the side whose outcome
// occurred, by content, and we map A → Sides[0] / B → Sides[1] afterward.
voteRequest := ai.VoteRequest{
    Title:          bet.Title,
    Description:    bet.Description,
    SideA:          bet.Sides[0],   // just "A"
    SideB:          bet.Sides[1],   // just "B"
    RecentMessages: recentMessages,
    // …memory doc, stake, participant count
}

No admin, no challenger, no “your side.” The model reads two neutral options and the evidence and has to commit to which outcome actually happened. Whose bet it is stopped being its problem — and ours. The model is exactly as literal as before; we just made the only thing it can take literally be the truth.

Map after, and refuse anything you can't place

With identity gone, mapping the answer back is boring on purpose — A is position 0, B is position 1 — with one rule that matters more than it looks:

// "A" → Sides[0], "B" → Sides[1]; an exact side-name answer is honored too.
// Anything else is a FAILED vote, not stored junk — a junk value would count
// toward the quorum but match no side, wedging the tally permanently.
actualSide := ""
switch strings.ToUpper(strings.TrimSpace(vote.VotedSide)) {
case "A":
    actualSide = bet.Sides[0]
case "B":
    actualSide = bet.Sides[1]
default:
    for _, s := range bet.Sides {
        if strings.EqualFold(strings.TrimSpace(vote.VotedSide), s) {
            actualSide = s
        }
    }
}
if actualSide == "" {
    return fmt.Errorf("AI vote %q did not map to a side of bet %s", vote.VotedSide, bet.BetID)
}

The temptation with an LLM is to keep whatever it says — it's usually reasonable. But this vote feeds a quorum. A value that matches no real side doesn't get politely ignored downstream; it counts toward the total votes cast while never contributing to any side's majority, so the bet can sit one vote short of resolution forever. A creative answer stored is worse than a loud failure. We fail the vote, mark that moderator failed, and let the others decide.

The bugs hiding behind the bug

Fixing the inversion meant looking hard at how votes are tallied, and the quorum turned out to be its own minefield. Three more fixes came out of the same investigation:

• A majority of one. Human and AI moderators lived in separate tables, and the tally originally only saw the humans — so a single human vote was a “majority of 1” that resolved the bet before the AIs had even voted. We now merge them into one voting body and require a majority of everyone.
• The duplicate vote. When an AI votes, it also writes a row into the human-moderator table (so it shows up in listings). Counted naively, every AI voted twice. The merge de-dupes, representing each AI exactly once from its authoritative record.
• The vote that never comes. An AI moderator can fail — an API timeout, a refusal. Count a failed moderator toward the quorum and you're waiting on a vote that will never arrive, and the bet is wedged unresolved forever. Failed moderators are excluded from the count.

for _, ai := range aiModerators {
    if ai.Status == "failed" {
        continue // never going to vote — counting it would wedge the quorum
    }
    combined = append(combined, &models.Moderator{
        ModeratorID: ai.AIModeratorID, IsAI: true, Vote: sideOf(ai.VotedSide),
    })
}

There's a fourth, quieter one: the vote runs off a stream event that can be redelivered, and a long processing budget widens that window. So before an AI votes, we check whether it already has — re-voting would double-count and re-bill a paid web search.

The lesson

The model was never wrong. It was obedient to a frame that had a bug in it — and that's the whole point. An LLM does not infer your intent; it takes your structure literally. Correctness isn't something the model supplies on top of a loose prompt and good vibes; it's something you engineer into the prompt by leaving no room for a convenient interpretation.

Three things generalize past this app:

• Don't encode an assumption in a label the model can satisfy without checking. Ask about content — which outcome occurred — not about who, so a change in who's-who can't silently repaint the answer. Make the model be deliberate about the one thing that's actually true.
• Validate the answer against a closed set of what it's allowed to say, and fail loudly on anything you can't map. Do not store the model's improvisation and hope.
• In a voting or quorum system, a bad value isn't dropped — it's counted. The safest thing an uncertain moderator can do is not vote at all.

The uncomfortable version: the model will be exactly as careful as your prompt forces it to be, and not one notch more. If being right depends on the model quietly doing the sensible thing you didn't ask for, you don't have a feature — you have a bug with a delay on it.