The reward that fired twice
A challenge creator got a bonus for winning. They also got it for winning again, on the same challenge, because a retry doesn't know it's a retry.
By Meet ModiA challenge creator gets a bonus in in-app currency when their opponent's score crosses a threshold. Most creators got the bonus once. A few got it twice, on the same challenge, for the same win.
Nobody replayed anything on purpose. The completion handler just ran more than once.
The bug: the award logic trusted that completion only happens once. Completion doesn't work that way.
Take 1: award inline, every time
This ran on a background task queue. Background tasks in this system get retried after a crash, and the mobile client separately retries its own completion call after a timeout. Both paths call the same handler.
First run: score check passes, wallet gets incremented, status flips to completed. Second run, triggered by a retry a few seconds later: score check passes again, because nothing about the challenge row says this already happened. The wallet gets incremented a second time.
Take 2: a flag that lives with the award
The fix wasn't to stop retries from happening. Retries are the reason the system is reliable when a task crashes mid-flight. The fix was to make the award itself remember it already ran.
async function onChallengeCompleted(challengeId: string) {
const challenge = await db.challenges.findById(challengeId);
if (challenge.opponentScore < challenge.threshold) return;
const result = await db.challenges.updateOne(
{ id: challengeId, creator_xp_awarded: false },
{ $set: { creator_xp_awarded: true, status: "completed" } }
);
if (result.matchedCount === 0) {
// Already awarded on a previous run. Nothing to do.
return { xpGranted: 0 };
}
await db.wallets.increment(challenge.creatorId, {
xp: challenge.bonusXp,
});
return { xpGranted: challenge.bonusXp };
}The match condition creator_xp_awarded: false and the flag flip happen as one write against the challenge row. A retried handler runs the same update, matches nothing because the flag is already true, and returns a zero delta instead of touching the wallet.
What I should have done first
I should have asked, before writing the handler, how many times this function can run for the same challenge. The answer was never zero or one, it was always "at least once, possibly more," and every retry-based system answers that question the same way.
Once I started asking that question first, the flag-on-the-record pattern stopped feeling like a fix and started feeling like the default way to write this kind of function.
More Posts
The vector database was innocent
I built a RAG service over the OpenTelemetry docs, then pointed OpenTelemetry back at it to find out why answers took 16 seconds. It wasn't the LLM. It wasn't the vector search either.
A query that returned nothing nested inside it
The exact same logical query returned full nested data through the REST layer and empty shells through the service layer underneath it. Same fields requested. Same backend. One shorthand parameter that only one of the two actually understood.
One field, three different envelopes, and a fix that made it worse
The same subscription-status field came back nested two levels deep, one level deep, or bare at the top, depending on account state. My fix didn't catch that. It added math on top of a check that was already wrong.