The quiz where the answer was always C

Someone on the team ran a quick check across a batch of auto-generated quizzes: about 4,000 multiple-choice questions, four options each. If placement were random, each option slot should hold the correct answer roughly 25% of the time. Option C held it 41% of the time.

A student who just picked the third option every time, without reading a single question, would have scored better than guessing should allow.

The bug: the model generating the questions had a habit of putting the correct answer in the same spot, regardless of what the prompt asked for. Prompting isn't code. It's a suggestion with a compliance rate.

Take 1: tell it to vary the position

The first fix was the obvious one: add explicit instructions to the prompt telling the model to randomize which option holds the correct answer.

const prompt = `Generate ${count} multiple-choice questions about ${topic}.
Each question must have exactly 4 options.
IMPORTANT: Randomize which option (A, B, C, or D) contains the
correct answer. Do not favor any position. Vary it across questions.
Return JSON: { question, options: string[], correctIndex: number }`;

This helped. The skew dropped from 41% on option C down to around 33%. Still nowhere near the 25% you'd expect from real randomness, and still enough for a pattern-matching student to notice over a few weeks of daily quizzes.

The instruction competed with whatever positional habit the model picked up during training, and the habit kept winning a third of the time.

Take 2: shuffle it ourselves

The fix that actually worked didn't touch the prompt at all. After generation, shuffle the options server-side and remap the correct-answer index to match. The model's positional bias becomes irrelevant because the position it chose never reaches the student.

function shuffleAndRemap(question: {
  options: string[];
  correctIndex: number;
}): { options: string[]; correctIndex: number } {
  const indices = question.options.map((_, i) => i);

  for (let i = indices.length - 1; i > 0; i--) {
    const j = Math.floor(Math.random() * (i + 1));
    [indices[i], indices[j]] = [indices[j], indices[i]];
  }

  const options = indices.map((i) => question.options[i]);
  const correctIndex = indices.indexOf(question.correctIndex);

  return { options, correctIndex };
}

The shuffle is a standard Fisher-Yates on the option indices, not the strings, because we need to track where the original correct answer ended up. indices.indexOf(question.correctIndex) finds the new home of the option that used to live at the old correct index.

We verified it with a test that generates a large sample of questions, runs them through the shuffle, and asserts the distribution of correctIndex across all four slots stays within a few percentage points of 25% each. That test would have caught the original bug in about thirty seconds instead of however many weeks it took a human to notice a pattern by eye.

What I should have done first

I trusted the prompt because the model is good at language and this felt like a language problem. It isn't. Position bias is a distributional property of the model's training, and no amount of asking nicely rewires that. The moment something needs to be exactly uniform, it belongs in code that doesn't have opinions.

The tell, in hindsight, was that I measured the bias by eyeballing a spreadsheet before writing any test. If I'd written the distribution assertion first, take 1 would have failed it immediately and I'd have skipped straight to the shuffle.