The question the model kept repeating

A student hit "generate more practice questions" on the same topic four times in one sitting. The support ticket that followed included a screenshot: batch one had a question about balancing a chemical equation with iron and oxygen. Batch three had the same question, same numbers, one word changed.

The model has no memory of what it already produced for that user. Every generation call starts from zero, so from its point of view, batch three wasn't a repeat of anything. It was just a good question about iron and oxygen, again.

The bug: we were asking the model to be original. Originality isn't something a model can check for on its own, it's a property relative to a specific history, and we weren't giving it the history.

Take 1: ask harder for originality

“Generate fresh, original questions the student hasn't seen before, avoid repetition.” No actual data about what the student had seen, just the instruction.

This didn't reliably work, and it couldn't have. The model has nothing to check the new question against. Telling it to avoid repeating something it has no record of is like telling someone to avoid a word without saying which word. It can vary phrasing and still land on the same underlying question, same numbers, same setup, because nothing in the prompt distinguishes that specific question as off-limits.

Take 2: give it the actual list

The fix was to stop asking for a property the model can't verify and start giving it information it can act on. Before generating, pull the student's recent question texts on the same topic and put them directly in the prompt as things to avoid.

const MAX_QUIZZES_BACK = 5;
const MAX_STEMS = 15;
const MAX_STEM_LENGTH = 140;

async function getRecentStems(
  userId: string,
  topicId: string,
): Promise<string[]> {
  const recentQuizzes = await db.quizzes.find({
    userId,
    topicId,
    order: "createdAt DESC",
    limit: MAX_QUIZZES_BACK,
  });

  const stems = recentQuizzes
    .flatMap((quiz) => quiz.questions.map((q) => q.stem))
    .slice(0, MAX_STEMS)
    .map((stem) =>
      stem.length > MAX_STEM_LENGTH
        ? stem.slice(0, MAX_STEM_LENGTH) + "..."
        : stem,
    );

  return stems;
}

The lookback is bounded on three axes: a fixed number of recent quizzes, a fixed max number of stems pulled from them, and a fixed max character length per stem. Without those caps, a student who's generated forty quizzes on the same topic would balloon the prompt into something slow and expensive, most of it irrelevant to what they just saw five minutes ago.

The prompt then includes the list directly: "Avoid regenerating anything resembling these existing questions," followed by the stems. Now the model has something concrete to diff against instead of a vague instruction to be creative.

What I should have done first

There's a difference between telling a model what you want and giving it what it needs to get there, and I conflated the two for longer than I should have. "Be original" is a want. A list of forbidden stems is a need. The model can't act on the first one because it has no way to evaluate it against anything.

The fix ended up being less prompt engineering and more plumbing: a bounded database query. That's usually the tell that I was trying to solve a data problem with words.