Building the quiz before anyone presses play

A student enrolled in six classes opens one daily quiz on a typical day, sometimes two, rarely more. Generating a fresh quiz through the language model costs real money per call and takes a couple of seconds. Multiply six classes by every enrolled student by every school day and most of that spend produced a quiz nobody ever opened.

The alternative, generate strictly when a student opens the quiz, meant the one moment this feature is supposed to feel instant, it made them wait. A multi-second spinner on something called a "daily quiz" defeats the point of it.

The bug: we treated "prepare the quiz" and "generate the quiz" as the same step. They aren't. One is cheap and can happen for everyone. The other is expensive and should only happen for the one class a student actually opens.

Take 1: generate everything, up front

The first version ran a background job every morning that called the language model for every eligible class, for every enrolled student, before anyone had opened anything. It worked, in the sense that quizzes were always ready instantly. It also meant paying full generation cost for the four or five classes out of six that a given student would never open that day, every single day.

There was no cheap way to fix this while generation and preparation were the same step. You either paid for everything or made someone wait.

Take 2: split prepare from generate

The fix separates the two. A background job runs early and assembles the shell of each day's quiz for every eligible class: which topics apply, which material is in scope, pulled from data we already have and don't need the model for. That part is cheap and deterministic, so it's fine to do it for everyone regardless of who opens what.

async function assembleDailyQuizzes(date: string) {
  const eligibleClasses = await getEligibleClasses(date);

  for (const cls of eligibleClasses) {
    try {
      const topics = await resolveTopicsForClass(cls, date);

      await db.quizzes.upsert({
        classId: cls.id,
        date,
        topics,
        status: "assembled",
        questions: null, // generated lazily on first open
      });
    } catch (err) {
      // Assembly can retry on the next run. Don't let one class's
      // failure block the rest of the batch.
      logger.warn("assembly failed, will retry next run", {
        classId: cls.id,
        err,
      });
    }
  }
}

async function openQuiz(classId: string, date: string) {
  const quiz = await db.quizzes.findOne({ classId, date });

  if (quiz.status === "assembled") {
    const questions = await model.generateQuestions(quiz.topics);
    await db.quizzes.update(quiz.id, { questions, status: "generated" });
    return questions;
  }

  return quiz.questions;
}

The assembleDailyQuizzes job never touches the model. It builds the shell, status: "assembled", questions: null, and stops. The actual model call lives in openQuiz, and it only fires the instant a student presses play on that specific quiz. Five unopened classes stay in the assembled state forever and cost nothing beyond the cheap topic lookup.

The try/catch around each class in the assembly loop matters more than it looks. If resolving topics for one class throws, that's not a reason to fail the other four hundred classes in the batch. Assembly is allowed to be incomplete for a given class this run, since the next scheduled run will retry it. Swallowing that failure and moving on is the right call specifically because nothing downstream depends on assembly succeeding on the first try, only on it eventually succeeding before someone opens that quiz.

What I should have done first

I originally framed this as a caching problem, generate ahead of time so it's warm when needed, which is exactly the framing that leads to generating everything for everyone. The actual shape of the problem was a cost problem wearing a caching costume: most of what you'd be caching is never read.

Splitting prepare from generate only became obvious once I stopped asking "how do we make this fast" and asked "which part of this is actually expensive." Only one of the two steps touches the model. Everything else was free to do eagerly the whole time.