Two services, one topic, either deploy order

Two services that exchange messages are deployed separately and can be rolled back separately, so every message format change has to work in both orders and in both mixed states. Adding a field that old consumers ignore is safe in all four combinations. Changing what an existing field means, or requiring the consumer to understand the new one, is not, and the failure shows up as silently dropped work rather than an error.

The change

A transcription service had been publishing results to its own topic. It needed to publish a second kind of result, and the natural move was one topic carrying both, distinguished by a field.

The consumer lives in a different service, in a different repository, with a different deploy pipeline and a different on-call. The question is not whether the new format is better. It is what happens during the hour, or the week, when one side has shipped and the other has not.

Four states, not two

A plan that only works in one deploy order is a plan that forbids rollback, and nobody finds that out until the night they need one.

Additive, with a default on both sides

# Producer: same topic as before, one new field.
publish(SAVE_TRANSCRIPT_TOPIC, {
    "lecture_id": lecture_id,
    "transcript": text,
    "action": "store",          # new; old consumers never look at it
})

# Consumer: a message with no "action" is the historical kind.
action = msg.get("action", "store")

The default on the consumer side is what makes rollback of the producer safe. Without it, a new consumer reading messages from a rolled-back producer sees no action and has to decide what to do with a message it cannot classify, which in practice means dropping it.

And the old consumer ignoring the new field is not luck. It is a property of how it deserialises: it reads the fields it knows by name. A consumer that validates strictly and rejects unknown fields would fail here, which is worth checking before you assume this is additive.

Why not a new topic

A new topic is the textbook answer and it was wrong here for a specific reason: ordering. Results for the same lecture have to be processed in order, and two topics give you no ordering guarantee between them. Same topic, same partition key, ordering preserved.

It is also operationally cheaper. One topic to provision, monitor, set retention on and remember exists. That matters more than it sounds at the point where somebody is paged about it.

The rules I would apply next time

1. Add fields, never repurpose them. A field that changes meaning is undetectable to a consumer that reads it by name.
2. Default on the consumer side, so a rolled-back producer still works.
3. Write out all four producer/consumer states before writing the code. It takes two minutes and it is where the problem is visible.
4. Confirm the consumer tolerates unknown fields rather than assuming it.
5. Delete the compatibility default in a later release, once both sides have been stable long enough that a rollback would not reach the old format.

That last step is the one everyone skips, and skipping it is how a codebase accumulates defaults nobody can explain. It should be a ticket with a date, not a comment saying "remove later".

Deploy order is not the only coupling that shows up only between environments. Adding one request header is two lines of code and a gateway allow-list that differs per environment.