Two short transactions around an LLM call

Talkemi is a language-learning platform I built. Students take 25-minute live classes with real teachers, and between classes they practise with Emi, an AI mentor. In roleplay mode Emi plays a character in an everyday situation, and every message the student sends turns into a call to an LLM.

I wanted to know exactly what happens to one of those messages, so I followed it through the code. When the student presses Send, the server locks the chat session, checks it, saves the message and commits, and only then, with the lock released, does it call the model.

That order was deliberate, and it was right, but the failure cases were messier than I expected. If the model call failed, the student saw an error and their message disappeared from the screen, even though it was still sitting in the database, so typing it again sent it to the model twice. Moderation had a similar gap: a flagged message got the student a warning, but it had already been saved, and on their next turn it went to the model anyway as part of the chat history.

Keeping a slow LLM call out of a database transaction is good, common advice. It isn't free, though, and these were the cases where the bill showed up.

Why the lock is short

Before saving a message, the server checks that the session is still open, that it's under its message limit and that the text is valid. That's a check-then-act sequence, and without a lock it races: two requests arriving together could both count 49 messages under the default limit of 50 and both get in, or a message could slip into a session someone ended a moment earlier. The check and the save have to happen as one step.

In Django that step is select_for_update(), which locks the session's row in Postgres until the transaction commits, so a second request for the same session waits its turn. The model call deliberately comes after the commit. If it ran inside the transaction, the row would stay locked for as long as the provider took to answer, up to the 300-second timeout on our OpenAI client, and every other request on that session would queue behind it, "end session" included, while a Postgres transaction sat open, holding that lock, the whole time.

What releasing the lock early left to me

Holding the lock through the model call would have serialized every request to the same session, and that alone kept them in order. Releasing it early meant handling ordering, failures and retries explicitly, and there were three gaps to close.

The first was ordering. Two sends in a row to the same session would each pass the lock in turn and then call the model at the same time, so replies were saved in whatever order the provider finished, with nothing recording which reply answered which message. The other two were the failures I started with, and they had one cause: the server saved the message before either the model or the safety check had a say.

Two short transactions, and a state in between

The fix is two short transactions around the slow call, with the message's state written down in between, so that if anything fails halfway the database says exactly where it stopped. Here's the flow, simplified: the retry, idempotency and stale-attempt checks are left out for clarity.

moderate(content)                         # network call, no lock held
with transaction.atomic():                # 1: the decision
    lock session; check open, limit, nothing pending
    message = create(status="pending", client_key=key)
reply = call_model(history_up_to(message))   # no lock held
with transaction.atomic():                # 2: the outcome
    create(role="ai", reply_to=message); message.status = "answered"
# if the model fails: message.status = "failed"

Each piece closes one of the gaps:

• Moderation runs first, before any lock, because it's a network call too. A flagged message is never saved, so it can't slip into a later prompt. If the moderation service itself is down, the send fails closed: nothing is stored, and the student is asked to try again rather than being told their message was flagged.
• While the model is answering, the message is pending, and another send to the same session is turned away as "still replying" until the first one finishes. That brings ordering back without holding a lock during the call.
• Each reply stores reply_to, the message it answers.
• If the call fails, the message is marked failed and stays on screen with a Retry button. Retry answers that same message instead of creating a new one, and messages without an answer are left out of the history the model sees.
• The app sends a unique key with every message. If a flaky network delivers the same request twice, the server recognises the key and, instead of storing a copy, returns wherever that message has got to: the reply if it's answered, "still replying" if it's pending, or the failed message with its Retry button. A different message sent while one is pending gets "still replying".
• If the server dies mid-call, during a deploy for instance, a message could be left pending and block the session, so after six minutes it's marked failed. That happens the next time the session is opened or used, so the student sees it with a Retry button like any other failed reply. Six minutes is comfortably longer than the client's 300-second timeout, so a normally slow call should finish well before then, but that's an expectation rather than a guarantee: SDK retries or a slow worker can stretch a call past it.
• An outcome is only recorded by the request that still owns the attempt: the message has to be pending with the timestamp it had when that request claimed it. Expiry and retry both change that timestamp, so when an old call does outlive the six minutes, its late reply is dropped instead of becoming a second answer, and a late error can't mark a newer attempt as failed.

Testing each gap

Each gap got a test that drives the real endpoint:

• a flagged message is never saved, and the model is never called;
• a provider error leaves the message failed, and Retry answers that same message;
• a second send while one is pending is turned away;
• a repeated key returns the stored message, with a single model call;
• the history sent to the model leaves out unanswered messages;
• a message stuck as pending comes back as failed when the session is reopened, and Retry answers it.

Those tests run one request at a time, though, and a lock can't be proved that way. So one more test sends two messages at the same moment, each on its own Postgres connection, while the model call is held open. One request gets "still replying", the model is called once, and only one message is stored.

A concurrency test that passes either way proves nothing, so I checked that this one was really testing the lock. I removed select_for_update() and ran it again, and it failed: both messages were stored and the model was called twice. With the lock back, it passes, along with the rest of the AI module's 126 tests.

What it still doesn't promise

The unique key stops a duplicate message, and a repeated request while a reply is pending gets "still replying" instead of a second model call. That doesn't make the model call itself exactly-once. If the server dies after the provider has answered but before the reply is saved, that answer is lost, and the retry pays for a second call. I accepted that trade-off: it needs a crash at exactly that moment, it costs one extra call, and the student sees a Retry button rather than a broken conversation. Closing it fully would mean keeping a durable record of every provider call to reconcile against, which is more machinery than one extra call is worth here.

What a student sees now

Most of this is invisible to the student, and that's how it should be. A message the server can't accept is turned down before it's stored, with a reason that's true. Once a message is accepted and stored, it always has a visible outcome: it's answered, it's still pending while the model replies, or it's failed with a Retry button that answers that same message. Pressing Send twice, a request the network delivers twice or a slow provider no longer produces a duplicate turn or a reply out of order.

Underneath, the transaction still lasts only as long as the database work, so a slow provider doesn't keep a row locked or a transaction open, yet every message now carries a state you can follow from Send to reply. Failures, repeated requests and simultaneous sends each end somewhere defined, and the conversation on the student's screen is the same one the model sees.