Add messages to a response while it runs (steering between tool rounds) #360

A long response with tools can run for a minute or more: search, read some pages, compare, write. While it runs, the person often wants to adjust it ("skip Delta", "under $300", "actually make it Lisbon"). Today an app's only option is to cancel the response and start a new one, which throws away the lookups already done.

What we wanted was to send messages into the running response: after a round of tool calls, before the model's next request, add what the person has sent since, and the model reads it alongside the tool results.

Why it's easy to get wrong. The added messages have to go into the request and the transcript, in the same form and in the same place (after the round's tool outputs). If the in-flight request and a later replay of the transcript differ, a prompt cache that matches on exact prefixes (Anthropic's, for example) misses on every later turn. That's the same problem #325 fixed for tool calls. An app can't do this from outside: it doesn't control what goes into the next request mid-loop.

What we did. Our fork adds one method to ToolExecutionDelegate (gtokman/AnyLanguageModel@8f2d9f8):

/// Asked after each round of tool calls, before the model's next request.
/// The messages are added to the transcript as prompts, in order,
/// after the round's tool outputs. The default adds nothing.
func pendingMessages(in session: LanguageModelSession) async -> [String]

The OpenAI provider asks for messages after each round, in both the streaming and nonstreaming loops. It appends each one as a .prompt entry and as the matching user message in the request, so replay sends them exactly as they were sent. Our server keeps a per-conversation inbox. A small model also answers the person right away ("Got it, I'll leave Delta out."), and the main model is told it already said that, so it doesn't acknowledge twice.

The shape is a question for you. A delegate method fit our server, but there are other designs:

• a method on the session to queue a message for the response in progress (session.enqueue("skip Delta")), which is closer to how an app thinks about it;
• tying it to dynamic instructions or profiles (Integrate dynamic profiles with OpenAI Tool loops #344–Integrate dynamic profiles with Ollama Tool loops #348), since those already re-resolve the request each round;
• only Strings, or full Prompts (images, for example)?

And a semantics question: if the response ends with no further tool round, should messages still pending start a new turn, be returned to the caller, or be dropped? Our server interrupts the response if the model is writing its final text, and starts a new response from the messages (the lookups stay in the transcript; the unfinished text doesn't). Any that arrive as the reply closes go back to the app, which sends them as a normal message.

Does this belong in the library, and if so, which shape would you prefer? Happy to do the PR.