ContextTurn),
until it ends the rollout with EndRollout. run_chat_rollout(model, policy, ...) alternates the
policy’s turns with the model’s. behaviors builds policies from smaller parts: a user
simulator writes the user’s messages, and a tool policy answers the model’s tool calls.
These come from behaviors.implementations.context_policies.
Scripted users
ScriptedUserSimulator(messages=(...)) sends one scripted message each time it is consulted,
then returns EndRollout(). context_policy_from_user_simulator(user, initialization=...) makes
it a policy, as in the conversation step.
That policy never offers the model tools, and raises if the model calls one anyway.
Simulated users
Two user simulators write the user’s messages with models instead of a script:BaseModelUserSimulatorsamples each message from a base model (a pretrained model without chat tuning) that continues a transcript in the user’s voice.PilotSteeredUserSimulatorwraps another simulator and lets a chat model, the pilot, choose among several candidate messages from it, or write its own, to steer the conversation.
BaseModelUserSimulator needs the models
extra. Like a scripted user, each becomes a policy through context_policy_from_user_simulator,
or serializable_context_policy_from_user_simulator for a policy you can
checkpoint.
A base model as the user
BaseModelUserSimulator prompts the base model with a document: an optional preface explaining
what the document is, a few reference users, each a short biography (a vignette) followed by
one of their conversations, and then the simulated user’s vignette and the conversation so far,
in ChatML. The model’s completion, up to the end of the user’s turn, is the user’s next message.
When the completion contains </transcript>, the user has left: the simulator ends the rollout,
and nothing around the marker is sent. The model is called through the completions endpoint
(/v1/completions) of an OpenAI-compatible server, such as vLLM or sglang.
Its settings are a BaseModelUserSimulatorConfig, a tagged dataclass you can keep in your
step’s configuration entity:
profileis the simulated user.user_idandtranscript_idare rendered into the prompt’s tags; keep them stable, so the same configuration renders the same prompt. Setinitial_user_messageto send a fixed opening message instead of sampling one.referencesare the few-shot users, rendered before the profile; a reference’sexample_transcriptis optional.use_preface(on by default) opens the prompt with the preface.stop_sequencesdefault to the ChatML turn boundaries.max_tokens,temperature, andtop_pleftNonetake the server’s defaults.
BaseModelUserSimulator(config, base_url=..., api_key=...) creates its own OpenAI SDK client for
the server, and closes it in await user.aclose(). Pass an AsyncOpenAI client of your own with
client=... instead (not both) to share one, for example an OpenAICompatibleProvider’s
client (see Model providers); the simulator then
leaves it for you to close. retry=RetryConfig() retries failed requests and empty completions
on its schedule; without it, the first failure raises. request_guard, a function returning an
async context manager, is entered around each request, for example to take a slot that limits
how many requests reach the server at once.
Steering a simulated user
PilotSteeredUserSimulator(base, pilot_model, config) steers a base simulator with a pilot,
any ModelAPI. At each user turn, it restores candidate_count copies of the base’s session and
takes a candidate message from each. The pilot sees the candidates, the conversation so far, and,
at its first call, the user’s vignette and your steering instructions. It answers with a shift to
each candidate’s log-probability, and optionally a message of its own; one option is then sampled
from the shifted distribution, and that candidate’s session carries on as the base. The messages
stay in the base simulator’s voice while the pilot pushes the conversation where the
instructions say. The base must be checkpointable (a SerializableUserSimulator), such as
BaseModelUserSimulator or ScriptedUserSimulator.
Its settings are a PilotSteeredUserSimulatorConfig:
user_biois the vignette the pilot sees;pilot_instructions, the steering guidance only the pilot sees.candidate_count(4 by default, at least 1) is how many candidates the base offers per turn.allow_custom_text(on by default) lets the pilot write the message itself. Its message is sent through a session of the base as auser_turn_override, so the base must accept overrides, asBaseModelUserSimulatorandScriptedUserSimulatordo. Turn it off to keep every message the base’s own.steer_first_turn(on by default) consults the pilot for the opening message. Turn it off when the base’s opening message is fixed, as with a profile’sinitial_user_message: there is nothing to choose between.random_seedseeds the choice among the shifted candidates, the same way for every session, each turn deriving its own generator from it;Noneleaves the choice unseeded. To vary the choices across replicas, give each replica its own seed.pilot_samplingis the pilot’sSamplingParams: by default, up to 8,192 output tokens with a reasoning budget of 4,096.
retry=RetryConfig(). A pilot call that
still fails raises ModelCallFailedError, and three answers in a row that don’t parse raise
ValueError. Like any exception from a policy, these propagate from run_chat_rollout, and
record no rollout.
A step with a simulated user
This step has a steered, base-model user talk to the subject model. The simulators’ settings are in the configuration entity; where the base model is served is not, since it doesn’t change what is sampled:run_chat_rollout closes the sessions it starts, and the steered simulator closes the base
sessions it starts, candidates it didn’t choose included. Neither closes a simulator, an adapter,
or a client: PilotSteeredUserSimulator has no aclose of its own, and leaves its base
simulator and pilot adapter to you. Close each one you construct in finally, as above.
What simulated turns record
Each turn either simulator produces records the text behind it in its metadata, which the policy combinators copy onto the context turn (see Metadata). ASimulatedUserText holds the text
as produced, raw_text, which may still have surrounding whitespace or the </transcript>
marker, and its source: "sampled" from the model, "fixed" by the configuration (an
initial_user_message), or "override", a message chosen outside the simulator, such as the
pilot’s own. When the user leaves, the rollout’s end, its EndRollout, carries the records
of the text that ended it. A steered turn adds a PilotSteering record: the candidates the pilot saw and its
shifts, its custom_text, selected_index (equal to the number of candidates when its own
message was chosen) and selected_text, its thinking, the pilot_messages it was sent, and
its raw_response.
Checkpointing a simulated user
Both simulators are serializable: a session’sget_state() can be saved and restored, as
Checkpointing sessions
describes. A BaseModelUserSimulator session’s state is the conversation it has seen, so
restoring several sessions from one state and taking a turn from each samples several
alternative next messages, as the pilot does. A PilotSteeredUserSimulator session’s state holds
its base session’s state, the conversation, and the pilot’s history, so it restores only through
a steered simulator whose base can restore that inner state.
reply_messages is the subject model’s reply to the first message, a tuple of messages. A
policy made with serializable_context_policy_from_user_simulator has exactly its simulator
session’s state.
Tools
Tool definitions describe tools to the model; behaviors never runs a function because a tool call names it. The tool policy executes calls and validates their inputs. A tool policy implements these async methods, whose structural protocols areToolPolicy and ToolPolicySession in
behaviors.interfaces.context_policies:
ToolPolicy.start_session()creates an independent tool session for one rollout.available_tools()returns the tool definitions:ToolParams, each with a name, a description, a JSON schema of its input,strict, andinput_examples(passNoneexplicitly for an optional field you don’t use).generate_tool_turn(tool_uses)returns aToolTurnwith oneToolMessagefor eachContentToolUse, in the same order and with matchingtool_call_ids.aclose()releases the session’s resources.
NoTools is a tool policy that offers none.
Two combinators make a tool policy part of a context policy:
context_policy_from_user_and_tools(user, tools, initialization=...)sends model turns that contain tool calls to the tool policy, and every other turn to the user simulator, which sees the model’s messages since it was last consulted. By default (show_tool_calls_to_user=False), the simulated user doesn’t see tool calls or their results.context_policy_from_tool_policy(tools, first_turn_messages=..., initialization=...)starts with fixed messages, answers each model turn’s tool calls, and ends the rollout when the model takes a turn without tool calls. Cap a runaway loop withrun_chat_rollout’smax_turns.
Custom conversation flows
For a flow the combinators don’t express (a user who answers depending on what the model said, tools offered only at some turns), write the policy as an async generator: eachyield gives the
next context turn, and the model’s reply comes back as the value of the yield:
turns), not a running generator: each rollout starts its own.
Yield EndRollout explicitly to finish, since a generator that simply returns raises an error.
Its metadata, like a turn’s, holds tagged records (see
Metadata).
For a reusable policy with settings of its own, subclass GeneratorContextPolicy, overriding
generate_turns, and generate_initialization if its sessions need one.
A generator-based policy can’t be checkpointed, because a suspended generator can’t be saved. A
policy that must be checkpointed implements the ContextPolicy and ContextPolicySession
protocols directly (see
Checkpointing sessions). The
combinators above are one reasonable way to sequence users and tools; a flow with different rules
can be written the same way, with their source as a reference.