Skip to main content
A context policy is the model’s environment in a rollout. It supplies the session’s initialization, such as a system message, then each turn the model sees next (a ContextTurn), until it ends the rollout with EndRollout. run_chat_rollout(model, policy, ...) alternates the policy’s turns with the model’s. behaviors builds policies from smaller parts: a user simulator writes the user’s messages, and a tool policy answers the model’s tool calls. These come from behaviors.implementations.context_policies.

Scripted users

ScriptedUserSimulator(messages=(...)) sends one scripted message each time it is consulted, then returns EndRollout(). context_policy_from_user_simulator(user, initialization=...) makes it a policy, as in the conversation step. That policy never offers the model tools, and raises if the model calls one anyway.

Simulated users

Two user simulators write the user’s messages with models instead of a script:
  • BaseModelUserSimulator samples each message from a base model (a pretrained model without chat tuning) that continues a transcript in the user’s voice.
  • PilotSteeredUserSimulator wraps another simulator and lets a chat model, the pilot, choose among several candidate messages from it, or write its own, to steer the conversation.
Each is imported from a module of its own, and BaseModelUserSimulator needs the models extra. Like a scripted user, each becomes a policy through context_policy_from_user_simulator, or serializable_context_policy_from_user_simulator for a policy you can checkpoint.

A base model as the user

BaseModelUserSimulator prompts the base model with a document: an optional preface explaining what the document is, a few reference users, each a short biography (a vignette) followed by one of their conversations, and then the simulated user’s vignette and the conversation so far, in ChatML. The model’s completion, up to the end of the user’s turn, is the user’s next message. When the completion contains </transcript>, the user has left: the simulator ends the rollout, and nothing around the marker is sent. The model is called through the completions endpoint (/v1/completions) of an OpenAI-compatible server, such as vLLM or sglang. Its settings are a BaseModelUserSimulatorConfig, a tagged dataclass you can keep in your step’s configuration entity:
  • profile is the simulated user. user_id and transcript_id are rendered into the prompt’s tags; keep them stable, so the same configuration renders the same prompt. Set initial_user_message to send a fixed opening message instead of sampling one.
  • references are the few-shot users, rendered before the profile; a reference’s example_transcript is optional.
  • use_preface (on by default) opens the prompt with the preface. stop_sequences default to the ChatML turn boundaries. max_tokens, temperature, and top_p left None take the server’s defaults.
BaseModelUserSimulator(config, base_url=..., api_key=...) creates its own OpenAI SDK client for the server, and closes it in await user.aclose(). Pass an AsyncOpenAI client of your own with client=... instead (not both) to share one, for example an OpenAICompatibleProvider’s client (see Model providers); the simulator then leaves it for you to close. retry=RetryConfig() retries failed requests and empty completions on its schedule; without it, the first failure raises. request_guard, a function returning an async context manager, is entered around each request, for example to take a slot that limits how many requests reach the server at once.

Steering a simulated user

PilotSteeredUserSimulator(base, pilot_model, config) steers a base simulator with a pilot, any ModelAPI. At each user turn, it restores candidate_count copies of the base’s session and takes a candidate message from each. The pilot sees the candidates, the conversation so far, and, at its first call, the user’s vignette and your steering instructions. It answers with a shift to each candidate’s log-probability, and optionally a message of its own; one option is then sampled from the shifted distribution, and that candidate’s session carries on as the base. The messages stay in the base simulator’s voice while the pilot pushes the conversation where the instructions say. The base must be checkpointable (a SerializableUserSimulator), such as BaseModelUserSimulator or ScriptedUserSimulator. Its settings are a PilotSteeredUserSimulatorConfig:
  • user_bio is the vignette the pilot sees; pilot_instructions, the steering guidance only the pilot sees.
  • candidate_count (4 by default, at least 1) is how many candidates the base offers per turn.
  • allow_custom_text (on by default) lets the pilot write the message itself. Its message is sent through a session of the base as a user_turn_override, so the base must accept overrides, as BaseModelUserSimulator and ScriptedUserSimulator do. Turn it off to keep every message the base’s own.
  • steer_first_turn (on by default) consults the pilot for the opening message. Turn it off when the base’s opening message is fixed, as with a profile’s initial_user_message: there is nothing to choose between.
  • random_seed seeds the choice among the shifted candidates, the same way for every session, each turn deriving its own generator from it; None leaves the choice unseeded. To vary the choices across replicas, give each replica its own seed.
  • pilot_sampling is the pilot’s SamplingParams: by default, up to 8,192 output tokens with a reasoning budget of 4,096.
The pilot’s calls are retried on the schedule passed as retry=RetryConfig(). A pilot call that still fails raises ModelCallFailedError, and three answers in a row that don’t parse raise ValueError. Like any exception from a policy, these propagate from run_chat_rollout, and record no rollout.

A step with a simulated user

This step has a steered, base-model user talk to the subject model. The simulators’ settings are in the configuration entity; where the base model is served is not, since it doesn’t change what is sampled:
run_chat_rollout closes the sessions it starts, and the steered simulator closes the base sessions it starts, candidates it didn’t choose included. Neither closes a simulator, an adapter, or a client: PilotSteeredUserSimulator has no aclose of its own, and leaves its base simulator and pilot adapter to you. Close each one you construct in finally, as above.

What simulated turns record

Each turn either simulator produces records the text behind it in its metadata, which the policy combinators copy onto the context turn (see Metadata). A SimulatedUserText holds the text as produced, raw_text, which may still have surrounding whitespace or the </transcript> marker, and its source: "sampled" from the model, "fixed" by the configuration (an initial_user_message), or "override", a message chosen outside the simulator, such as the pilot’s own. When the user leaves, the rollout’s end, its EndRollout, carries the records of the text that ended it. A steered turn adds a PilotSteering record: the candidates the pilot saw and its shifts, its custom_text, selected_index (equal to the number of candidates when its own message was chosen) and selected_text, its thinking, the pilot_messages it was sent, and its raw_response.
Since every steered turn keeps the pilot’s whole request, steered rollouts are much larger than the conversations they hold.

Checkpointing a simulated user

Both simulators are serializable: a session’s get_state() can be saved and restored, as Checkpointing sessions describes. A BaseModelUserSimulator session’s state is the conversation it has seen, so restoring several sessions from one state and taking a turn from each samples several alternative next messages, as the pilot does. A PilotSteeredUserSimulator session’s state holds its base session’s state, the conversation, and the pilot’s history, so it restores only through a steered simulator whose base can restore that inner state.
reply_messages is the subject model’s reply to the first message, a tuple of messages. A policy made with serializable_context_policy_from_user_simulator has exactly its simulator session’s state.

Tools

Tool definitions describe tools to the model; behaviors never runs a function because a tool call names it. The tool policy executes calls and validates their inputs. A tool policy implements these async methods, whose structural protocols are ToolPolicy and ToolPolicySession in behaviors.interfaces.context_policies:
  • ToolPolicy.start_session() creates an independent tool session for one rollout.
  • available_tools() returns the tool definitions: ToolParams, each with a name, a description, a JSON schema of its input, strict, and input_examples (pass None explicitly for an optional field you don’t use).
  • generate_tool_turn(tool_uses) returns a ToolTurn with one ToolMessage for each ContentToolUse, in the same order and with matching tool_call_ids.
  • aclose() releases the session’s resources.
NoTools is a tool policy that offers none. Two combinators make a tool policy part of a context policy:
  • context_policy_from_user_and_tools(user, tools, initialization=...) sends model turns that contain tool calls to the tool policy, and every other turn to the user simulator, which sees the model’s messages since it was last consulted. By default (show_tool_calls_to_user=False), the simulated user doesn’t see tool calls or their results.
  • context_policy_from_tool_policy(tools, first_turn_messages=..., initialization=...) starts with fixed messages, answers each model turn’s tool calls, and ends the rollout when the model takes a turn without tool calls. Cap a runaway loop with run_chat_rollout’s max_turns.
A tool policy can’t end a rollout; only a user simulator or a full context policy can. An environment that should end when, say, a submit tool completes a task needs a context policy of its own (below).

Custom conversation flows

For a flow the combinators don’t express (a user who answers depending on what the model said, tools offered only at some turns), write the policy as an async generator: each yield gives the next context turn, and the model’s reply comes back as the value of the yield:
Pass the generator function (turns), not a running generator: each rollout starts its own. Yield EndRollout explicitly to finish, since a generator that simply returns raises an error. Its metadata, like a turn’s, holds tagged records (see Metadata). For a reusable policy with settings of its own, subclass GeneratorContextPolicy, overriding generate_turns, and generate_initialization if its sessions need one. A generator-based policy can’t be checkpointed, because a suspended generator can’t be saved. A policy that must be checkpointed implements the ContextPolicy and ContextPolicySession protocols directly (see Checkpointing sessions). The combinators above are one reasonable way to sequence users and tools; a flow with different rules can be written the same way, with their source as a reference.