ChatRollout says about how it ended, how behaviors retries and records
failures, and what that means for a fxtr step that returns the rollout.
How a rollout ends
A rollout’sstatus is one of:
RolloutCompleted: the context policy ended the rollout withEndRollout, which is recorded on the rollout’send.RolloutMaxTurns: the rollout reachedrun_chat_rollout’smax_turns.RolloutFailed: a model call failed in a category the rollout records (see Recorded failures).
max_turns counts
completed model turns, and the limit is checked before the policy is asked for its next turn.
So a script with exactly N prompts and max_turns=N ends as RolloutMaxTurns without the policy
being asked for its final EndRollout: allow one more turn, max_turns=N + 1, when how the policy
ends matters. With max_turns=None, the default, the runner imposes no limit.
A model turn’s stop reason, such as StopMaxTokens, is separate from the rollout’s status: a
completed rollout can still contain a reply cut off at the token limit.
Failures and retries
A model call’s expected failures are returned as data:ModelAPI.generate returns a
ModelCallFailure with a category (context_length, content_filter, rate_limited,
overloaded, authentication, invalid_request, or server_error), a description, and the
provider’s HTTP status. An error in the implementation itself raises. The provider adapters turn
off their SDKs’ own retries, so that these settings decide every attempt:
- A session of
chat_session_model_from_apihas no retry schedule by default (retry=None). Passretry=RetryConfig()to retry failures infailure_categories_to_retry, by defaultDEFAULT_FAILURE_CATEGORIES_TO_RETRY:rate_limited,overloaded, andserver_error. - For one request,
generate_with_retries(api, request, retry=..., failure_categories_to_retry=...)takes the same settings, both required, as in the judge step. Once attempts run out, it returns the last failure.
RetryConfig sets the schedule: max_attempts (3 by default, counting the first),
initial_backoff_seconds (1), backoff_base (3), max_backoff_seconds (120), and
jitter_seconds (up to 5 seconds added to every wait at random).
Recorded failures
When a session’s model call still fails after its retries,run_chat_rollout either records the
failure or raises it. By default, failure_categories_to_record=("context_length",): a rollout
whose context outgrows the model’s window is returned as RolloutFailed, with the conversation so
far, including the context turn the model never answered. A failure in any other category raises
ModelCallFailedError, unless you add its category to failure_categories_to_record.
A recorded failure is a returned value, so a fxtr step that returns the rollout succeeds, and fxtr
caches the failed rollout as the step’s result (see
Failures: return or raise?). Record
a category only for failures every attempt would reproduce, and decide deliberately how scoring
treats them, as missing or as failed, rather than scoring them as successes.
Watching a rollout as it runs
Pass an asyncevent_callback to run_chat_rollout to receive the rollout’s events as they
happen:
ContextTurnEventandModelTurnEvent, each carrying a completed turn, the same object that lands on the returned rollout;ModelStreamEvent, carrying a delta of the model’s output while it generates a turn, when the model’s session streams (sessions of a streaming adapter, such as the provider adapters, do);RolloutEndEvent, last, carrying how the rollout ended: thestatusandendthe returned rollout has. A rollout that raises reports none.
StreamRestart tells a consumer to reset what it has assembled of the turn so far. The completed
turns and the returned rollout are the durable content.
To watch a rollout in the viewer while its step runs, pass a ChatRolloutPreview publishing to
the step’s context.preview:
ChatRollout (see
Partial values): the finished turns, and
while the model streams, a partial last model turn holding its output so far. status and end
are None until the rollout ends, then what the returned rollout has. A project registering the
behaviors renderers (behaviorsRenderers from
fxtr-view-behaviors) sees it in the same conversation view as the finished rollout, marked in
progress, with the reply being streamed at the bottom, and the step’s row in its run’s table says
the turn the rollout is on (“turn 6 · streaming…”). To keep another callback too, call the preview
from your own: await preview(event).
The same renderers show a partial Transcript, TranscriptGroup, or AgentRun that a step
publishes itself, with context.preview(Partial(Transcript, ...)): the messages or transcripts so
far, each one still being written (itself a partial value) marked as streaming.