1. Check your setup
Before you start, complete Installation, and check that you have:- The tools: uv, Docker (or another Postgres server), Node, pnpm, and git, at the versions
Installation lists. Check with
uv --version,node --version, andpnpm --version. - A running Postgres server:
docker pslists thefxtr-postgrescontainer from Start Postgres. If it’s stopped, rundocker start fxtr-postgres. - An API key for each model provider you’ll call:
ANTHROPIC_API_KEY,OPENAI_API_KEY, or both. The running example uses Anthropic models, but its code pairs every model with its provider, so switching to OpenAI or mixing the two is a change to a few lines (see Write the launcher). For other providers, see OpenRouter.
2. Design your experiment
For your first experiment in fxtr, you can start with a question about how models behave, such as:- Do models keep a correct answer when the user pushes back?
- How does the wording of a system prompt change how a model responds?
- What do models make when asked to draw or describe themselves?
- What you’re comparing. Usually this is models, but it could be system prompts or other settings. Pick at least two conditions you expect to differ.
- What you’re sweeping over. One prompt tells you little about a model in general, so measure the behavior across several variations, such as topics, phrasings, or tasks, and take several samples of each.
- How you generate a transcript. Often one turn of conversation with a fixed prompt is enough. More involved experiments use several turns or tools.
- What you measure. A pure function of the transcript, such as its length or whether an answer is correct, or more often a judge model that follows a rubric. Have the judge explain its reasoning before it gives a score or a label.
- How you summarize. For example, a mean score or the share of each label per condition.
The running example
The rest of this guide follows one question: which language models write the funniest jokes? Its design:From design to code
Each part of a design becomes a piece of fxtr code:- Entities: Data records that represent objects in your program, such as configuration objects, measurement results, or a report that collects the final results. Each entity type has a unique type ID, which fxtr uses to visualize it.
- Steps: The functions that do the work. For our joke evaluation:
tell_jokeasks one model for one joke,grade_jokegrades one joke, andsummarizeaverages one model’s grades. - Arrays hold many values indexed by named dimensions, including the variables you are comparing and those you are sweeping over. Each step is mapped over some of them to run computations in parallel. For instance, in the joke evaluation,
model,topic, andsampleare dimensions, and fxtr callstell_jokeonce per model, topic, and sample. - A workflow connects the steps, and a launcher script supplies its inputs and runs it as a job.
3. Set up the project
Create a project and install its dependencies. The commands below use the namejoke-eval (feel free to substitute your own).
fxtr new writes an empty project and makes it a git repository.
behaviors is our library for calling language models in fxtr, which your steps can use.
Next, connect the project to your database:
.env, which is gitignored:
.env
4. Write the experiment
Experiments in fxtr generally consist of at least two files:- the experiment module, a library that defines the main experiment workflow and other core components,
- the launcher, which specifies what the experiment runs on and is responsible for launching it in fxtr.
Define the entities
An entity is a frozen dataclass that fxtr can store and show in the viewer. Each gets a type name of its own. You’ll usually want:- a configuration for each kind of model call, holding every setting that affects its output, so the results always point at the exact settings that produced them;
- a measurement entity, such as a grade or a verdict, with the judge’s rationale and its full reply;
- a report that references everything the experiment produced, so the launcher, and later a results view, can find it all from one place.
ChatRollout from behaviors.
Here’s an example configuration entity for how to generate a joke from a topic:
The other entity types
The other entity types
Write the steps
A step is an async function decorated with@step. fxtr stores each call’s result in its
cache, and reuses it when the same call comes up again. Write one step per call you’d make by
hand: one conversation, one judgment, one summary. fxtr handles running them many times.
For our joke evaluation, we can use three steps:
tell_joke(config, topic)formats the prompt and holds a one-turn conversation withrun_chat_rollout. It returns theChatRollout, a behaviors entity holding the whole conversation.grade_joke(judge, topic, rollout)takes the joke’s text from the rollout, sends the rubric to the judge, and reads the<score>and<rationale>tags from its reply. A missing joke or an unreadable score is recorded as a grade with no score. A failed model call raises, so that fxtr retries it when you resume the job.summarize(grades)receives one model’s grades, as an array overtopicandsample, and averages their scores.
The tell_joke step
The tell_joke step
Connect them in a workflow
The workflow function defines the overall structure of your experiment. Instead of running steps directly, it schedules them usingrun_step, and defines both the inputs that each step should
receive and also the dimensions that the steps should be mapped over.
Our joke evaluation can be defined using the following workflow:
tell_jokesis mapped overmodelandtopic, andreplicasadds asampledimension, so its result is an array of rollouts indexed by[model, topic, sample].grade_jokesis mapped over all three. The judge, which has no dimensions, goes to every call, and each call gets the topic matching its joke.summarizeis mapped overmodelonly, so each call receives one model’s grades over the remaining dimensions,[topic, sample].
Put it all together into an experiment module
For this experiment, we can put the entity types, steps, and workflow function into a single experiment module. Here’s what it looks like for the joke evaluation:src/joke_eval/experiment.py
src/joke_eval/experiment.py
src/joke_eval/experiment.py
pyproject.toml, so that fxtr can find its steps and workflow when it
resumes a job or shows one in the viewer:
pyproject.toml
Write the launcher
The launcher is a script which is responsible for configuring and launching the experiment. This is usually a good place to specify global configuration arguments or inputs that you want to run your experiment over. The simplest possible launcher is just amain script which connects to fxtr, stores the
configurations, builds the workflow’s inputs, and runs it. This one asks one model for one joke
about one topic:
launch.py
launch.py
launch.py
launch.py
Show conversations in the viewer
The viewer can show your conversations using the renderers from behaviors. Addfxtr-view-behaviors to the dependencies in views/package.json:
views/package.json
views/src/bundle.ts:
views/src/bundle.ts
5. Run the pilot
In a second terminal, start the viewer from the project, and leave it running:- Open the card of the step that generates conversations (
tell_jokes) to see its result array. Each value links to a conversation. - Open a measurement from the judge step (
grade_jokes) to read the judge’s rationale. - Open the result of the summary step (
summarize) to see the result per condition.
6. Change it and run it again
Whenever you launch the experiment again, fxtr reuses every cached result whose step gets the same inputs, so only new work costs model calls. The two most common changes to a first experiment are scaling it up and adding a condition.Scale up to the full run
Once the pilot looks right, you can launch the full run. For the joke evaluation, that’s all 10 topics with 3 samples each:summarize, now gets more inputs than in
the pilot, at the same cache address (jokes-v1/summarize[model:"haiku-4.5"]). fxtr won’t
silently replace a cached result with one computed from different inputs, so it stops and reports
a cache conflict. You don’t need the pilot’s summaries any more, so you can resume the job and
overwrite them, using the job ID it printed:
Add a condition
Next, you can add a condition as a new key of the dimension you compare. In the example, add Opus 5.5 toMODELS in launch.py:
model:
What a change reuses
Managing the step cache explains how each of these
works, and the ways to resolve a conflict.
Build a results view
The viewer’s default views show every array and entity, but you can customize the view to show a more informative view of the data. With the fxtr skills installed, try asking your coding agent for one, describing what you want to see. For the example:Next steps
You’ve built, run, and changed your first experiment. From here, you could:- Refine the rubric, or add a second judge to see whether the judges agree (in the example, whether Opus favors its own jokes).
- Study a follow-up question about something you noticed in the conversations.
- Start an experiment on another question, by hand or with the
first-experimentskill.
Core concepts
How projects, steps, workflows, jobs, and arrays fit together.
Arrays and parallel computations
Dimensions, mapping steps over arrays, and replicas.
Managing the step cache
What a launch reuses, when it stops on a conflict, and how to resolve one.
Calling language models
Conversations, judges, and summaries in fxtr steps.