Skip to main content
This guide walks you through a first experiment in fxtr: a small evaluation of how language models behave. You’ll design the experiment, write its code, run a small pilot, look at the results in the viewer, and then grow it. Along the way you’ll meet the main ideas in fxtr: entities, steps, workflows, arrays, and the step cache. The example experiment in this guide evaluates language models writing jokes, but you can study anything you’d like!
We strongly recommend walking through this experience interactively with an AI agent. To try it out, invoke the first-experiment skill:
From a terminal, if you haven’t already:
Then, in Claude Code:

1. Check your setup

Before you start, complete Installation, and check that you have:
  • The tools: uv, Docker (or another Postgres server), Node, pnpm, and git, at the versions Installation lists. Check with uv --version, node --version, and pnpm --version.
  • A running Postgres server: docker ps lists the fxtr-postgres container from Start Postgres. If it’s stopped, run docker start fxtr-postgres.
  • An API key for each model provider you’ll call: ANTHROPIC_API_KEY, OPENAI_API_KEY, or both. The running example uses Anthropic models, but its code pairs every model with its provider, so switching to OpenAI or mixing the two is a change to a few lines (see Write the launcher). For other providers, see OpenRouter.

2. Design your experiment

For your first experiment in fxtr, you can start with a question about how models behave, such as:
  • Do models keep a correct answer when the user pushes back?
  • How does the wording of a system prompt change how a model responds?
  • What do models make when asked to draw or describe themselves?
A good first question needs no external data, compares a few conditions (often a few models), and can be measured from the conversations themselves. Once you have one, decide:
  1. What you’re comparing. Usually this is models, but it could be system prompts or other settings. Pick at least two conditions you expect to differ.
  2. What you’re sweeping over. One prompt tells you little about a model in general, so measure the behavior across several variations, such as topics, phrasings, or tasks, and take several samples of each.
  3. How you generate a transcript. Often one turn of conversation with a fixed prompt is enough. More involved experiments use several turns or tools.
  4. What you measure. A pure function of the transcript, such as its length or whether an answer is correct, or more often a judge model that follows a rubric. Have the judge explain its reasoning before it gives a score or a label.
  5. How you summarize. For example, a mean score or the share of each label per condition.
You should generally start small: a few conditions, a handful of items, and one measurement. Once you have something simple working, you can scale it out.

The running example

The rest of this guide follows one question: which language models write the funniest jokes? Its design:

From design to code

Each part of a design becomes a piece of fxtr code:
  • Entities: Data records that represent objects in your program, such as configuration objects, measurement results, or a report that collects the final results. Each entity type has a unique type ID, which fxtr uses to visualize it.
  • Steps: The functions that do the work. For our joke evaluation: tell_joke asks one model for one joke, grade_joke grades one joke, and summarize averages one model’s grades.
  • Arrays hold many values indexed by named dimensions, including the variables you are comparing and those you are sweeping over. Each step is mapped over some of them to run computations in parallel. For instance, in the joke evaluation, model, topic, and sample are dimensions, and fxtr calls tell_joke once per model, topic, and sample.
  • A workflow connects the steps, and a launcher script supplies its inputs and runs it as a job.
You can read more about these pieces in Core concepts.

3. Set up the project

Create a project and install its dependencies. The commands below use the name joke-eval (feel free to substitute your own).
fxtr new writes an empty project and makes it a git repository. behaviors is our library for calling language models in fxtr, which your steps can use. Next, connect the project to your database:
Then add your keys to the project’s .env, which is gitignored:
.env
The project now holds:
See Project configuration for more details on the project structure. Next, make an initial commit:

4. Write the experiment

Experiments in fxtr generally consist of at least two files:
  • the experiment module, a library that defines the main experiment workflow and other core components,
  • the launcher, which specifies what the experiment runs on and is responsible for launching it in fxtr.
If you want to get started immediately, you can skip ahead to the complete experiment module and launcher for the joke eval. Below we’ll walk you through the pieces and what they mean.

Define the entities

An entity is a frozen dataclass that fxtr can store and show in the viewer. Each gets a type name of its own. You’ll usually want:
  • a configuration for each kind of model call, holding every setting that affects its output, so the results always point at the exact settings that produced them;
  • a measurement entity, such as a grade or a verdict, with the judge’s rationale and its full reply;
  • a report that references everything the experiment produced, so the launcher, and later a results view, can find it all from one place.
You can also use existing entities like ChatRollout from behaviors. Here’s an example configuration entity for how to generate a joke from a topic:
You can expand below to see the other types we’ll need:
See Entities and custom types for info on how to define entities.

Write the steps

A step is an async function decorated with @step. fxtr stores each call’s result in its cache, and reuses it when the same call comes up again. Write one step per call you’d make by hand: one conversation, one judgment, one summary. fxtr handles running them many times. For our joke evaluation, we can use three steps:
  • tell_joke(config, topic) formats the prompt and holds a one-turn conversation with run_chat_rollout. It returns the ChatRollout, a behaviors entity holding the whole conversation.
  • grade_joke(judge, topic, rollout) takes the joke’s text from the rollout, sends the rubric to the judge, and reads the <score> and <rationale> tags from its reply. A missing joke or an unreadable score is recorded as a grade with no score. A failed model call raises, so that fxtr retries it when you resume the job.
  • summarize(grades) receives one model’s grades, as an array over topic and sample, and averages their scores.
Here’s the step that generates a joke:
The behaviors guide Calling language models explains some common conversation and judge patterns.

Connect them in a workflow

The workflow function defines the overall structure of your experiment. Instead of running steps directly, it schedules them using run_step, and defines both the inputs that each step should receive and also the dimensions that the steps should be mapped over. Our joke evaluation can be defined using the following workflow:
  • tell_jokes is mapped over model and topic, and replicas adds a sample dimension, so its result is an array of rollouts indexed by [model, topic, sample].
  • grade_jokes is mapped over all three. The judge, which has no dimensions, goes to every call, and each call gets the topic matching its joke.
  • summarize is mapped over model only, so each call receives one model’s grades over the remaining dimensions, [topic, sample].
Arrays and parallel computations explains mapping in depth, and Workflows and steps explains how workflows run.

Put it all together into an experiment module

For this experiment, we can put the entity types, steps, and workflow function into a single experiment module. Here’s what it looks like for the joke evaluation:
src/joke_eval/experiment.py
Register your module in pyproject.toml, so that fxtr can find its steps and workflow when it resumes a job or shows one in the viewer:
pyproject.toml

Write the launcher

The launcher is a script which is responsible for configuring and launching the experiment. This is usually a good place to specify global configuration arguments or inputs that you want to run your experiment over. The simplest possible launcher is just a main script which connects to fxtr, stores the configurations, builds the workflow’s inputs, and runs it. This one asks one model for one joke about one topic:
launch.py
However, you might want to expand this to include helper functions and command line flags for configuring the shape of your experiment. For instance, here’s a more fully-featured launcher for the joke evaluation:
launch.py

Show conversations in the viewer

The viewer can show your conversations using the renderers from behaviors. Add fxtr-view-behaviors to the dependencies in views/package.json:
views/package.json
Register its renderers in views/src/bundle.ts:
views/src/bundle.ts
Then install it, rebuild, and commit everything:
Before you spend money on model calls, you can ask a coding agent to check the experiment: have it run the pilot on a scratch database schema, with the function that builds model clients (make_api in the example) replaced by a fake model. This catches mistakes in the steps, the workflow, and the launcher.

5. Run the pilot

In a second terminal, start the viewer from the project, and leave it running:
Next you can launch the evaluation:
The launcher prints the job’s ID and a link, and the viewer switches to the job. It shows the workflow with a card for each step, which fills in as the step’s calls finish. Try clicking through the results:
  • Open the card of the step that generates conversations (tell_jokes) to see its result array. Each value links to a conversation.
  • Open a measurement from the judge step (grade_jokes) to read the judge’s rationale.
  • Open the result of the summary step (summarize) to see the result per condition.
You can use the viewer to read a few conversations and their measurements, and figure out if there is anything that you want to change in the experiment before you scale it up.

6. Change it and run it again

Whenever you launch the experiment again, fxtr reuses every cached result whose step gets the same inputs, so only new work costs model calls. The two most common changes to a first experiment are scaling it up and adding a condition.

Scale up to the full run

Once the pilot looks right, you can launch the full run. For the joke evaluation, that’s all 10 topics with 3 samples each:
It reuses all of the pilot’s results and computes only the new ones: in the example, the other 54 jokes and 54 grades. Then it stops:
A step that reduces over a dimension that grew, here summarize, now gets more inputs than in the pilot, at the same cache address (jokes-v1/summarize[model:"haiku-4.5"]). fxtr won’t silently replace a cached result with one computed from different inputs, so it stops and reports a cache conflict. You don’t need the pilot’s summaries any more, so you can resume the job and overwrite them, using the job ID it printed:
This reruns only the summaries. Resuming doesn’t run the launcher, so launch once more to print the summary. Every step is now cached, so it finishes at once, with no model calls:

Add a condition

Next, you can add a condition as a new key of the dimension you compare. In the example, add Opus 5.5 to MODELS in launch.py:
Commit and launch the full run again. fxtr computes the new key’s conversations, measurements, and summary at new addresses, and reuses everything else. This time nothing conflicts, because no step reduces over model:

What a change reuses

Managing the step cache explains how each of these works, and the ways to resolve a conflict.

Build a results view

The viewer’s default views show every array and entity, but you can customize the view to show a more informative view of the data. With the fxtr skills installed, try asking your coding agent for one, describing what you want to see. For the example:
Visualization and custom renderers explains how views work.

Next steps

You’ve built, run, and changed your first experiment. From here, you could:
  • Refine the rubric, or add a second judge to see whether the judges agree (in the example, whether Opus favors its own jokes).
  • Study a follow-up question about something you noticed in the conversations.
  • Start an experiment on another question, by hand or with the first-experiment skill.
To learn more about the ideas in this guide:

Core concepts

How projects, steps, workflows, jobs, and arrays fit together.

Arrays and parallel computations

Dimensions, mapping steps over arrays, and replicas.

Managing the step cache

What a launch reuses, when it stops on a conflict, and how to resolve one.

Calling language models

Conversations, judges, and summaries in fxtr steps.