Installation
Install fxtr and set up your project and database.
Your first experiment
Design and run an experiment of your own, with your coding agent as a guide.
Core concepts
Projects, steps, workflows, jobs, arrays, and entities.
How it Works
- Define your experiment as simple, readable, environment-independent code. Each experiment or analysis is expressed as a Python package containing one or more workflows, which schedule expensive or nondeterministic steps into a graph of dependent computations. This experiment definition serves as an ideal “spec” for what you want to compute, without worrying about data storage, concurrency, or looking up previously-computed results. The dataflow and data types belong to you and are highly customizable.
- Let the system drive it to completion. fxtr traces and durably executes your experiment code, stores the data, and gives you live observability into what is running and what is scheduled. If your jobs crash or get interrupted, you can seamlessly resume them, without recomputing expensive results.
- Iterate freely. Want to increase the range of your sweeps, or add a new analysis step to your experiment? Simply edit the workflow code in place and launch a new job. fxtr automatically re-uses previously computed results across jobs, as long as they have the same name (or cache address) and receive the same inputs.
- Customize data visualizations and reports. Your coding agent can register custom renderers for your experiment workflows and data types, and have them directly integrated into the viewer.
- (Under construction) Share your results. You can share your code as a git repo to enable others to reproduce your experiment, and export and import jobs to share the data from your analyses. Soon, we plan to make it possible to publish experiments and share them via a hosted platform.
Components of a fxtr Project
fxtr experiments are organized into projects, which have two components.
- The project repo is a Git repo that defines how to execute and visualize your experiment or analysis. It contains a few pieces:
- Workflow functions: Python functions that define the overall structure of your experiment, configure sweeps, and schedule individual nondeterministic steps into a directed acyclic graph. Workflow functions are expected to be deterministic, and will be re-run each time a job is started or resumed.
- Step functions: Python functions that execute nondeterministic or expensive work, such as querying a language model. fxtr executes each of these once and then caches the results. Later runs re-use existing step results. To ensure reproducibility, steps should be environment-independent: you should get equivalent results regardless of which computer you run the steps on. (So, querying a language model API is OK, but reading or writing files on a local filesystem is generally not.)
- Entity types: Data types used by your experiment. Entities can point to other entities, and are used to save data like transcripts, judgements, and experiment configurations.
- Renderers: React components that define the visual presentation of a workflow, step, or entity. These renderers are registered with the system and can be used to visualize and inspect the results generated by the experiment.
- The experiment database is a database that holds the data and results related to a project. It includes:
- Job outputs and experiment state: Each execution of a workflow in a project is persisted to the database, including all steps and workflows that it ran. Job records can be exported and imported to share your results.
- The step result cache: The result of each successful step execution is cached, and stored under a cache address. Future jobs will re-use results from the cache, avoiding duplicate work. The cache can also be cleared manually to avoid conflicts or regenerate results from scratch.
- Project arrays: Arrays of data that you directly store into the database from an external source. These arrays can then be passed as inputs to workflow functions when starting a job.