Skip to main content
The tests exercise detection, runtime decisions, callback behavior, stream handling, config parsing, and saved-trace checks. Most use fake agents and fake embeddings, so they don’t need network access or provider credentials. For the concepts behind these checks, read tests, traces, and evals. For the step-count helper and its limits, see agent trajectories and convergence.

​
Run the tests

From the repository root:
One test imports the demo scenario definitions and skips if their framework dependencies aren’t installed. To include it, install the framework extra first:
The runtime test module contains 49 tests. It checks that repeated calls alone don’t force a stop, healthy polling advances, retryable failures can recover, permanent failures stop, and action/handoff cycles are detected. Discovery also runs the documentation automation tests. See contributing and docs maintenance for those checks.

​
Replay the progress-aware fixtures

examples/v02_labeled_traces.json contains seven labeled runs:
  • An exact search loop with empty output.
  • Healthy polling with changing status.
  • A rate-limit failure followed by success.
  • Repeated authentication failure.
  • Alternating search/summarize actions without progress.
  • A closed planner/researcher/reviewer handoff cycle.
  • Healthy pagination with advancing pages.
The tests compare each run’s final decision with its expected_decision value. You can inspect the same runs directly:

​
Run the detector evaluation

The evaluation module needs python-dotenv, which is included in the demo extra:
With OPENAI_API_KEY unset in both the environment and .env, the command runs the offline exact-detector evaluation and prints illustrative convergence scores. If that key is set, it also runs provider-backed comparisons and a judge example, which need the relevant SDKs and can incur API charges. The exact-detector evaluation uses a separate six-case in-code dataset. It measures whether detector evidence fires, not whether the runtime policy reaches STOP. That matters because repetition can be a warning while useful work continues. These results come from LoopDetector(threshold=3) on the six-case fixture: two true positives, zero false positives, one false negative, and three true negatives. The missed case is a paraphrase loop, which exact signatures don’t catch. These are small fixture results, not production benchmarks or whole-policy accuracy. The printed convergence examples use fixed step-count lists. They illustrate a scoring function; the command does not measure real agent latency or token savings.

​
Evaluate your own workload

Collect healthy retries and polling as well as stuck runs. Start with observe-only reports, label the outcomes, then check changes against both groups. A detector that catches every loop but interrupts normal polling is not a useful improvement. Include the trace, settings, and expected decision when reporting a detection issue. See saved-trace analysis for the accepted input format. To turn observed failures into before/after comparisons, follow the evaluation workflow. For a task-focused walkthrough, add a trace regression check.