Consumer Agent Testing Tool (CATT) tests how an assistant uses your service. Run a scripted or model-driven agent against an MCP server, an MCP app, a website or an HTTP API. Check what the service did, inspect the evidence, and compare results across builds.
Muse and Instinct are supported surrogate emulation targets. Muse remains the default. Both use the shared connector testing contract, including routing, memory, payments, browser/HTTP tasks and scheduling. See the Instinct research for its sources and limits. The model that drives a trial is separate from the assistant being emulated.
Contributing | Security | Changelog | MIT license
A test host. CATT is a surrogate: it approximates an assistant's host, not its private implementation. A passing test does not certify compatibility with Muse or any hosted assistant. Each Muse-specific behavior has a recorded basis:
documented
,unofficial
,assumed
orunknown
. This project is not affiliated with or endorsed by Meta, Instinct, OpenAI or Anthropic.
Lane | Command | What it checks | Status and scope |
---|---|---|---|
MCP connectors | connector-run SCENARIO |
Use the server's tools, apply account and approval rules, protect private values, and check the service's recorded outcome and the agent's answer. | Muse and Instinct surrogates: --emulation-target muse|instinct . See connector lane. |
Connector extensions | connector-run SCENARIO with extensions: |
Routes, memory, custom API clients, browser actions, wallet cards, background jobs, untrusted content, channel limits and simulated users. | Supported with the lane. user_simulator is exploratory and cannot run in scripted tests or decision contracts. See extensions. |
Scenario scaffolding | scaffold URL --output FILE , scaffold --from-run RUN --output FILE |
Read a tool catalog or turn an agent run into scripted calls. Calls no service tools. Replace every TODO(scaffold) before running. |
Supported. One Streamable HTTP connector; no OpenAPI input. |
Muse connector readiness | check URL --suite muse-guidelines |
Review the catalog and optional manifest against the pinned connector guidelines without calling tools. | Documented constraints, not admission. See readiness checks. |
Shared surrogate readiness | check URL --suite surrogate --emulation-target instinct |
Check the catalog, allowlist, schema syntax and selected projection without calling tools. | CATT's contract for either target; see shared contract. |
Personal Agent Protocol | check ORIGIN --suite poppy |
Read poppy.json . With a manifest, probe sessions, sign-in and one declared operation on the company's hosts. |
Draft 0.1; no production Muse evidence. |
Muse gadget registration | check FILE --suite gadget |
Check a link.register message and command specs against the pinned SDK limits. Opens no connection. |
Documented limits, not admission. The SDK's terms say it is not a developer platform. |
MCP protocol contracts | check TARGET |
Check schemas, version negotiation, sessions and operation contracts; run the pinned conformance suite. | Supported. Generic MCP evidence; see readiness checks. |
Embedded MCP apps | surface-run SCENARIO |
Render the app in Chromium and check forms, tool calls, resources and service responses. An agent driver chooses from the scenario's scripted steps. | Supported. Generic MCP Apps host, not Muse. See host scope. |
Websites | browse POLICY , run --surface browser |
Act on accessibility snapshots under request rules. Takeover pauses the run (exit 3); the CLI has no handback. | Native Chromium, without OS isolation. Browser scenarios use only the packaged reservation portal. |
HTTP APIs | run --surface api-tools , inspect , check |
Bind approval to each proposed call and check backend outcomes. | Supported with a mapping. Your API also needs a loopback deployment and private control adapter. See API scenarios and API trials. |
Evidence | report , export , replay , diff |
Inspect reports, redact exports, verify playback and compare runs. | Supported. Playback and diff require sealed runs. |
Development extensions | --development-extensions COMMAND |
CLI binaries, isolated browsers, cloud workers, local Glimmer, generated OpenAPI and Prism mocks. | Outside the supported release. See development extensions. |
The connector lane records the basis for each Muse-specific behavior. Tool classes
and approval prompts follow documented guidelines. The muse-cli
projection and
template client follow unofficial mirrors; the native projection is an assumed
generic baseline. The neutral cli
projection supports both targets as a CATT
treatment. Instinct runs have a separate source record and label the
shared host algorithms as assumptions. Native/CLI MCP is CATT's test adapter;
no public Instinct MCP integration contract was found.
Run the local booking example under Instinct without a model or account:
uv run consumer-agent-testing-tool connector-run examples/connector/instinct-booking.yaml \
--driver scripted --script examples/connector/fixture-booking.calls.json \
--start "python -m consumer_agent_testing_tool.connector_fixture --port {port} --variant good --auth none"
Set emulation_target: instinct
in your scenario or pass
--emulation-target instinct
. Both targets use native
or cli
projection and the generic
MCP client. Instinct defaults to plain-text replies and does not run Muse guideline
checks. Its local wallet fixture covers payment behavior; live payment rails,
trusted-network protocol, Vault and Files remain outside the
shared contract.
Three quickstarts follow. The connector lane and the popular MCP server quickstarts run on loopback with no account; the Coverage Cat sandbox quickstart calls a hosted sandbox. Test your own service turns them into a workflow, and the documentation index lists every page.
You need Python 3.12 or newer (CI tests 3.12 on Linux and 3.14 on macOS),
uv, and Node.js 20 or newer with npm (the
installed packages declare >=20
; tested with 25.9). The recorded release
evidence covers macOS. From a clone of this repository, install the Python and
Node dependencies and the Chromium build the browser lanes use:
uv sync --extra openrouter --extra dev
npm --prefix tooling ci --ignore-scripts
npm --prefix tooling/payment-ui ci --ignore-scripts
PLAYWRIGHT_BROWSERS_PATH="$PWD/.cache/ms-playwright" node tooling/payment-ui/node_modules/playwright/cli.js install chromium
The connector lane, the readiness checks and the scripted runs need only
uv sync
. Node and Chromium are for the MCP Apps and browser lanes.
On Linux (Debian or Ubuntu), install Chromium with its system libraries instead. Linux is not covered by the recorded release evidence: the commands below match the CI workflow but have not been qualified here.
sudo apt-get update && sudo apt-get install -y nodejs npm # or Node 20+ from nodesource or nvm
curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync --extra openrouter --extra dev
npm --prefix tooling ci --ignore-scripts
npm --prefix tooling/payment-ui ci --ignore-scripts
PLAYWRIGHT_BROWSERS_PATH="$PWD/.cache/ms-playwright" \
node tooling/payment-ui/node_modules/playwright/cli.js install --with-deps chromium
Three features rely on macOS Seatbelt (sandbox-exec
) and report unsupported
or blocked on Linux rather than running unconfined: the --offline
readiness
mode, the network-denied validator workers behind run --surface api-tools
, and
the local-model memory profile. The connector lane, the scripted runs, the
readiness checks and the Docker-backed stdio targets do not depend on it.
Check the setup with doctor
:
uv run consumer-agent-testing-tool doctor
It checks Node against the floors the installed packages declare, the
tooling/payment-ui
packages against their pinned versions, and Chromium and
its headless shell under .cache/ms-playwright
. It reports the optional
protocol tooling, and checks Docker only for development lanes or configured
stdio MCP and CLI targets. It makes no model requests, even when an existing
configuration or --model-profile
selects a model. Each failed or blocked check
prints one line on stderr with its reason and, for a setup problem, the command
that fixes it.
To explicitly run the provider preflight, which sends two short model requests
and may incur charges, add --model-preflight
:
uv run consumer-agent-testing-tool doctor --config agent.yaml --model-preflight
Scripted runs need no model account, and the scripted commands in all three
quickstarts need no configuration file. Agent runs need a model profile and an
API key; Model providers shows how to create one. Supply
credentials through environment variables, and keep them out of scenarios and
shared evidence. consumer-agent-testing-tool.yaml
, the default settings file,
is local and git-ignored, and no profile is shipped.
This quickstart stays on loopback and needs no account. Complete Setup, then start the Cedar Stays-like test service in a second terminal. It prints one JSON line with its URLs when it is ready; Ctrl-C stops it:
uv run python -m consumer_agent_testing_tool.connector_fixture --port 8765 --variant good --auth none
Replay the example's scripted calls, with no model:
uv run consumer-agent-testing-tool connector-run examples/connector/fixture-booking.yaml \
--driver scripted --script examples/connector/fixture-booking.calls.json
The script searches rooms, gets a quote, books and gives a final answer. The
host fills the guest's email from the scenario's private:
values, which the
model never sees, and the scripted user answers each approval card from the
scenario's permissions:
rules. After the run, the host checks the fixture's
private oracle (one confirmed booking, no duplicate charges) and the booking's
status through get_booking
. It also checks that each sent call matched its
approval card (approval.dispatches_match
, approval.disclosure_bound
) and
that the final answer made no false claim and relayed the total and the
confirmation (answer.no_false_claims
, answer.relay_complete
).
The command prints the run directory and Connector report: .../report.html
,
and exits 0 when every required check passed. Runs are sealed under
runs/connector-<id>/
. Share the output of export
, not the run directory:
its private/
folder holds host-private values.
For an agent run, create agent.yaml
with the init
command in
Model providers, set OPENROUTER_API_KEY
in your shell, and run:
uv run consumer-agent-testing-tool connector-run examples/connector/fixture-booking.yaml --config agent.yaml
The model preflight runs first and prints its own evidence path. The agent then
sees the four allowed tools as native functions, with the host-private email
removed from their schemas, and writes the other arguments itself. This uses
paid model inference. To change the treatment without editing the scenario,
--projection cli
offers the same tools through one exec
command for either target,
--exposure search-only
loads their parameters through tool_search
,
--output-schema strict
refuses results that break a tool's outputSchema
as
the official MCP SDK does, and --trials N
runs independent trials and prints
pass^k. The run lock records the projection, exposure, client-profile and
output-schema overrides under overrides
.
connector-run
exits 0 when every required check passed; 1 when a check
failed, the answer made a false claim or an agent limit was reached; 2 for a
setup, harness or reporting error; and 3 when the trial is blocked or
unsupported, for example by an uncertain write to reconcile, an unobservable
check or a client-profile limit. To start from your own server, run
scaffold
; to replay a recorded, shareable agent trial without a model, turn
it into a calls file with scaffold --from-run
. The
connector lane reference covers scenarios, flags and
evidence. The examples
list each shipped scenario with its fixture command, and
lane extensions covers routes, custom
connectors, the browser sub-agent, wallets, schedules and the other
extensions.
The connector lane works with any MCP server over Streamable HTTP, not only the
bundled test service. This quickstart points it at "Everything"
(@modelcontextprotocol/server-everything
,
MIT), the reference server the MCP project maintains for client authors. Its
weather tool returns simulated data, so it needs no account or key. Start it in
a second terminal; the first run downloads the pinned release from npm:
PORT=3001 npx -y @modelcontextprotocol/server-everything@1.0.0 streamableHttp
See what the host infers from the live catalog. scaffold
runs the handshake
and tools/list
, calls no tool, and writes a scenario skeleton with suggested
tool classes from the server's annotations:
uv run consumer-agent-testing-tool scaffold http://127.0.0.1:3001/mcp --output my-everything.yaml
uv run consumer-agent-testing-tool check http://127.0.0.1:3001/mcp --suite muse-guidelines
The second command reviews the catalog statically against the pinned Muse connector guidelines. Then replay the shipped example, which asks for the weather in Chicago, first with native tools and then through the Muse-style command-line projection over the unofficial Muse template client:
uv run consumer-agent-testing-tool connector-run examples/connector/everything-weather.yaml \
--driver scripted --script examples/connector/everything-weather.calls.json
uv run consumer-agent-testing-tool connector-run examples/connector/everything-weather.yaml \
--projection muse-cli --client-profile muse-template-unofficial-2026-10 \
--driver scripted --script examples/connector/everything-weather-cli.calls.json
Both exit 0. The scenario offers the model three of the server's tools; a
reviewed allowlist keeps the rest off its list, including get-env
, which
returns the server process's environment variables, and
gzip-file-as-resource
, which fetches any URL it is given. The server has no
private oracle, so the outcome check calls get-structured-content
itself
after the agent stops, and the answer checks look for the temperature and
conditions the server returned. With a model profile, let an agent do the same
task three times and print pass^k (paid inference). The scenario declares no
reset URL, so connector-run
warns that the trials share the server's state;
this weather tool keeps none:
uv run consumer-agent-testing-tool connector-run examples/connector/everything-weather.yaml \
--config agent.yaml --trials 3
check
also runs readiness suites that read catalogs and documents instead of
running an agent. With the connector-lane quickstart's fixture still running,
review its catalog and the example manifest against the pinned Muse connector
guidelines:
uv run consumer-agent-testing-tool check http://127.0.0.1:8765/mcp --suite muse-guidelines \
--manifest examples/connector/readiness.yaml
Suite | Command | Exit codes |
---|---|---|
muse-guidelines |
check URL --suite muse-guidelines [--manifest FILE] |
0 no fail finding, 1 fail findings, 2 setup error, 3 the client profile cannot reach the server |
muse-guidelines drift |
check --suite muse-guidelines --drift-check |
0 unchanged, 1 drifted, 2 not fetched |
poppy |
check ORIGIN --suite poppy [--manifest FILE] |
0 no fail finding, 1 fail findings, 2 setup error, 3 blocked |
gadget |
check FILE --suite gadget |
0 no fail finding, 1 fail findings, 2 the document cannot be read |
contract (default) |
check TARGET [--interface-profile PROFILE] |
0 every required check passed, 1 a check failed, 2 setup, harness or reporting error, 3 blocked or unsupported |
The guidelines review lists the server's tools and calls none; --drift-check
fetches the live guidelines, and only when given. Poppy discovery only reads.
Its probes run only with --manifest
and act on the company's own hosts: they
start sessions, sign in the declared test account and confirm one declared
operation. The gadget suite reads one local file. The contract suite checks a
target named in the configuration file (--config
). For a kind: mcp
target
without declared contract_cases
, its operation-contracts
check is
unsupported and the run exits 3.
A clean readiness result is not an admission. Meta reviews connectors by hand,
Poppy is draft 0.1 with no production Muse evidence, and the Gadget SDK's terms
say it is not a developer platform. Readiness checks
describes the rules, manifests, --offline
behavior and the
mcp-contract-2026-07-28
profile.
From a source checkout or unpacked source distribution, complete Setup, then run:
uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox.yaml --driver scripted
The example creates a fictional shopper's draft in Coverage Cat's hosted
sandbox at https://www.coveragecat.com/api/consumer/mcp
. It needs no
Coverage Cat account, operator credentials or model API key.
The first umbrella_consumer_prefill
call sets sandbox: true
, as specified in
the consumer MCP manifest.
That argument activates the server's sandbox. The scenario's test_only: true
flag merely records the test author's intent.
The host fixes the tool arguments and keeps the returned access token private.
It permits two tools, umbrella_consumer_prefill
and consumer_intake_issues
,
and checks that:
- Prefill returns
sandbox: true
. - Follow-up retains
sandbox: true
. - Follow-up returns the original application ID.
- The server reports
application_submitted: false
.
Each run creates a new draft and stops before submission. A failed or missing sandbox check stops the workflow. These checks cover draft creation and follow-up, not payment widgets, browser behavior or independent backend checks.
To let a model choose the steps, create agent.yaml
with the init
command in
Model providers, set OPENROUTER_API_KEY
in your shell, and run:
uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox.yaml \
--driver agent --config agent.yaml
The agent driver is a scripted action menu: Spark Open picks one ready
scenario step per turn, and the host runs it with the scenario's arguments. The
model never sees the server's tools, schemas or raw results, only the result
fields the scenario exposes, and the evidence is labeled scripted action menu
.
The host supplies the test inputs and checks the results. This version uses
paid model inference through Together, with ZDR enabled and fallback disabled
by default.
Verified 15 October 2026: both scripted and Spark Open agent runs passed all four sandbox checks. Both model responses identified Together; both requests used the configured ZDR policy. See the recorded quickstart results.
Each command prints an outcome line, such as Surface run completed (exit 0)
,
then Surface report: .../report.html
. Open the report to inspect the workflow
and its checks. The same directory contains result.json
for the structured
result and events.jsonl
for the redacted event timeline. Agent runs also store
model and tool records under agent/
.
Replace the placeholders below with the run directories printed by your commands:
uv run consumer-agent-testing-tool report runs/surface-YOUR_RUN_ID
uv run consumer-agent-testing-tool export runs/surface-YOUR_RUN_ID
uv run consumer-agent-testing-tool diff runs/surface-BASELINE_ID runs/surface-CANDIDATE_ID
Share the generated export. Keep the full run folder private. While a run can
resume, checkpoint.private.json
holds the credentials, tool results and host
inputs it needs; a completed run, or one that failed with exit 1 and left no
write unconfirmed, keeps only a scrubbed record.
To test your own MCP service, copy the example and replace its endpoint, allowed origins, tool policies, test data and response checks. Surface workflows explains UI steps, the agent driver and resuming runs.
The checkout example adds review, submission, Monoline offer selection, consent, bind follow-up, document upload and simulated payment-method setup. Its 14 steps contain 62 assertions. All seven tools use the consumer umbrella MCP, and every service step must return the same sandbox intake.
Current integration status, 15 October 2026: blocked at submission.
Coverage Cat returns quote fields that its MCP output schema excludes. It also
returns minimum_auto_limits
as a string where the schema requires an object.
The host rejects the response. The draft quickstart passes, but the full checkout
test has not passed end to end. See the
failure report and separate diagnostics.
Run this command to reproduce the failure or retest after the service contract is corrected:
uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox-checkout.yaml \
--driver scripted \
--replies examples/coveragecat-sandbox-checkout-replies.json
The reply files supply a fictional shopper's answers: approval to submit,
credit consent, the Monoline $2M offer at $389, sufficient auto limits, declined
excess UM/UIM, and “No” to both carrier questions. These fixtures do not authorize
transactions for a real shopper. A changed quote or missing or negative approval
stops the next write. The two generated declarations PDFs are labelled
SYNTHETIC TEST ONLY
.
Once submission works, the example pauses with exit code 3 at
sandbox_document_review_poll
. Only at that checkpoint, wait at least the
reported 60 seconds and resume the printed run:
sleep 60
uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox-checkout.yaml \
--driver scripted --resume runs/surface-YOUR_RUN_ID \
--replies examples/coveragecat-sandbox-checkout-after-review.json
The host pauses for your reply but does not measure how long you wait. The after-review fixture approves the simulated payment callback.
For an agent-driven run, replace --driver scripted
with
--driver agent --config agent.yaml
on both commands. A resumed run must
keep its original scenario, driver and model configuration.
The sandbox supplies mock payment credentials. The example sends the returned
seti_sandbox_...
with the fixture pm_sandbox_consumer_agent_testing_tool
, then requires
payment_method_on_file
, waiting_on_carrier
, and no charge confirmation.
Billing checks inspect setup metadata rather than rendered fields. No Stripe
form opens, no card details are collected, and no fallback URL is used. This
test establishes neither premium collection nor active insurance. Test the
embedded payment UI and Stripe test-mode setup separately.
The quickstarts above are the pieces of one workflow. For an MCP server you own:
- Write a scenario.
scaffold URL --output my-connector.yaml
reads the live catalog (--auth-env VAR
for an API key, or--auth-from FILE
with a scenario that holds an OAuth block) and writes a skeleton. Replace everyTODO(scaffold)
: the user's goal in their own words, the allowlist of tools a reviewed connector would expose, tool classes your documentation settles, approval rules for the scripted user, host-private values such as the user's email, and checks that read the real outcome, from a private state URL (an oracle) or one of your read tools. - Check readiness.
check URL --suite muse-guidelines --manifest FILE
reviews the catalog and your declarations against the pinned connector guidelines. A clean result is a documented constraint, not an admission. - Pin the happy path. Replay a scripted calls file with
--driver scripted
; it needs no model and belongs in CI. After an agent trial you trust,scaffold --from-run RUN --output FILE
turns it into one. - Run agent trials.
--trials N
with a model profile gives pass^k. Run the same scenario under each treatment a Muse-like host may use:--projection muse-cli
,--exposure search-only
,--client-profile muse-template-unofficial-2026-10
and--output-schema strict
. The run lock records each one. - Add realism. Lane extensions add competing connectors and connect-on-request, payment with single-use cards, background jobs, untrusted content, channel limits and a simulated user.
- Compare builds.
diff
compares two sealed runs and exits 1 on an outcome regression. Share the output ofexport
; the run directory'sprivate/
folder holds host-private values.
For an MCP app with its own UI, start from the Coverage Cat scenario and
surface-run
. For a website, use browse
with a browser policy, or the
connector lane's browser
extension. For an HTTP API, write a mapping for
run --surface api-tools
, or test whether an agent can build its own client
from your docs with the custom_connector
extension.
Run a consumer test covers the other lanes.
Create a profile with init
. This one uses Muse Spark Open
(meta/muse-spark-open
) through OpenRouter, routed only to Together, with zero
data retention (ZDR) required and fallback disabled; set OPENROUTER_API_KEY
in
your shell before an agent run:
uv run consumer-agent-testing-tool init --provider openrouter --model meta/muse-spark-open \
--routing-provider together --config agent.yaml
init
also stages, beside the configuration, the API scenarios, the browser
scenarios and the mapped-API example. It stages only scenarios the supported
CLI can run, and refuses providers that have only a configuration template
(openai
, anthropic
, gemini
, azure-openai
, vertex-gemini
,
openai-compatible
) unless --development-extensions
comes before init
.
init --provider none --config scripted.yaml
writes a configuration for
scripted commands that need one. init
refuses to overwrite an existing file.
Profiles that init
writes are labeled specified/unverified
. doctor --model-preflight
, run
and the agent drivers of connector-run
, surface-run
and browse
print a
warning for any profile not labeled supported
, and the run lock records the
label. Model IDs that contain contributor
select a mode in which the provider
may train on your prompts; such a profile needs
allow_training_on_prompts: true
(init --allow-training-on-prompts
). See
model providers for every adapter, its checks and
its failure labels.
Consumer Agent Testing Tool requests zero data retention (ZDR) by default, even when
routing.zdr
is omitted. To change this, edit the selected profile in
consumer-agent-testing-tool.yaml
or the file passed to --config
:
model_profiles:
openrouter-dev:
provider: openrouter
api: chat-completions
model_id: meta/muse-spark-open
auth:
mode: api_key
env: OPENROUTER_API_KEY
routing:
only: [together]
allow_fallbacks: false
require_parameters: true
data_collection: deny
zdr: true # Set to false to disable this tool's per-request ZDR requirement.
defaults:
model_profile: openrouter-dev
Use unquoted YAML booleans: true
or false
. Consumer Agent Testing Tool applies the setting to
every OpenRouter request, including requests that continue after a tool result,
and records it in the routing evidence.
Setting zdr: false
removes this request-level requirement. It leaves
data_collection: deny
, the pinned provider and the ban on fallbacks in place.
Your OpenRouter account or guardrails may still enforce ZDR. Request logs show
the policy sent; they cannot independently verify the provider's retention
practices. See OpenRouter's ZDR documentation.
Checked 15 October 2026: OpenRouter listed Together for Spark Open in both its model and ZDR endpoint catalogs. A live tool-call test passed with the routing settings above. Both responses identified Together, and the model used the tool result correctly. See the test record.
To choose another provider, set routing.only
or pass --routing-provider
to
init
. Requests never switch
providers or models silently.
Runs use Inspect's dedicated openrouter
provider, which replays earlier
reasoning in structured form rather than as text. Each response must name the
pinned provider. Responses do not name endpoint variants, so a pin such as
nebius/fp4
is recorded as variant unverified
. Streaming is not supported.
See model providers.
With zdr: true
, only ZDR-listed endpoints are eligible. On 15 October 2026,
Muse Spark 1.3 and Muse Wonder were served only by Meta on OpenRouter and were
not ZDR-listed, so no ZDR-compliant OpenRouter route existed for them. A
request for them is expected to fail, but no live run has recorded the error.
When OpenRouter's error cites the data policy, it is labeled
no endpoint satisfies the routing/ZDR policy
. With routing.only: [together]
,
the init
default, it may instead be labeled
routing policy: no allowed provider
.
Sources: Meta announcement, OpenRouter model, provider endpoints, and ZDR endpoints.
The standalone smoke test and older consumer helper read the same --config
and --model-profile
, including privacy settings. Profiles that init
writes
leave reasoning effort unset.
.venv/bin/python scripts/openrouter_smoke.py --config agent.yaml \
--model-profile openrouter-dev
This command tests the model connection with synthetic tool data. It needs an API key and an available route to the pinned provider. It makes no requests to the service you plan to test.
init --provider meta
writes a profile for Meta's Model API:
uv run consumer-agent-testing-tool init --provider meta --model muse-spark-1.3 --config meta.yaml
The profile uses the Responses API at https://api.meta.ai/v1
and reads the key
from MODEL_API_KEY
or META_API_KEY
; if both are set, they must agree. Add
--api chat-completions
for Chat Completions, or --key-env NAME
to use another
variable, which is then the only one read. The adapter uses the OpenAI SDK from
the openrouter
extra.
The host refuses requests that Meta rejects, such as a forced tool choice, before sending them (exit 3). Meta adds steering context to every prompt on its servers, so captured requests are incomplete, and every Meta run records this as an observation gap. Streaming is not supported, and the API is pay-as-you-go. See Meta Model API. No live Meta Model API run has been recorded; the adapter is tested against local fixtures.
This section covers the embedded MCP apps (surface-run
), browser (browse
)
and HTTP API lanes. To let an agent use your MCP server's own tools, use the
connector lane.
A scenario defines the endpoint, allowed origins, tools, private arguments, UI
actions, consent requirements and result checks, and optionally an API key
for the MCP connection (auth
). Use endpoints and synthetic transactions you
have permission to test.
consumer-agent-testing-tool surface-run path/to/scenario.yaml --driver scripted
consumer-agent-testing-tool surface-run path/to/scenario.yaml --driver agent --config agent.yaml
consumer-agent-testing-tool surface-run path/to/scenario.yaml --resume runs/existing-run
consumer-agent-testing-tool browse path/to/browser-policy.json --actions path/to/actions.json
The host declares MCP Apps support as a client, finds each tool's UI through its
tools/list
definition and renders it in a native browser. With
--driver agent
, the model picks among the scenario's steps (a scripted action
menu) and never sees the server's tools, raw results or private inputs. Service
checks determine the result; a rendered form or a model's claim of success
cannot prove that a transaction completed.
browse
drives a restricted browser through accessibility snapshots and element
references, with no selectors, scripts or CDP. Its policy file sets the start
url
, the permitted origins
and request rules
, snapshot_max_chars
(default 20,000), history_snapshots
(2) and dialogs
(agent
, accept
or
dismiss
), as in examples/browser-policy.json.
Long snapshots are paged with {"kind": "snapshot", "offset": N}
. Refused or
failed actions come back to the agent as observations with a status
, and
dialogs and file choosers are answered with kind: dialog
or kind: upload
.
With --driver agent
, the provider preflight runs before the browser opens.
The result's observations
block counts truncated snapshots and refused
actions. When the agent requests a takeover, the run pauses (exit 3). The CLI
has no handback, so control does not return to the agent in that run. In
native mode, WebSockets opened by dedicated workers and WebRTC TCP relays are
not enforced. The policy field webmcp: true
, off by default, lists
page-registered WebMCP tools in the snapshot and lets the agent call them with
kind: webmcp
; that lane is labeled "generic WebMCP page tools, not Muse"
(basis: assumed
), and its evidence notes that Meta documents WebMCP for its
AI glasses, a Meta product that is not Muse. See
restricted browser behavior.
HTTP API tests use explicit mappings with the scenario runner. Mappings can
send declared request headers and put the target credential in a custom API-key
header. In API and browser trials, invalid tool proposals are returned to the
model as tool errors, and exhausted limits fail the trial; see
API scenarios. To test
whether an agent can build its own client from an API's documentation instead,
use the connector lane's custom_connector
extension. Prism, Schemathesis and
generated-OpenAPI tools are available only as development extensions. Their
five packages with unresolved original license notices are excluded from the
supported release, not treated as cleared.
surface-run
follows the exit codes in spec §18:
Exit | Meaning |
---|---|
0 | Completed |
1 | Failed: an assertion, a service rejection, a UI interaction failure or an agent error |
2 | Setup, infrastructure, integration or reporting failure |
3 | Blocked or unsupported: waiting for a reply or approval, an unconfirmed write, a limit, or missing coverage |
Before Surface report: ...
it prints one line:
Surface run <status> (exit N) at <stage> [<class>]: <reason>
. result.json
records the same failure_class
and failure_stage
, and whether the run is
resumable
. A completed run, or one that failed with exit 1, is final unless
the failure left a write unconfirmed: that write may have taken effect, so the
run stays resumable to reconcile it. Blocked runs and setup failures can resume.
run
and browse
use the same four codes, except that exhausting a turn,
tool-call or time limit is a failure (exit 1). Only Inspect operator and custom
limits, and in run
a deadline that passed before an attempt's first
generation, block instead (exit 3). run
prints each evidence path on stdout.
browse
prints the final observation and its evidence path when the run
completes or pauses for takeover, and nothing on stdout otherwise. For any
outcome other than completed, both also print one line on stderr:
<label> <status> (<class>) at <check>: <reason>. Evidence: <path>
. The label
names the step, for example Model preflight
, Trial 2/10 of stale_total
,
Resumed trial of stale_total
or Browse
(doctor
uses Doctor check
). A
run
with a failed or blocked trial, or with --trials
above 1, also prints a
summary line with the counts. With --trials
above 1 it adds the suite
pass^k, the mean over scenarios, for example
Run completed: 0 failed, 0 blocked and 4 completed of 4 trials; pass^1 1.00, pass^2 1.00. Evidence: ...
.
The suite's metrics also break the trials down by scenario
(statistical decisions). run DIR
refuses
(exit 2), before the preflight, a directory whose scenario files share an id.
connector-run
prints the same stderr line, labeled for example
Connector trial 1/1 of fixture-booking-happy-path
; its codes are listed in the
connector-lane quickstart. With --trials N
, one
code stands for all trials: 2 when any trial's evidence is not shareable, else 1
when any trial failed, else 2 when any hit a setup, harness or verification
error, else 3 when any was blocked, else 0. The Connector trials:
line on
stderr gives pass^k and names the trials that are not shareable or ended in an
error. The readiness suites' codes are in Readiness checks.
The embedded-app lane is generic MCP Apps host evidence (MCP Apps 2026-01-26),
for hosts such as ChatGPT or Claude. It is not Muse evidence: Muse documents no
third-party UI. Each result.json
says so under mcp_apps_host
.
The host implements:
- the client capability (
io.modelcontextprotocol/ui
withtext/html;profile=mcp-app
), UI discovery from the tool'stools/list
definition, and HTML resources as text or base64 blobs; tool-input
with the call's arguments,tool-result
, and ahostContext
(theme, display mode, locale, time zone, platform, container width and maximum height, and tool information);- a content security policy built from the resource's
_meta.ui.csp
, or the specification's restrictive default when the resource declares none, and the declared camera, microphone, geolocation and clipboard-write permissions; - in the native renderer, a sandbox proxy on an origin separate from the host
page. With
browser_origins
, the host page also has Chromium's local-network-access permission, every redirect hop is checked again (GET or HEAD to an allowlisted origin only), and each direct write is recorded as an unconsentedhost_action
before it is sent; - view
tools/call
requests for the step's allowlisted tools, andresources/read
andresources/list
for the step'scallback_resources
; ui/message
, delivered once to agent runs in the next observation a model reads, asapp_messages
; the view is told when the host does not deliver it.ui/update-model-context
,ui/open-link
, display-mode, log and teardown requests are recorded, and the latest model-context update reaches agent runs asapp_context
;- size changes,
ui/resource-teardown
before the view closes, andtool-cancelled
when the view never received the call's result; _meta.ui.visibility
, for view calls and for tool steps;- MCP 2026-07-28 servers, when the scenario sets
version_negotiation: auto
. The default is the 2025initialize
handshake; - an API key on the MCP connection (
auth
), as a bearer token, a custom header or a query parameter; - a legacy
window.openai
bridge.
It does not implement a model turn for a view's message, resource templates,
_meta.ui.domain
, display modes other than inline, partial tool input,
host-context change notifications, OAuth on the MCP connection, or consent for
a view's direct writes in the native renderer. The legacy bridge and the
isolated renderer have no sandbox proxy. See
MCP Apps host.
consumer-agent-testing-tool report runs/existing-run
consumer-agent-testing-tool export runs/existing-run
consumer-agent-testing-tool replay runs/existing-run
consumer-agent-testing-tool diff runs/baseline runs/candidate
Replay checks the evidence seals and stops on a mismatch. It never substitutes a live request. To test the service again, start a new run. Reports mark missing measurements as unknown and keep partial totals distinct from complete totals.
Reports open with the verdict, which names the failing step and check, then
the checks and host capabilities, and a per-step timeline that shows what the
model received in full. Then come a connectors table for runs with several
connectors, the surface views (what the service published, what the model was
offered, what it received at each step, and what only the developer sees),
the metrics, and the viewers and linked evidence: agent and preflight runs,
their Inspect logs and the inspect view
command, and, locally, a captured
Playwright trace. Gaps and limits follow, then the run lock and the full
recorded result, each collapsed; see
report layout. report
on an agent
sub-run of a sealed run returns its sealed report unchanged.
Evidence files and exports hide credentials, registered secrets, contact details and other personal fields such as names and addresses. Model views keep the user's own data visible: they hide only credentials and declared secrets, plus browser password fields and, in a resumed browser scenario, text typed in an earlier attempt. The provider boundary refuses any model request that still contains a registered credential. Export also applies the run's own private values, which are never exported, and refuses (exit 2) when a staged file, or a member of an Inspect log or other archive, still holds one. See redaction.
diff
compares two sealed runs, such as surface, browser or connector runs. A
change to the package source counts as a run-lock change, so compare runs made
from the same checkout. It exits 0 when they are
comparable with no outcome regression, and 1 when an outcome regressed: a worse
status, or a passing check that no longer passes or is missing. It exits 3 when
they are not comparable, because of a run-lock change outside --treatment
or
an unsealed or unverifiable run. Generated IDs and service timestamps are
normalized; run locks are compared exactly. --ignore POINTER
(repeatable)
excludes a JSON pointer under /outcomes
, /capabilities
or /milestones
,
and --treatment /server
declares an intentional server build change. See
replay and diff.
For more detail, see the documentation index, connector lane, lane extensions, readiness checks, consumer tests, browser diagnostics, measurement coverage, model timing, and release extensions, which includes development history.
Scripted connector runs need no model and no key, and they replay the same calls every time, so they
work as a pull-request check. connector-run
can start the service itself and write reports a CI
system reads:
uv run consumer-agent-testing-tool connector-run my-connector.yaml \
--driver scripted --script my-connector.calls.json \
--start "python -m my_service --port {port}" \
--junit reports/connector.xml --summary-json reports/connector.json
--start COMMAND
runs the service on a free loopback port. The command is split like a shell command but not run in a shell.{port}
in the command is that port. The service also gets it asPORT
and the scenario as${PORT}
.- The tool waits until
--ready-url
, or the scenario's first connector endpoint, answers HTTP. It waits at most--ready-timeout
seconds (default 30). - When the run ends, the service's process group gets SIGTERM, then SIGKILL after 5 seconds.
- Its output goes to
runs/service-<id>.log
, outside the sealed run. That log isn't redacted, so pass secrets to the service through the environment. The run lock records the command with credentials redacted. - A service that exits early or never becomes ready exits 2.
--junit FILE
writes one test suite per trial and one test case per check. Blocked checks are<skipped>
.check --suite muse-guidelines
accepts it too.--summary-json FILE
writes the verdict, exit code, checks, trials, pass^k and run paths. Its format isschemas/summary-v1.schema.json
.--summary-json -
prints only that JSON on stdout and moves the other lines to stderr.- Both reports are also written when the command stops with an error. See CI outputs and services.
The exit codes in Outcomes and exit codes decide the step. A 3 (blocked
or unsupported) is not a pass.
examples/ci/connector-lane.yml
is a GitHub Actions template with
two jobs. On pull requests, it runs the scripted calls and the readiness review. Nightly or on demand,
it runs agent trials under each Muse-like treatment, and only when a model key secret is set.
.github/workflows/ci.yml
runs this repository's tests and shipped
examples on macOS and Linux. Coding agents should start with AGENTS.md
.
All matrix jobs feed the required CI
check; Linux failures are not allowed to pass silently.
Workflow validation, tracked-file hygiene, secret scanning and schema freshness are checked too.
Plain pytest
runs the supported collection:
uv run pytest -q
tests/conftest.py
gives each session a temporary directory, home and cache
under .cache/pytest/
, removes them after a passing session, and removes the
surface and browser run directories its tests create under runs/
. The full collection takes about
twenty minutes and makes no network request beyond loopback and no model
request. pytest --development-extensions
also collects the experimental
tests, which may need other runtimes, tools and external evidence.
Async cleanup and unhandled thread exceptions fail tests instead of remaining warnings.
See CONTRIBUTING.md for the review workflow and
the launch checklist for the private-to-public transition.
The supported release is local-consumer-v1
: the connector lane and its
extensions, the readiness checks, consumer MCP apps, the restricted browser,
mapped HTTP APIs, transactional consent, continuation, evidence and replay,
measurements, the distribution, and the OpenRouter and Meta Model API
adapters. Real-Muse calibration, cloud provisioning, local Glimmer, browser OS
isolation and live Bedrock qualification are outside it; see
release scope for the full boundary.
Release evidence is generated per release and is not committed.
docs/release-gates.json
is the committed policy:
the gates, the scope and the exclusions. The
release workflow runs on a v*
tag. It records
every gate on macOS, requires the audit to pass, and attaches three things to
the GitHub release:
- the wheel and the source archive, with signed build provenance;
release-evidence-<commit>.tar.gz
, which holdsrelease/release-gates.json
(the recorded ledger) andrelease/evidence/
.
Use consumer-agent-testing-tool release-audit
to check release readiness. It
reads release/release-gates.json
when evidence has been recorded or extracted
there. Without it, it reads the committed policy and reports every gate
unverified. Exit code 0 means release-ready for local-consumer-v1
; exit
code 3 means at least one required gate needs attention. The evidence binds a
digest of the source, so a source change or an expired gate (seven days after
recording) cannot inherit an earlier passing result. To audit a published
release, check out its tag, extract its evidence archive at the repository root
and run the audit before the gates expire.
Day to day, you don't need any of this. Run the tests for what you change, and let CI run the full collection on each pull request.
A separate live Spark Open test passed through Together with ZDR enabled. It verified native tool calls, continuation after a tool result, and the requested routing policy. It did not test streaming, independently verify retention practices, or establish compatibility with every consumer workflow. The Coverage Cat checkout failure described above remains an open integration issue.
The audit requires five gates:
Gate | Evidence required |
---|---|
Regression | Every test in the default supported collection passes, with no skips |
Distribution | Fresh offline installation, original dependency notices, artifact integrity and packaged consumer workflows |
Transactional consent | Passing consent, continuation and effect checks in the current regression |
Replay/diff | Passing evidence-integrity and comparison checks in the current regression |
Surrogate parity | Exact Muse/Instinct native/CLI cases, memory lifecycle, routing, payment, readiness and extension coverage |
To record evidence locally instead, for example before tagging, run the same
steps as the workflow, in this order. verify_distribution.py
regenerates the
dependency notices, which are part of the source fingerprint, so the
distribution comes before the regression. The regression takes about twenty
minutes. Everything is written under release/
, which git ignores:
.venv/bin/python scripts/verify_distribution.py
.venv/bin/python scripts/record_distribution.py
.venv/bin/python scripts/run_regression.py release/evidence/regression-run.xml
.venv/bin/python scripts/record_regression.py release/evidence/regression-run.xml
.venv/bin/python scripts/record_release_tests.py release/evidence/regression-run.xml
consumer-agent-testing-tool release-audit
run_regression.py
refuses an existing report path, so remove release/
or
pick a new name to record again. It records workspace-relative paths and writes
the home directory as ~
in the log.
Distribution checks use caches in the workspace and the installed browser.
They need no Docker VM, cloud worker or model credentials. If a dependency cache
is missing, populate it separately with
scripts/verify_distribution.py --prepare-cache
; that setup step uses the network.
The installation evidence covers macOS only.
The audit verifies source fingerprints, artifact hashes, required checks and expiry dates. It returns 0 when every required gate passes, or 3 otherwise. Excluded capabilities remain labelled as excluded. Qualification inputs explains what each gate needs.
The repository retains worker, isolation, local-model, calibration and
generated-OpenAPI experiments, and the CLI/skill lane for a developer's own CLI
binary (run --surface cli-skill
), which needs Docker and an oracle that
returns the reservation fixture's snapshot schema. The normal CLI rejects these
paths. To use them, put --development-extensions
before the subcommand and
install the separate development-extensions
dependency group and any required
tooling. These experiments are outside the supported release. The supported way
to give an agent a CLI over an MCP server is
connector-run --projection muse-cli
.
The optional Node API tools have known dependency advisories. Do not use them with untrusted inputs while those findings remain open.
The original specification and earlier test records document the project's history; they do not define the scope of this release.
MIT; see LICENSE. The license does not cover the Coverage Cat and CATT names or
the logo; see TRADEMARKS.md. Third-party components keep their own
licenses; the distribution ships their notices under third_party/
.