# AI user tests An AI user test gives an agent a task written in plain English and a URL, and lets it work out its own way there in a real browser. You get the recording, every step it took, and the friction it hit. Findings you keep land in your Usero feedback inbox, where they cluster with your other reports and can become an AI-authored pull request. Read the limits before you read your first run: the per-step loop reads page structure rather than pixels, for cost reasons rather than because it cannot, and roughly half of what a run flags on a given night is worth acting on. It finds a different category of problem than a person does. It does not replace user testing with people, and Usero ships both. ## Before your first run You need three things. **A staging environment.** Runs click, type and submit on their own. The setup form warns you when the host does not look like staging, and you should take the warning seriously. Production is not a supported target. **A way in.** The usual one is a test account you created yourself, scoped to nothing you care about. Do not reuse a real account, and do not use credentials that also work in production. A run can also sign itself up, if you pick that option: it gets its own throwaway mailbox for the length of the run, so it can finish a magic link or a verification email. Bot protection still stops an agent at signup on most modern SaaS, so pre-creating an account is the shorter path when you only want the flows behind the login. **One task, written as a goal.** Start with one. Add more once you have seen what a run looks like. ## Writing a task Write the outcome you want, not the clicks that get there. The point is watching it find its own way. ``` Good: Sign up for an account, create your first project, and invite a teammate. Good: Find the billing page and change the plan to the annual option. Bad: Click Sign up, enter an email, click Continue, then click Create project. ``` A task written as steps turns the run into a slow test script and removes the only thing that makes it useful: the chance that it takes a route nobody wrote down. Some things that make tasks work better: - **One outcome per task.** "Sign up and create a project" is one arc. "Sign up, create a project, invite a teammate, export a CSV, and change your plan" is five, and a failure early on hides everything after it. - **State the starting condition when it matters.** "You are already logged in as an existing user" saves the run from attempting a signup that was never the point of the task. - **Name the goal in the user's words, not your internal ones.** If your team calls it a workspace and the UI says organisation, write the task the way a new user would think about it. Mismatched vocabulary is a finding in itself. - **Avoid tasks that need a code out of band**, such as an SMS confirmation, unless you have a way to hand it over. ## Test accounts and access The account you hand over should be able to complete the task and nothing else that matters. - Scope it to a throwaway workspace or organisation with no real customer data in it. - Turn captcha off on staging for the flows you want covered, or hand the run an account that is already past the wall. If your staging signup is behind a captcha, expect signup itself to be untestable until you do one of those two things. - Email verification needs a mailbox the run can reach. Pick "Let it sign itself up" and the run gets one: an address of its own that lives for the length of the run and is emptied when the run ends. It only follows links that point back at the host you set as the target, so a stray email cannot send it elsewhere. - Credentials are injected per run and are not persisted in the run history. ## Safety Every run is fenced by default: - **Origin lock.** The run stays on the origin you configured. Links that leave it are not followed. - **Blocked actions.** Payments, deletions and outbound email to addresses outside your configured domain are refused. - **Step and time caps.** A run stops at its step cap rather than wandering indefinitely. None of that makes a run safe against a production database. Point it at staging. ## Reading a run A run page has three parts. **The recording.** An rrweb session replay in the same player as your other session replays. Each finding deep-links to its moment, so you can watch what happened instead of taking the write-up on trust. **The step trace.** Every action in order, with what the run expected and what it got. This is where a wrong finding usually explains itself. **The findings.** Split into what the run was sure about and what it was not, each typed against a fixed set of categories: | Category | What it means | | ------------------- | ------------------------------------------------------------------- | | Dead end | The path stopped and there was no way forward from where it landed. | | Broken flow | A step that should have advanced the task did not. | | Ambiguous label | A control said one thing and did another. | | Missing affordance | The next step existed but nothing on the page pointed to it. | | Unrecoverable error | An error state with no way back. | | Copy confusion | Wording that sent the run the wrong way. | There is no usability score. A single number would invite the claim that this replaces research, and it does not. Findings marked less certain are the ones where the run could not tell whether your product or its own confusion caused the problem. Read them as questions. Bin what is noise; dismissing a finding teaches the dedup pass not to file it again next time. ## Sending a finding to the inbox Keep the real ones and send them through. A finding arrives in your feedback inbox with the category, the severity the run assigned, what it tried, where it got stuck, and a link into the recording at that moment. From there it behaves like any other feedback: it clusters with related reports and can go to an AI-authored pull request that you review and merge. Nothing merges on its own. Runs are path non-deterministic, so the same task can take a different route on two consecutive nights. Dedup fingerprints the friction rather than the step number, which is why the same snag does not re-file every night. ## A sensible schedule Two or three tasks against staging, run nightly: signup, the first real action in your product, and whatever you shipped this week. Most nights come back clean. Quiet is the normal result, and the value is the night it is not. ## Starting a run from CI Everything the runs page does, a script can do. Start a run against your staging deploy, poll it, and decide whether to promote the build. Authenticate with an API key from Profile, API keys, sent as a bearer token. The key belongs to you and reaches every project you are a member of; a key that is not on a project gets a `404`, the same answer as a project that does not exist. ```text title=Endpoints GET https://usero.io/api/v1/clients/{clientId}/ai-user-tests POST https://usero.io/api/v1/clients/{clientId}/ai-user-tests GET https://usero.io/api/v1/clients/{clientId}/ai-user-tests/{runId} ``` ### Start a run ```bash title=curl curl -X POST https://usero.io/api/v1/clients/YOUR_CLIENT_ID/ai-user-tests \ -H "Authorization: Bearer $USERO_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "Signup, build 4821", "targetUrl": "https://staging.example.com", "taskPrompt": "Sign up for an account and create your first project.", "environment": "staging", "access": { "mode": "password", "email": "runner@example.com", "password": "..." } }' ``` | Field | Type | Required | Description | | ------------- | ------ | -------- | ------------------------------------------------------------------------------------------------ | | `name` | string | yes | Up to 120 characters. Put the build or commit in here so a finding traces back to what shipped. | | `targetUrl` | string | yes | Where the run starts. A bare host gets `https://` prepended. Point it at staging. | | `taskPrompt` | string | yes | Up to 1000 characters. Write the outcome, not the clicks. See [Writing a task](#writing-a-task). | | `environment` | string | no | Which inbox the findings land in. Omit it and they go to your default environment. | | `access` | object | no | How the run gets in. Defaults to `{ "mode": "none" }`, a logged-out visitor. | The access modes are exclusive, and each takes its own fields: | `access.mode` | Fields | What happens | | ------------- | ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | `none` | | The run sees what a logged-out visitor sees. | | `password` | `email`, `password`, `loginUrl` (opt) | It types them into your sign-in form once, at the start. | | `session` | `storageState` | It starts already signed in. Send the contents of a Playwright `auth.json`, either as a string or as the parsed object. | | `inbox` | | It gets a throwaway mailbox for the run, so it can finish a verification email. | Credentials and pasted sessions are used for the length of the run and nothing else. They are not stored, not written to the run history, not logged, and not returned in any response. Rotate them anyway: they are typed into a browser you do not control. The response is `201` with the run under `run`, already carrying its status. A run whose dispatch failed comes back as `status: "failed"` with an `errorMessage` rather than as `queued`, so a poll loop is never started against a run that will not move. ### Read a run ```bash title=curl curl https://usero.io/api/v1/clients/YOUR_CLIENT_ID/ai-user-tests/RUN_ID \ -H "Authorization: Bearer $USERO_API_KEY" ``` You get the status, how it ended, the step count, the replay id, and the findings with their categories, severities and confidences. Add `?include=steps` for the full action trace, which is left out by default because it dwarfs everything else on a poll. ### Deciding pass or fail Gate on `verdict`. It is one value covering everything a reader is allowed to conclude, and it exists because "no findings" is not the same claim as "the product works". | `verdict` | What it means | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------- | | `in-flight` | Queued or running. Nothing is final. Keep polling. | | `clean` | It got through the task and flagged nothing. The only value that is a pass. | | `findings` | It got through the task and flagged something. Read them before you block the build; roughly half are worth acting on. | | `did-not-finish` | It gave up, or ran out of steps or time. It tested nothing past where it stopped, so a finding count of zero says nothing here. | | `stopped` | The run failed or was cancelled before producing a result. This is a problem with the run, not evidence about your product. | | `unknown` | No record of how it ended. Not a pass and not a failure. | Treat anything other than `clean` and `findings` as "no result", not as a failing build. A run that hit a login wall on step one tells you about your test account, and failing the deploy on it trains your team to ignore the check. ```bash title=poll until the run settles while :; do verdict=$(curl -sf "https://usero.io/api/v1/clients/$CLIENT_ID/ai-user-tests/$RUN_ID" \ -H "Authorization: Bearer $USERO_API_KEY" | jq -r .verdict) [ "$verdict" = "in-flight" ] || break sleep 10 done echo "verdict: $verdict" ``` ### List runs ```bash title=curl curl "https://usero.io/api/v1/clients/YOUR_CLIENT_ID/ai-user-tests?environment=staging" \ -H "Authorization: Bearer $USERO_API_KEY" ``` Runs are environment-scoped, matching the dashboard: a staging run and a production run never appear in the same list, because their findings go to different inboxes. Omit `environment` for your default one. The 50 most recent runs come back, newest first. ## Troubleshooting **The run got stuck at a captcha.** Turn it off on staging for that flow, or start the run from an account that is already past it. **A finding is clearly wrong.** Open the step trace at that step and check what the run could see. Dismiss it, and it stops being filed. If a whole run reads as fiction, check whether the target was mid-deploy: a deploy landing during a run produces findings about a half-loaded application. **No recording for a run.** The step trace is still complete. Recordings are missing when the browser session drops before the first snapshot uploads. **Everything came back clean and you do not believe it.** Likely. A run is more patient and better at reading a page than any of your users, so it underreports friction. A clean run is weaker evidence than a person completing the same task cleanly.