AI user tests

An agent runs a task through your product every night and tells you where the path confused it.

Write the task in plain English, point it at staging with a test account, and it opens a browser and works out its own way there. You get the recording, every step it took, and the friction it hit. Each finding you keep lands in the Usero inbox, where the AI can draft the fix as a pull request.

Start free, no cardRead the docs
Nobody notices the plan picker lost its back button until a user emails about it three weeks later. Your tests all passed. Nothing in the diff looked wrong.

Structural friction is the category that leaks. A button that only appears after email verification, a dialog whose close X sits under a sticky header, a link pointing at an app slug that was renamed last quarter. Unit tests do not catch it because every file is individually correct. End-to-end tests do not catch it because they follow the path you already knew about, and they assert on selectors rather than on whether a person could find them. Code review does not catch it because two correct files can still collide at runtime. So it sits there until someone hits it, and for a team of four with no research function, "someone" means a paying customer. The gap is that nothing attempts your product the way a stranger would, on a schedule, without a script.

How it works

AI user tests in Usero.

Disclosure: I build Usero, so weigh that. A run takes two inputs, a task written as a goal and a staging URL with a test account you control. Claude drives a real browser against it, reading page structure each step, deciding its own next click, and logging what it expected against what happened. When it finishes it hands back three things: an rrweb recording you scrub in the same player as your session replays, the full step trace with timestamps, and a findings list sorted into what it was sure about and what it was not. Findings are typed against a fixed taxonomy, dead end, ambiguous label, broken flow, missing affordance, unrecoverable error, copy confusion. No usability score, because a number invites the claim this replaces research. You skim the list, dismiss what is noise, and send the real ones to the inbox. From there the existing pipeline applies: cluster it, open a pull request against the repo, review the diff yourself. Nothing merges on its own.

It finds the bugs that live between two correct files

Running this against our own preview deploy turned up a sticky dialog header painting over the close button, from two components that are each fine on their own. It also found a Connect GitHub button pointing at a GitHub App slug that no longer exists, because the value came from a CI variable that was never renamed. That one lives outside the repository. Grep cannot see it, typecheck cannot see it, and a reviewer reading the diff cannot see it.

Unscripted, so it can dead-end where nobody thought to look

A regression test walks the path its author wrote down. A run gets a goal and picks its own route, so it can wander into the flow you forgot existed. Two runs of the same task took different paths on our own product, one through the dashboard and one through Forms. The test passing and the agent trying three other things before finding the button are different pieces of information.

The recording and the step trace, so you can check its work

Every finding carries the step it came from and a timestamp into the recording, so you can watch the moment rather than take the write-up on faith. The trace shows each action, what it expected, and what it got. When a finding is wrong, and some will be, thirty seconds of playback tells you.

Findings become pull requests, not another report

A friction report on its own is one more tab to check. Send a finding to the inbox and it joins the rest of your feedback, gets deduped against earlier runs, and can go straight to an AI-authored pull request you review and merge. That is the difference between knowing about the broken back button and having the fix waiting when you wake up.

Usero AI user tests run list showing three runs, two with flagged findings and one clean
The run list. Two runs flagged something, one came back clean. Most nights on a healthy product look like the third row.
A single AI user test run showing findings split into "Worth a look" and "Less certain"
One run. Findings split into what it was sure about and what it was not, each with the step number it came from and a Send to inbox action.
Step trace listing 23 browser actions with timestamps, beside the limits panel
Every step it took, with what it expected at each one. The limits panel sits on the run page, not only on this site.
New run form with a red warning that the entered host does not look like staging
Starting a run. It stays on the origin you give it and skips payments and deletes, and it argues with you when the host does not look like staging.

The limits

What a run cannot see, said up front

The name says user testing and the software is not a user. It is worth being blunt about that before you run one, because the first run is where people decide whether to trust this. Here is what it misses and how often it is wrong.

  • It only looks at the pixels once, at the end

    Screenshots go to the pass that writes up severity. The per-step loop works from page structure alone, which is a cost call rather than a law: a screenshot in that loop measured 10.7x slower per step, so contrast, visual hierarchy, clutter and "I did not notice the button" go unremarked, and a control that is technically present and visually buried reads to a run as present. A per-step visual pass is an option we have not taken, and vision models can already assess contrast and clutter to a degree. What that would still not give you is taste, or a sense of what a hurried person fails to notice, so the perceptual judgement stays a job for a person.

  • It is more patient and more web-literate than any of your users

    It does not get bored, it does not get annoyed and leave, and it will happily find the aria-label your customer never would. That makes it underreport friction. When a run completes a task cleanly, that is weaker evidence than a person completing it cleanly.

  • It finds different problems than a person does, so it does not replace research

    Published measurements of agreement between agent-identified and human-identified usability problems come out worse than chance. That is the strongest argument against this whole category and we think it is correct. Complementary coverage is the claim. Substitution is not, and anyone selling you substitution is selling you something.

  • Roughly half of what a run flags is worth acting on

    Measured on products we did not build, 5 of 8 findings were real, and 4 of 8 cleared the harsher bar of something a founder would actually fix. Expect to bin some of every run. The findings list is sorted by how sure it was for exactly that reason, and low-confidence items are phrased as questions rather than verdicts.

  • It needs a test account, and it is not a marketing-page scanner

    This works on authenticated in-product flows, which is where we measured 62.5% of findings as real against 12.5% on public marketing pages. It needs a scoped account on staging that you create. It does not sign itself up, partly because that is the honest surface and partly because bot walls stop an agent at signup on most modern SaaS.

The honest objection

I could point Claude at my own app and do this myself. Why pay for it?

You could, and if you only want to do it once you should. We built the first version of this by hand in an afternoon. What took the rest of the week was the part that makes it useful on the tenth night rather than the first: an observation layer that only shows the agent what a person could actually see, because ours quietly manufactured false findings twice before we caught it; dedup that fingerprints the friction rather than the step number, because the same task takes a different route on different runs; the recording captured and played back; and the pipe from a finding to a reviewed pull request. If you want to build that, the honest answer is that it is a week of your time and then it is yours.

FAQ

Quick answers about ai user tests.

Does this replace user testing with real people?

No. The per-step loop runs on page structure rather than pixels, because a screenshot there measured 10.7x slower per step, so anything perceptual goes unremarked. That much is a cost call we could revisit. Published research also puts agreement between agent-found and human-found usability problems at worse than chance, and no amount of pixels gives an agent taste. It finds a different category of problem: dead ends, broken flows, labels pointing at the wrong thing. Keep running sessions with real people for everything perceptual, and for hearing someone hesitate.

How is this different from end-to-end tests or an agentic QA tool?

QA verifies a flow you scripted and tells you whether it still passes. A run gets a goal instead of a script and picks its own path, then reports where the path confused it even when nothing is technically broken. "The test passed but it tried three other things before finding the button" is a finding a passing test cannot produce.

What do I have to set up?

A staging URL, a scoped test account on it, and a task written as a goal. Payments, deletes and off-origin navigation are blocked, and runs stay on the origin you configure. If your staging has a captcha in front of signup, expect to turn it off there or hand the run an account that is already past it.

Can I point it at production?

The setup flow argues with you when the host does not look like staging, and you should listen to it. This clicks, types and submits on its own. Staging plus a throwaway account is the supported path.

What does a finding look like when it reaches my inbox?

It arrives as a feedback item with the category, the severity the run assigned, what it tried, where it got stuck, and a deep link into the recording at that moment. From there it behaves like any other feedback: it clusters with related reports and can go to an AI-authored pull request you review.

How often should I run one?

Nightly against staging on the two or three flows that matter, which for most teams means signup, the first real action, and whatever you shipped this week. Most nights come back clean. The value is the night one does not.

Turn that feedback into a pull request.

Free tier. No credit card. Two-minute install. The AI opens the PR, you merge it.

Get started free

Keep reading

Compare tools