AI User Testing on Our Own Product: The Bugs It Found Were Not in the Code

Fewer people ship AI user testing than you would expect. Loop11 launched an AI browser agent in March last year, there are a handful of academic systems, and after that the category thins out fast.

We were thinking about building our own, and before I spent anything on container infrastructure I wanted a cheap version by hand: a claude -p loop driving a logged-out Playwright browser against our preview deploy. Two tasks, observe and decide and act until it finished or gave up.

I wanted to see if the premise held before I paid for it.

It didn’t, on the first attempt. Task A was “sign up and submit your first piece of feedback”, and the agent did it in ten steps with no wrong turns and no retries. Task B was “find the pricing and work out whether GitHub is supported”, and it finished in one step, by reading the answer off the landing page and never clicking anything.

Three findings across both tasks. One was real, one was pedantic, and one was flat wrong.

Where did the three findings come from?

1 in 3 is survivable for a first run. What killed it was where the three came from. All of them came out of the element list on step one, before the agent had clicked a single thing. Steps 2 through 10 returned confusion: null every single time. Nine steps of clicking produced nothing.

And the one real finding was a static link audit in disguise. MuseumLandingPage.tsx:1718 passed PRO_TRIAL_SIGNUP_PATH to every tier card inside the same .map, so the Free tier’s “Start free” button and the Team tier’s CTA both routed to /signup?redirectTo=/checkout?plan=pro. Five identical hrefs in the served HTML. A real bug, worth fixing, and grep -rn finds it in about a second. No browser required, no agent required.

The wrong one is worse. It flagged our deliberate anonymous “try it before you sign up” flow as a broken link, filed at high severity and high confidence. That’s exactly the failure mode our own customer research says is fatal: one first run full of false positives and a buyer’s “an agent is not my user” prior is confirmed for good.

What was I handing the agent?

The easy read is that the idea doesn’t work. I don’t think that’s what happened. I had handed the agent 6,000 characters of page.innerText plus every link URL on the page, and then asked it to behave like a person.

Nobody navigating a website can see the hrefs. Nobody reads the whole page including the footer before deciding where to click. Information scent, scroll depth, and visual hierarchy are the things you want an agent to struggle with, and I had deleted all three from its input. Task B turned into a reading comprehension exercise because the answer was sitting in the string I handed it.

So I wrote down a gate before touching anything, to stop myself moving the goalposts later. Fix the observation layer, re-run across three tasks, and continue only if at least two real findings come from interaction rather than step-one static analysis. If interaction still turned up nothing, kill it, or reframe it as the flow auditing it had already proved it was good at.

What changed across the three rounds

v1 scoped the text to the viewport, fed screenshots into the decide call and the analyst, hid the hrefs, removed the URL bar so it had to click, and forced it to triage console errors instead of ignoring them. That last bit worked immediately: the CORS error v0 had captured and silently dropped got correctly triaged as third-party noise. But el.innerText returns '' inside our app shell, so the deeper in-product task got unlabeled buttons and no text, and both of its high-severity findings were artifacts of that.

v2 added a textContent fallback, a force-click retry, and page-change detection. Still wrong. So instead of reading the findings a third time, I ran the observe function against the live page and printed what the agent was being shown.

19 of the 45 controls it could see were invisible to a human. Collapsed <details> cards, plus a md:hidden mobile duplicate of the nav. Asking an element about itself with getComputedStyle tells you the element thinks it is visible. It does not tell you whether anything is painted at that point on the screen. The agent had been clicking ghosts for two rounds and filing the failures as product bugs.

v3 hit-tests with elementFromPoint and asserts on the count. Zero invisible elements. The deep task that had been flailing completed in two clicks, and reproduced on a second run.

Six real findings from interaction, against the two the gate asked for.

The two bugs it found

Preview’s “Connect GitHub” button 404’d. It pointed at github.com/apps/local-feedbee-agent, which does not exist. The slug comes from a GitHub repo variable, PREVIEW_GITHUB_APP_SLUG, read at .github/workflows/test-and-deploy.yml:369, and it still held a value from before we renamed the app. Local .dev.vars had the correct one.

Here is the run that caught it, trimmed:

{
  "runId": "v3-B-1786165789019",
  "taskId": "B",
  "goal": "You have already signed up and you are logged in. Connect your
           GitHub repository so that feedback can be turned into pull requests.",
  "finish": "observation-sanity-fail",
  "steps": 3,
  "findings": [
    {
      "title": "\"Connect GitHub\" sends users to a GitHub App page that 404s,
                so the repo can never be connected",
      "severity": "high",
      "category": "dead-end",
      "stepIndex": 2,
      "basis": "interaction",
      "confidence": "high"
    }
  ]
}

basis: "interaction" is the field the whole spike was about. The finding exists because something got clicked.

Every file in the repository was correct. There was nothing to grep for, nothing for typecheck to catch, nothing a reviewer could have spotted in a diff, and no test that failed. The bug did not live in the repo. Clicking the button on a real deploy is the only thing that finds it. The customer version of this is a wrong env var in staging, a stale API key, a flag left off in one environment, and none of those are visible to anything that reads code.

Finding it was one thing. What convinced me was fixing it. The app id in the config, 1974130, resolves through GitHub’s API to local-usero-agent: we had renamed the app, and renaming does not change the id, so only the slug rotted. Two lines of config later, and I re-ran the same agent against preview. It signed up, clicked Connect GitHub, and landed on a live install page. The thing that found the bug confirmed the fix.

The “Start a new form” dialog’s close X does nothing. Two files, both fine:

// app/components/forms/SurveyTemplateGallery.tsx:39
<DialogHeader className='px-6 pt-6 pb-4 border-b border-border
                         sticky top-0 bg-background z-10'>

// app/components/ui/dialog.tsx
<DialogPrimitive.Close className='absolute right-4 top-4 rounded-sm
                                  opacity-70 ...'>

That SurveyTemplateGallery header was the only sticky z-10 DialogHeader in the codebase, and it paints over the close button, which had no z-index of its own. Hit-test the X and you get the header back.

Neither file is wrong on its own. They are only wrong in the same render. To find it you need the page drawn and a click that lands on the wrong thing. (It is medium severity, not high. Radix still closes the dialog on Esc and on the overlay, so people are annoyed rather than trapped. The agent overcalled it.)

Those two are the reason to bother with any of this. Bugs that live between two correct files, or outside the repo entirely. Static analysis is structurally unable to see them.

It found three more, all real and all smaller. Five footer links are mailto: while /privacy and /terms exist as live routes. /pricing has zero inbound links from the landing page, because the nav “Pricing” is an in-page anchor. And there is a two second window after submitting feedback where the toast says “Feedback added” and the dashboard still says “No feedback yet”.

Why I wouldn’t call this usability testing

It found bugs. It didn’t get confused. Across every run it never got lost, never hesitated, never gave up and tried a different path out of frustration. That puts this much closer to flow regression testing than to usability research, and I’d rather say so than dress it up. Flow regression testing is the label I’d put on it if I were buying.

The published research is not kind to anyone claiming more. Agreement between agent-identified and human-identified usability problems comes out worse than chance in the measurements I’ve seen. Agents read the DOM, not pixels, so contrast, clutter, visual hierarchy and “I didn’t see the button” are all invisible to them. They are also far more patient than any real person. Complementary coverage is a defensible claim. Substitution is not one, and buyers in this category have already been burned enough to smell it.

Coverage is also only as good as the task list you write. It sweeps nothing on its own.

Cost, since it’s the part nobody publishes: about $0.20 and 60 to 90 seconds per step. So a 25-step run is around $5 and half an hour of continuous browser time. That half hour is the infrastructure question we still have to answer. The whole spike, all four rounds, came to $7.14.

Where it sits now

A spike, and nothing more. We call it AI user tests when we talk about it internally. There’s no feature, no date, and the biggest unknown (where the browser runs, and what half an hour of residency per run costs us) is unresolved.

But the bugs were real, they’re fixed, and I’ve stopped thinking of the observation layer as plumbing. It manufactured false positives in two consecutive rounds and I nearly read them as evidence the idea was dead. If you try this yourself, print what your agent can see before you read a single one of its findings.

Frequently Asked Questions