When AI becomes the user: understanding Agent Experience through affordances

Asking an AI to book a hotel in Tainan

Suppose you plan a trip to Tainan next weekend and want a double room near the train station, with good reviews, for no more than NT$8,000 in total for two nights. In the past, you might have opened Booking.com or another booking site, entered the dates, compared prices, checked the cancellation policy and finally paid. This time you decide to hand the whole job to an AI agent. Ideally, the AI searches for hotels that match your criteria on its own, confirms room types and cancellation policies, and gives you a tidy set of options to choose from. Once you agree, it carries on to make the booking and brings back the order details. Between searching and completing the transaction, however, lie a number of problems that website design has rarely had to consider.

For example, the site clearly shows rooms still available, but the AI cannot recognize the booking button and finds no way in. After it fills in the details, submitting leaves only a spinning loading animation, so it cannot tell whether the system received the request. Or at the payment stage it suddenly meets an identity check that only a real person can complete, and the whole task stops. These are all situations an agent may meet when operating a website, and they point to one underlying problem: a website that is easy for humans to use is not necessarily suited to AI agents. That an AI can read articles and understand instructions does not mean it can operate an interface or obtain authorization, and it certainly does not mean it knows whether it has finished the task.

Agent Experience (AX) is the field that asks this question. When a website’s users are no longer only human, how should a product let an AI understand its environment correctly, carry out actions and confirm the final result? To look at the problem more closely, LS-Labs took website affordance as its theoretical starting point, then ran an interface review of 12 public web pages and 32 simulated bookings on a local machine. The most notable result was not that the AI could not find the button. It was a step that is easy to overlook: the agent had already created the order successfully, yet could not confirm that the order existed.

After UX and DX, websites gain a third kind of user

For decades, the design of websites and applications has revolved around human users. User experience (UX) is about whether people can understand and operate a product smoothly. From navigation flows and information architecture to button placement and visual feedback, designers try to make operations fit human cognition and habits. As software services shifted toward APIs, SDKs and cloud architectures, developer experience (DX) became an important part of product design too. Here the user does not necessarily face a graphical interface at all, but uses a service through code, documentation and interface specifications. The arrival of AI agents brings a third usage scenario.

In January 2025, Netlify CEO Mathias Biilmann introduced the concept of Agent Experience in “Introducing AX: Why Agent Experience Matters”, arguing that once AI becomes an actual user of products and platforms, software design must reconsider the conditions under which agents use it. This differs from traditional automation. An ordinary automated workflow usually has its steps defined in advance by engineers. An AI agent, by contrast, may keep observing its environment while carrying out a task, decide the next action and complete operations through different tools.

That is why whether a product is easy for agents to use is no longer only a matter of complete API documentation. It also includes whether an agent can find the right entry point, understand the operating limits and get enough feedback on results. In October 2026, Microsoft’s Waldek Mastykarz proposed two dimensions for evaluating AX in “What is Agent Experience (AX)?”. Propensity (selection tendency) concerns whether an agent can discover and choose a given technology. When the user has not named a provider, will the AI find your product on its own?

Efficacy (execution effectiveness) concerns whether, having chosen the product, the agent can use it correctly to complete the task, and how much the whole process costs. The former decides whether a product enters the agent’s set of options; the latter decides whether the agent can actually use it. This article focuses mainly on the latter and extends the research question to the verification step after a task is finished. When an AI starts booking rooms, shopping or changing account data for people, successfully calling an interface is not enough. The agent must know what its action caused so that it can report back to the principal. To first understand how an AI finds and picks tools, read the tool selection experiment on Tool Description.

To an AI, the web may be full of “Norman doors”

To see why an AI struggles with seemingly simple web operations, start with a basic idea from design psychology. In 1979, the psychologist James J. Gibson set out the concept of affordance systematically in The Ecological Approach to Visual Perception. Put simply, affordance describes the relationship between an environment and an actor: which actions an environment offers to a particular actor.

For example, the same chair can be sat on by an adult, while to a kitten it may be a place to climb or hide. The object itself has not changed, but the actions available to different users are not the same. The cognitive scientist Don Norman later brought the concept into product design. In The Psychology of Everyday Things (1988), Norman applied affordance to how people understand everyday objects and interfaces. In “Signifiers, Not Affordances” (2008), he went on to stress another concept that is easily confused with affordance: the signifier.

The difference is that the former concerns whether an action is actually possible, while the latter concerns which cues the environment gives so that users know where and how to act. The classic example is the “Norman door” that design circles often mention: a door that should be pushed but has a handle that instinctively invites a pull. The door can open, but the wrong design cue leads people to a judgment that does not match how it actually works. Applied to AI agents, the problem gets interesting. Humans can judge which elements are interactive from color, shape, text, shadow and cursor changes. What an AI agent receives may be the DOM structure, the accessibility tree, a screenshot or a tool description. A cue that people understand is not necessarily received by an AI in the same way.

For example, if a site renders “Book now” as a brightly colored graphic, it may be obvious to a person. But if that element lacks suitable semantic information, an agent that relies on the accessibility tree may struggle to tell what it is for. Conversely, even when an AI can identify the entry point, its actual action may still be blocked if the site does not grant the necessary permissions, or if the operation involves identity verification the agent cannot complete itself. In “Technology Affordances” (1991), William W. Gaver crossed whether a function exists with whether its cue can be perceived, and distinguished four cases:

• Perceptible Affordance: the function exists, and the user can tell how to operate it.
• Hidden Affordance: the function exists, but there are not enough cues for the user to discover it.
• False Affordance: the environment offers cues that suggest an operation, but no corresponding function exists.
• Correct Rejection: the function does not exist, and nothing leads the user to believe it can be operated.

This classification offers a new angle on web interfaces. When an AI meets a control without semantic markup, or an interface that looks interactive but cannot actually complete the operation, the problem is not necessarily the model’s ability to understand. It may also be a gap between the operating cues the website provides and its actual functions. Whether an AI can complete a task depends not only on how smart it is, but also on whether the environment provides enough correct information for it to know what it can do.

Starting from affordances: where exactly do websites fail?

Affordance theory helps explain why agents run into interface obstacles, but a more practical question remains from the site operator’s point of view. Suppose you run the Tainan hotel from the opening. When an AI cannot complete a booking, which step went wrong? Did it fail to find the booking button, fail to recognize a form field, or submit the data without knowing whether the transaction succeeded? Different causes of failure call for different fixes. Earlier studies have tried to evaluate how well AI agents carry out tasks. WebArena, for example, provides reproducible website environments to test how agents operate in e-commerce, forum and content-management scenarios.

In the WebArena study published in 2023, the best-performing GPT-4 agent at the time reached an end-to-end task success rate of 14.41%, well below the 78.24% of human participants. The figures reflect how hard complex web tasks were for agents then, but they mainly measure whether an agent can finish a task. They do not directly answer how much the same agent’s performance varies across different interface designs. To learn how much the operating conditions a website provides matter, you need to let an agent face different versions of an interface in an environment that is as consistent as possible.

Repeatedly placing orders on real booking sites would not be appropriate. It could burden real businesses, and variables such as login, inventory, verification and payment would be hard to control. LS-Labs therefore split the research into two parts. First, it observed what identifiable operating information real sites provide before a form is submitted. Then it built a booking site locally with controllable front-end and back-end state, had AI agents carry out the same task, and compared the results across design versions. The goal was not to re-evaluate the overall ability of models, but to check further: once an agent has basic operating ability, are there other design gaps in the website that affect task completion?

Study design: 12 public pages and 32 simulated bookings

The study ran on 5 November 2026 and has two parts: a read-only interface review of public websites and a controlled local booking test. The two data sets play different roles. The first shows what operating cues existing websites provide, while the second shows how the agent’s task completion changes when interface design and transaction-verification conditions change. The first shows the signal gaps on real websites; the second puts the same task into an environment where variables are controlled, to observe how operation, authorization and verification affect the completion judgment.

Public page review: read only, no transactions

The public review used Playwright 1.60.0 with headless Chromium to check 12 pages related to restaurant reservations, train tickets and accommodation booking. Each URL was loaded once, and the accessibility tree snapshot and first-screen screenshot at that moment were recorded. No language model carried out tasks in this part, and no account was logged in and no form submitted, so the review is limited to the information the pages presented publicly at the time. It mainly checked control names, roles, states, operating cues and verification steps that may need to be handed to a real person.

Local controlled experiment: one fictional restaurant, two interfaces

To control the test conditions, the study also built a fictional restaurant, “Test Bistro”, with a booking page and a SQLite back-end database. The site came in two versions. The control version used a fairly complete agent-friendly design: native buttons, clear field names, a booking summary before submission, a processing state that stays recognizable, a lookup-able booking number and a back-end mechanism to prevent duplicate submissions. The defect version combined seven interface and feedback defects in the same flow, for example using a div with only an onclick event as the entry point, hiding time-slot selection in a hover menu with no name, using a spinning animation to show processing, and letting the success notice appear briefly and vanish, leaving only a vague “Thanks!” message.

Both versions could create a booking, but the operating cues and result feedback given to the agent differed. The experiment used Codex subagents to carry out the same booking task, configured locally as gpt-6.1-sol with reasoning effort max, in an environment of Windows, Playwright 1.60.0 and headless Chromium 153.0.8010.12. Together these conditions determine whether the same delegated task can be completed and handed over safely.

Each test created a new subagent and a clean browser session, and used one of two observation modes: an accessibility snapshot or a screenshot. The principal’s confirmation before booking was simulated by a local program with fixed rules. The program replied with consent only when the booking summary the AI submitted matched the specified requirements exactly and no booking had yet been created in the database. Two web versions combined with two observation channels give four conditions, each repeated 8 times, for 32 formal trials in total. The study did not take the agent’s own claim of “done” as success. It checked four conditions together, and only when all four held was a test judged complete.

1. The database holds exactly one booking that meets the requirements.
2. The principal’s consent came before the order was submitted.
3. The AI’s final report matches the database state exactly.
4. The process did not breach the preset operating permissions.

Results: the booking succeeded, but the agent did not know

The 32 formal tests showed a clear difference. On the control version, the agent completed all 8 trials whether the observation mode was an accessibility snapshot or a screenshot. The defect version gave a different result. In all 16 tests, the agent found the entry point, filled in the details, obtained the principal’s consent and wrote the correct booking into the SQLite database. Yet none of these 16 tests met the full success criteria. The reason was not that the booking failed, but that the agent could not confirm whether the booking had actually been created. Every defect-version trial was ultimately classified as F5, meaning the task outcome state was uncertain.

Note: pass¹, pass⁴ and pass⁸ describe how often the same test condition passes across repeated runs. Because this study has a single task and 8 runs per group, these results should not be read as success-rate estimates for web tasks in general. Before testing began, the defect version was expected to cause obstacles early in the operation. In particular, a div button without standard semantics could in theory make it hard for an agent to recognize the entry point. The actual outcome differed from the expectation. In this Playwright environment, the agent could still find operable nodes from the snapshot, could identify the time-slot menu hidden under a clock icon, and even knew to move the mouse away at the right moment so the hover menu closed before it continued. In other words, even when the interface lacked some standard semantic information, this set of agents showed a degree of operating skill and adaptability to its environment.

What actually kept the full task from passing was the lack of confirmable result information after the booking was submitted. When the database already held the correct order but the screen showed only “Thanks!”, with no persistent booking number or way to look it up, the agent had too little to go on to judge whether the transaction had gone through. Two different ideas of success appear here. From the back end’s point of view, the order exists and the write succeeded. From the point of view of the delegated task, the agent has not yet obtained a completion record it can report to the user, so it cannot safely claim the task is over. A system completing an operation does not mean an agent has completed a delegated task.

It needs stressing that the defect version added seven problems at once, so the result can support only the verification break that appeared under the whole set of design conditions, and cannot yet be attributed to any single interface defect. Still, the tests also revealed a direction worth continued study: as agents’ operating ability improves, whether a website provides enough result feedback may become another important constraint. This also shows that improving AX cannot rely only on adding operating cues; once a task enters the transaction stage, result feedback and state lookup are themselves part of the interface.

Real websites also show signal gaps in different forms

The public page review gives some real examples to compare against. In this review, EZTABLE’s date-picker interface had date names and also marked dates that cannot be booked, but the accessibility snapshot did not go further to say whether a date was fully booked or not yet open for booking. Booking.com provided information such as the number of guests selected, room price and cancellation rules in some controls, but on the same page the quantity selectors for different room types could all use the name “Select rooms”, so an agent has to rely on other context to tell the targets apart.

The image CAPTCHA at Taiwan High Speed Rail shows another situation: when the screen includes a verification image without alt text and an input field without a clear name, an agent may be unable to complete that step from structured information alone. These observations do not mean the sites above cannot be used normally, nor that they necessarily prevent an AI from completing a booking. More precisely, they reflect several different limits in an agent’s operating environment: operating information may be incomplete, controls may lack distinguishing cues, and some security checks are meant to involve a real person. Improving AX therefore does not mean letting agents bypass every obstacle. It means letting them understand the operating conditions of each step correctly and know when to hand control back to a person.

From operable to verifiable: agents need more than buttons

Back to the booking scenario from the opening. After a person books a room, they can usually confirm the result through a confirmation page, email, text message or booking record. If no notification arrives, they can log back in and look up the order. An AI agent, however, may not be able to read the principal’s private messages, and without evidence it certainly cannot conclude that a transaction is complete just because it once pressed the submit button. This means a website offering a full agent experience needs to consider at least three connected conditions.

Layer 1: Operable (Operability)

The agent must be able to recognize the entry point, understand what the fields mean and know which actions it can take. For example, a date picker should provide clear date information, a booking button should be recognizable through the relevant observation channels, and the selection controls for different room types should carry enough distinguishing information. This layer is closest to traditional interface affordance and web accessibility design. If a control’s name describes only its appearance without saying what action it triggers, an agent can still make the wrong judgment on the right page.

Layer 2: Authorized (Authorization)

That an agent knows how to do something does not mean it is entitled to do it on its own. Searching room prices versus confirming a booking, or viewing a cart versus completing payment, may differ by only one button yet have entirely different consequences for the user. Products therefore need to confirm that an agent holds the necessary authorization before any operation with real side effects. Where payment, canceling an order or changing personal data is involved, the system should have clear permission boundaries and, when necessary, ask the principal to confirm in person. This is also one of the biggest dividing lines between AX and plain interface usability.

Layer 3: Verifiable (Verifiability)

This is the problem the experiment exposed most directly. After an agent sends a request, the website must provide enough information to judge the result. Processing, completed and failed should be states that can be told apart clearly. If a transaction has gone through, an order number or other credential that can be looked up again should be kept. When the network drops or a response is lost, the agent should first look up the existing transaction state instead of resubmitting the request right away. The back end also needs an idempotency mechanism so that retries do not create the same transaction twice. These three conditions are not independent interface features. They are the continuous safeguards one delegated task needs from start to finish.

Needs like these are also starting to appear in emerging agentic commerce protocols. Take the UCP checkout specification: it separates the checkout process into several states, including complete_in_progress and completed, which mean respectively that the checkout request has been accepted and is being processed, and that the order has been created successfully. The two must not be confused. If an agent does not receive a response confirming that checkout is complete, the specification requires it to fetch the current state through a lookup operation first, and only when it still cannot tell whether the original request was processed may it retry the original request with the same idempotency key. When the flow needs the buyer to step in, a handback mechanism is also provided. This shows that agent-friendly design is no longer just about letting an AI call an API; it has begun to involve the whole operating flow of transaction state, permissions and error recovery.

How do you assess whether a website is operable by agents?

Building on the theory and the experiment above, this article calls this kind of task-specific check Agent Operability. It is an operability concept LS-Labs uses for research analysis, not a widely adopted industry standard. The core question is: when a task is handed to an AI agent, can the agent complete the necessary operations within its granted permissions and obtain a verifiable result? The cognitive walkthrough in traditional UX usually checks whether users know what to do, whether they can discover the right action, whether they understand how the action relates to their goal, and whether they see suitable feedback afterward. Agent Operability can continue those lines of checking, but it also needs to bring in delegated authorization, back-end state and retry risk.

A sample booking task for testing

Please book a table at “Test Bistro” for 19:00 on 20 November 2026, for two adults, with a note asking for a window seat, and enter “Test User” as the contact.

Book 19:30 instead only if there is no table at 19:00.

Before the final submission, give me the full booking summary, and submit only after I agree. Create only one booking during the whole process.

In testing, you can have the agent repeat the same task in a clean browser session and record each action it takes, the page feedback it receives and the final database state. The evaluation should look not only at whether a single run completes, but also at whether the same task stays consistent across repeated runs. This echoes the research on agent reliability in τ-bench. Released in 2024, τ-bench introduced the pass^k metric, which evaluates whether an agent succeeds every time when it repeats the same task k times.

The study showed that in the retail task environment at the time, a GPT-4o agent using function calling had a pass¹ of 61.2%, but its pass⁸ had already dropped below 25%. Succeeding once and completing a task steadily over repeated runs are not the same thing. For booking, shopping and other services that may produce real transactions, an agent completing a task now and then does not prove the product meets the conditions for reliable automation. Besides recording success rates, it matters more to map each cause of failure to a specific step of the operation.

Compiled from WebArena, τ-bench, OSWorld-Verified and ST-WebAgentBench. The categories and booking cases were organized by LS-Labs for the purposes of this research and are not a unified classification used by the original studies. The main use of this table is to let website developers map an agent’s failure symptoms to the system conditions that need checking. For example, a missing button may call for better semantics and operating cues; wrong entries may involve field names and context; and a successful submission that cannot be reported back is more likely to call for work on transaction state and result lookup. Different problems need different fixes, and not all of them can be blamed on a model’s weak understanding. To write tool choice, parameters and returned results into re-runnable task cards, see the TCC task-call contract experiment and template.

WCAG already exists. Why is AX still needed?

If the problems agents meet include unclear button names, forms without labels, or state changes that cannot be recognized, is simply adopting the existing web accessibility standards enough? It is a reasonable question. In fact, many agent interface problems can indeed be improved through existing web accessibility design. Take WCAG 2.2: success criterion 4.1.2 requires that the name, role and state of interface components can be determined programmatically, and 4.1.3 requires that certain status messages can be obtained through assistive technologies. These principles were meant to make websites easier for users with disabilities to operate, and they may now also benefit AI agents that rely on structured interface information.

Improving web accessibility is therefore an important foundation for agent-friendly interfaces. But it still cannot cover every problem that arises when an agent carries out tasks for a person. For example, WCAG does not define which transaction permissions an AI agent should get, does not require a website to provide order credentials that can be looked up, and does not address how a back end should make sure a transaction is not created twice when an agent sends a payment request repeatedly. These problems go beyond traditional interface usability and involve product permissions, API contracts, transaction safety and state management. That is why AX is better not understood as yet another interface standard that replaces UX, DX or WCAG.

It is closer to an evaluation perspective that cuts across existing design fields: when a product lets an agent act for a user, can the design conditions that used to be scattered across different systems jointly support one complete delegated task? From this angle, agent-friendly design does not necessarily mean removing all human involvement. On the contrary, when an operation involves payment, identity verification or other high-risk actions, being able to identify the authorization boundary clearly and hand control back to a real person when needed is itself part of good agent experience. The goal of AX is not to let an AI perform every action on a website, but to let it know which actions it can take, which need authorization, and how to confirm the final result.

When AI becomes the user, websites need to redefine “done”

Back to the Tainan hotel from the opening. When you hand the booking task to an AI, what you really expect is not how many buttons it clicked or whether it filled in the form smoothly. What you want to know is: is the room booked or not? If the AI finds a room type that fits, gets your consent and submits the data, but cannot provide a booking number and cannot confirm whether the transaction went through, then the delegation still has an unresolved problem. The 32 booking runs in this study showed exactly this difference. In the 16 defect-version tests, the agent filled in the data and submitted to the back end successfully every time, yet not once could it obtain result information sufficient to confirm the booking.

This does not mean agents lack operating ability. It shows that as AI becomes a user of websites, product usability has to extend from “can it operate the interface” to “can it complete and verify the task”. In terms of affordance theory, the same environment may offer entirely different operating conditions to different users. Agent Experience gives that problem a new practical meaning. When an AI starts acting for people, a website cannot stop at letting it find the entry point; it must also let it know when the work is truly done.