Driving Product Confidence in 1 Day with AI-Assisted UsabilityTesting
Conducted unmoderated testing to evaluate tap accuracy in dense charts and validate design decisions against stakeholder feedback
Role
UX Designer
Timeline
1 day
Team
UX, Product, Architects
Tools used
Figma, Dscout, ChatGPT
Summary
During a design review of the fuel savings chart, stakeholders raised concerns about tap accuracy on tightly spaced bars and proposed solutions like adding extra steps or scrolling.
These changes would have increased interaction cost and complexity.
To validate the concern with real users, I ran an AI-assisted unmoderated usability test, creating screeners and tasks with ChatGPT and conducting the study on Dscout, all within a single day.
-
No usability issues were reported around tap accuracy
-
7/7 users successfully interacted with the chart without any mis-taps
-
Users completed tasks confidently and without friction
Outcome
-
Avoided unnecessary feature changes (extra steps, scrolling behavior)
-
Reduced decision-making time from days/weeks to a single day
-
Enabled a data-backed design decision, aligning stakeholders quickly
Impact
Challenge
-
What should the right screener look like to ensure relevant participants (mobile users, touch interaction)?
-
How should tasks be framed so users behave naturally, without bias toward tapping?
-
Would participants clearly understand the task in an unmoderated setup without guidance?
Key challenges included:
This was my first time evaluating tap accuracy in a dense interface.
Unlike typical usability tests, I needed to design a study that captured interaction precision (mis-taps, accuracy, confidence) without biasing users.
All of this had to be done quickly while maintaining a reliable and high-quality study.
My Approach
To validate the design quickly, I needed to set up an unmoderated usability test end-to-end within a day, including recruitment and task design.
1. Designing the Screener (with AI + Judgment)
I started by referencing my previous screener questions and used them as a base.
Then, I provided context and the testing goal to ChatGPT to generate a new screener.
However, the initial output was misaligned with my objective. It focused on discoverability instead of tap accuracy and interaction precision.
Instead of using it directly, I:
-
Identified the gap in interpretation
-
Refined my prompt to clearly emphasize tap accuracy in dense UI
-
Iterated with ChatGPT until the output aligned with my goal
This helped me quickly arrive at relevant screener and knockout questions tailored to the study.
2. Recruiting Participants
I launched the screener on Dscout and was able to quickly recruit the right participants, ensuring:
-
Mobile-first users
-
Comfortable with touch interactions
3. Crafting the Usability Mission
Next, I used ChatGPT to generate a usability test mission.
The initial response included multiple scenarios and task variations.
I curated the most relevant ones and refined them further through multiple iterations.
Key improvements I focused on:
-
Making the task natural and goal-driven (not instructing users to “tap”)
-
Ensuring clarity for unmoderated execution
-
Reducing ambiguity so users could complete tasks independently
I continuously iterated with ChatGPT, refining prompts and outputs until the mission was clear, unbiased, and aligned with the testing objective.
My prompt
Response I took
Response I rejected
Execution
I launched the study on Dscout as an unmoderated usability test, enabling quick turnaround and real user interaction without scheduling constraints.
-
7 participants completed the study successfully
-
All participants used mobile devices, ensuring realistic touch interaction
-
The study captured screen recordings, task responses, and verbal feedback (think-aloud)
-
This setup allowed me to observe:
-
How users interacted with tightly packed bars
-
Whether they experienced mis-taps or hesitation
Outcome
The usability test confirmed that tap accuracy was not an issue, even with tightly spaced bars.
-
7/7 participants successfully tapped the correct months without any mis-taps
-
All participants were able to complete the interaction quickly and without hesitation
-
Screen recordings clearly showed high tap precision, even for closely placed bars
However, some interesting behavioral patterns emerged:
-
Most participants evaluated the experience holistically, commenting on:
-
-
Year selection
-
Amount of data shown
-
Overall usability
-
-
Instead of focusing only on the tapping interaction
-
One participant did not follow the task sequence (September → December → November), which led to confusion and frustration
-
-
Despite this, their recording showed accurate and effortless tapping, reinforcing that tap precision was not the issue
-
Overall, the results provided strong evidence that:
-
The current design supports accurate and reliable interaction
-
The perceived risk of mis-taps raised by stakeholders was not observed in real user behavior
Recordings of participants
Learnings
Task clarity is critical in unmoderated testing
Even a small misunderstanding (like task order) can:
-
Break the flow
-
Create unnecessary frustration
Clear and simple instructions are essential, especially when no moderator is present.
Behavior > Self-reported feedback
Although one participant reported difficulty due to task confusion,
their actual interaction behavior (recordings) showed no issues with tapping.
This reinforced the importance of:
-
Observing real behavior
-
Not relying only on verbal or written feedback