< Projects

Creating a Usability Lab for $0

A usability session from my bootstrapped Usability Lab within Playground Global, testing a Wizard of Oz prototype of a smartphone in stealth development.

MY ROLE | Lead User Researcher:

  • Designed and built the usability lab

  • Designed, facilitated, and analyzed usability sessions

  • Created research recommendations & reports based on findings

  • Presented findings company-wide, including C-suite

  • Assisted the design team in implementing research findings

DURATION | Ongoing iterative testing every 2-3 weeks for 8+ months (my entire time at Essential Products)
IMPACT |
GUI usability & Industrial Design improvements to both products.
Voice Control usability improvements, improved sentiment scores.
Findings shared company-wide.
First-ever usability testing at Essential Products led to increased investment in User Research.
First usability report led to my conversion from contractor to FTE.

DIRECT STAKEHOLDERS | Design Director, Design team, Chief Product Officer, Head of Engineering

METHODOLOGIES | Usability Testing, Usability Lab, Sentiment Analysis, Wizard of Oz Prototyping

 

Background & problem

How do you conduct user research without a budget?

Essential Products, founded by Andy Rubin, was developing their second Smartphone (Gem) alongside a Smart Home Device (Hato).

The design team lacked any research presence, and hired me on contract as their first ever investment into User Research. I was tasked with improving the usability of 2 products in stealth development while also advocating & demonstrating the value of User Research to a CEO who had never hired a User Researcher before.

 
 

research questions

Smartphone (Gem) with a very unique form factor :

  • Are users able to complete basic tasks using Gem’s UI?

  • Can users discover and learn the voice assistant activation on Gem? Do they like the current activation method?

  • How do users interact with incoming notifications? Can they easily discover how to find quick settings and past notifications?

  • Can users easily find and switch applications?

  • How do users expect to interact with Gem (given its unique form factor and screen size)?

  • etc.

Smart Home (Hato):

  • Can users use our Smart Home device (Hato) to complete basic IoT tasks, such as dimming the lights or adjusting music volume?

  • How do users interact with Hato (voice/touch) at varying proximities?

  • Are Hato’s features, content, and functions both discoverable and learnable?

  • How do users react to, and interact with, Hato’s proactive suggestions?

  • How do users feel about Hato’s voice assistant? (sentiment analysis)

  • Do users understand when the voice assistant is listening?

  • etc.

 

methods

create a usability lab

Without a budget for User Research, I was tasked with assembling a usability lab with the technology available to me. Fortunately, we were in a high-tech incubator (Playground Global) that happened to have gadgets lying around. I was able to bootstrap a usability lab with:

  • Document camera for device interactions

  • Owl Meeting Camera for face capture & microphone

  • On-screen timer to measure time-on-task and mark observation times to make it easier to find relevant clips

  • Screen record on my Macbook to capture both camera inputs, device screen, and timer at the same time

We tested with internal participants (employees) due to zero budget.

Screenshot from a usability testing session with Gem (notice the unique form factor).

moderated usability sessions

Every 2-3 weeks, I would setup the usability lab and begin another round of testing. Some of the testing rounds included measurements around:

  • Time-on-task to compare task completion between design iterations

  • Likert rating scales to evaluate users’ perception of certain features and capabilities

  • Sentiment analysis for the voice assistant: I borrowed a “Voice Assistant Evaluation Rubric” from Microsoft, in which they provided participants with a set of ~24 adjectives to describe the assistant, and asked participants to choose the 3-5 adjectives that described the assistant’s personality

  • Wizard of Oz prototyping to evaluate the usability and desirability of certain features (e.g. how to activate the voice assistant) without investing in industrial design changes

Screenshot of a usability readout, communicating how information density negatively affected users’ ability to complete basic IoT tasks.

 

Wizard of Oz prototype highlight reel showing issues (and positives) with the usability of the activation for the voice assistant “Eva”. When I saw a participant place their finger on the sensor or activate a voice command, I would click the appropriate button on-screen. Most participant thought the device was fully functional. Prototype created by Scott Ysebert, UX Engineer.

 
 

key findings

home doesn’t feel like home (GEM)

The Gem home screen feels more like an app switcher than it does a typical smartphone “home.” A few design aspects contribute to this sentiment:

  • The lack of app entry points on screen (only 2.5 apps visible at a time)

  • Lack of customizability

  • Visual density

  • Persistent presence of notifications leads to a lack of privacy

  • Persistent settings

 

Screenshot of a presentation slide on the usability of Hato’s notification center

hato notifications are undiscoverable, but learnable

Users were unable to find the location of notifications due to:

  • The lack of indicator when a notification arrives

  • Icon is unrecognizable

  • Notification gesture can only be accessed from certain screens; if a user attempts it once and it doesn’t appear, they won’t try the gesture again

  • Feels more like a history of actions than a notification center

Once users are prompted with the correct gesture, they are easily able to access notifications in future tasks.

 

Screenshot of a highlight reel I created to showcase the usability issues of Gem’s text entry.


text input on gem is clunky and inefficient

Gem’s unique form factor posed a difficult challenge for designing keyboard text input. Users felt that text input on Gem was clunky and inefficient:

  • When typing, the left hand completely covers the sentence being typed.

    • There is no way to watch/proofread your text while you’re actively typing; you have to stop typing to see your text input.

  • Text suggestions are in the wrong place; users expect the text suggestions / auto-complete to be accessible by their left thumb since the left thumb is free; by not utilizing the free thumb, users feel their experience is inefficient.

 

These are just a few of many findings across multiple rounds of testing. Please contact me if you’d like to learn more.

 

impact

A few notable impacts:

  • Significant usability improvements to both devices; the design team would watch each session, review & discuss findings, quickly implement changes, then I’d share our findings and design decisions to the entire company. Rinse and repeat.

  • Changes to voice assistant’s voice, cadence, tone, and verbiage based on sentiment analysis led to improved sentiment scores with increased % of “positive” personality traits selected by participants.

  • After presenting my first round of usability findings, I was converted from a contract position to a FTE.

 

challenges & learnings

  • I learned quickly that user research is often an uphill battle. Not only to do your job, but to also demonstrate its value and justify its existence.

  • Working in such a fast-paced startup in the heart of the Silicon Valley was such an interesting learning experience, where I learned about all parts of product design — from industrial design to materials sourcing to image optimization to marketing and sales.

  • I learned to be scrappy, which is now an important part of my User Research Philosophy.

thank you…

Everyone at Essential was a joy to work with and learn from. Special thanks to Kumi Akiyoshi for hiring me in the first place and giving me the chance to demonstrate the value of user research!