I’ve been reading some companies’ documentations of their DIY user testing experiences in order to see what I can learn.
This article is documentation of a user testing experience based off of the book Rocket Surgery Made Easy by Steve Krug. The user-testing framework emphasized in this book is that of lightweight, budget usability testing. They test few users, but have high engagement of the actual programming team. The programming team themselves are observers and are directly involved in the analysis of the results of the test.
What I learned from this resource:
- Read Rocket Surgery Made Easy by Steve Krug
- Recruit participants from your social circle who you know fit the description of your intended demographic
- Offer compensation to participants for their time
- Present tasks to participants in the form of scenarios
- Provide the equipment; external keyboards/mouses can help establish a more familiar environment for participants uncomfortable with Mac/PC
- Have observers in a separate room from the test so that they can speak freely without distracting the participant
- Set up a webcam/microphone/speaker for external observations; consider Silverback for screen recording
- Collaborate on observations using Google Drive
- Arrive early and set up the testing environment well before the tests should begin
- Time the tasks to allow for a debriefing before the test and a debriefing after the test
- Understand how to debrief the observers: how to cleanup observations, categorize issues, and prioritize solutions.
- Color-coding observations could include problems, positive remarks, future development ideas
- Categorizing issues could include fix today, submit bug report, or bench for further discussion before proceeding
This isn’t the only company that I’ve seen using what’s called Lean UX. Meetup is also using this testing framework. On Slideshare, they go into detail about their journey to their current usability testing framework. They started with a single usabiilty researcher conveying findings to a team, then tried outsourcing their usabiilty testing, and finally settled on Lean UX based on Steve Krug’s book Don’t Make Me Think.
Some of the shared findings between the experiences of Meetup and Deveo include:
- Observers should be in a different room
- The programmers should be the observers
- Follow user tests with immediate discussion by the team
- Did their own recruiting within their community
- Provide compensation for your participants
There were also some differences:
- Morae (PC) vs. Silverback (Mac) for screen recording
The Meetup slideshare also gets into the idea of Coffeeshop Testing, where you take your screen recording software to a public space and offer to buy customers a coffee in return for 10 minutes of testing.
I particularly like Meetups’ summary slides at the end of the presentation listing takeaways, “Dos” and “Don’ts”, and other recommendations. Some of my favorites:
- Substitute frequency for precision
- If you thank participants, they may turn into evangelists for your product
- Use a screening process on potential participants
- Don’t be afraid to turn down “weirdos”
- Create a “Testing Day Checklist”
I know I’ll return to this slideshow for ideas when putting together my actual test documentation.
Admittedly, the enthusiasm these companies have for Lean UX is contagious. I thought I’d see the author in action and watch his demo.
Here’s what I learned.
There are five parts to a usability interview:
Create a script to cover the following so you don’t forget any of it:
- You’re testing the site, not the participant
- The participant needs to think out loud
- You can’t hurt our feelings — be honest and thorough
- Ask questions if you have to, but remember that we are trying to see how you would interact with this site if there wasn’t someone standing by to help you; so we may defer answers until the end
Have the participant sign a release form that gives permission to record the screen and the conversation; that acknowledges that there may be others observing the usability test in another room; and reassures that the recordings will be for internal use only.
Does the participant have any questions before we begin?
Get the participant talking with some initial questions about her background. What does she do all day?
Next, ask some questions to establish the participant’s familiarity with the product and general tech savvy. Ask any questions that will help you contextualize the user and compare her to your target demographic.
For example, how much time does she spend on the internet each week? What does he primarily use the internet for? What sites does she use the most? What are his favorite websites?
Remind the participant to talk out loud during her exploration of the starting page for the usability test. This is an opportunity to hear about the first impression of the site design and how well the purpose of the site is conveyed. Ask the user to describe what she thinks she will be able to do on the site.
- Whose site is it?
- First impressions?
- What is the site for?
- What are some things you can expect to do on the site?
Next, ask the user to do some tasks. Present the tasks as written scenarios. Read the scenario aloud, or ask the participant to read the task aloud. Then the participant should attempt to complete the task based on the scenario while thinking aloud through his exploration of the site.
When the participant completes the task, thank the participant and proceed to the next task. Save questions and answers for the very end.
At the end, thank the participant for their feedback. Now is the time to ask if they have any questions.
Once you’ve identified the problems, you can fix them. Krug reminds us that it’s not necessary to design a major change for every usability problem. Fixes can be as simple as moving some text from one place to another, or adding tabbed navigation to a table.
The demo was fast — only 20 minutes — and uncovered some easy-to-fix problems. It was good to see some of the ideals put into practice and to see how casual the encounter could be. But I appreciate Deveo and Meetup’s write-ups that go into more detail on how to set up the test and deal with the problems identified during the testing.
Bringing it back to Commotion
In summary, here’s the documentation I’d need to prepare for Commotion:
- A welcome script
- A release/consent form
- Demographic screener questions
- Moderator prompts
- Test Day Checklist
- User Testing scenarios that reflect our goals
- Observation spreadsheets
It also suggests that we should be doing usability testing on “whatever is ready when the tests are scheduled” as opposed to waiting for particular checkpoints along a development process.
The following links contain more information on Lean UX and experiences using it:
- Deveo – Reflections on Lean UX
- Made By Many – Selected Slideshare Presentations on Lean UX
- UX for Lean StartUps London
Testing Goal Script:
We believe that people like _(customer type)_ have a need for/problems doing _(need/action/behavior)_. We will know we have succeeded when _(quantitative/measurable outcome)_, or _(qualitative/observable outcome)_, which will contribute to _(Key Performance Indicator)_.
- Getting out of the Deliverables Business
- UX for Lean StartUps London