What We Learned About Usability Lab and Guerilla Testing

UX Researchers’ community is living its own Barcelona vs. Real Madrid when you ask them: “What’s your team? User Lab or Guerrilla Test? Like football conversations, these methods generate controversy. Definitely, when you need to choose a test method, you will find some rivalries.
At Opentrends we know that even building a User Lab, we still do Guerrilla Testing with customers.

If you don’t know which method to choose for your next test, we show you some tricks to make the right decision in your next research challenges.

Martin Belam describes guerilla testing as the art of spotting lone individuals in cafés and public spaces, asking them about your product prototype, and documenting their reaction and user experience.

Simply put, guerilla testing is the act of going to a public place, approaching people who might be willing to test your product, and generating user feedback from them. It’s relatively low-cost and simple.

Keep in mind, however, that guerilla testing isn’t as easy as going to the nearest public place and conducting your test where people are willing. You have to be strategic about your choice of location. Make sure that the place you’re holding the test is a place your target audience frequently visits.

For example, if you want to test a new mobile app designed for retail chains, your best option is conducting the guerilla test in a series of retail outlets. You may opt for a test in just one location. But the more locations you test, the more insights you get and the more feedback you can work with.

A Test Lab is a room where you evaluate in detail every single user action with specific tasks and exercises. The Lab has an adjacent room called Observer Room to analyze through a mirror (and in real-time) these actions. Like in Police Stations, users are suspicious people to interrogate. Test Labs usually have technology and software to measure actions and find UX/UI mistakes in any product or digital service. At Opentrends we’ve experimented with eye-tracking tests and sensors to understand crazy things: user emotions. Thus, we discover those processes where users aren’t aware of what are they doing.

Usability Test Labs are a heritage from marketing research. For years, marketers have performed focus groups in rooms following observation techniques and using artifacts as the interrogation room we said before. 

The lab, as a controlled environment for performing user testing, is more artificial and the insights we get are less spontaneous than Guerrilla Test, where you can’t take up more than 30 minutes of their lives. But in a controlled environment like a Lab, you can do other Research activities and Design Thinking exercises.

Context is essential in User Testing. That’s why we put it at the top of the list only in case you don’t want to read the full article. If you choose a User Lab to do user testing, people will probably go there with a clear goal in their minds: “don’t be dumb”. They maybe will dress, talk to behave differently as they usually do. They will preconfigure themselves in a way that their actions or answers won’t be focused on the test itself.

Usability lab testing might not drive the organic results you’re looking for because the test subjects are in a controlled environment. They know they’re being observed and that their reactions are being recorded.

Guerilla testing, on the other hand, delivers more candid results. The user testing is conducted in a real environment, so you can be more confident that people are making genuine reactions and comments about your product.

Make the results even more insightful by conducting guerilla tests in places where the target audience is normally found. For example, if you’re testing a flight-related app, approach people in the airport line. Or if you’re working on a food delivery app, conduct a test around office lunch hours or typical dinner time.

Some UX experts might suggest that guerilla testing works with any product, including vague concepts that you have just scribbled on a piece of paper. As a result, some researchers merely ask participants to give their feedback and print-out designs and concepts. This isn’t the right way to do it, though.

The purpose of guerilla testing is to find out how people react and respond to an actual product, even if it’s just a prototype. You won’t be able to gather deep insights by asking participants to flip through papers.

It doesn’t matter whether you already have a finished product for guerilla testing. As long as you have a prototype tester — preferable one that resembles the actual product — the more valuable feedback you’ll receive.

The election seems to be easy but there is a key factor before taking the final decision: the “fantastic” legal and privacy terms. Getting permissions and acceptances for Guerrilla Usability Tests is not always easy.

If you can’t get permissions for your Guerrilla Text on the ground, you should go to a Lab and customize it to set the most realistic context.

This is not a perfect scenario, but if you want to create a simulated context, focus on disturbing things like sounds (available online) to simulate people talking, planes taking off, or ringing mobiles. Also “cheat” on users by creating a way they’d feel: pressure them by setting up time-specific tasks; invite them to do combined tasks to prevent focusing on one thing only… By reproducing environments, you will have the chance to find new needs. 

Both decisions could work but always have one thing in common that may bother users and get them defensive: the signature of permission for taking pictures or/and recording voice and screen. This situation could be uncomfortable for some participants. Instead of doing that, try to take notes and leave users to act on their own.
 

If the context is important, test preparation is one of the success (or failure) keys. Guess having an amazing context but fail during the preparation. Avoid big fails like this. Both tests, Guerilla or User Lab could be as difficult to prepare as reliable on their results.

For example, if you go out finding users you don't need to set up rooms, devices, software tools and other settings. But you’ll need to be a hunter and catch as many users as you need on the go. Otherwise, your Guerrilla Test will fail.

In a Test Lab, everything should be scheduled and ready for users and testers. Maybe it is artificial but it’s amazing if you don’t like improvisation. If everything is well prepared, you won’t lose time between tasks. User Test Labs are taken from Marketing research and offer safe and controlled experiments with some people observing and recording movements, facial expressions, or whatever users look at. This is something we lived at the 2018 Mobile World Congress in Barcelona, where we “eye-tracked” everything.

All these gadgets and software can provide amazing insights. Of course, it doesn’t mean Guerrilla Tests don’t give us serious results. But there are some Researchers’ community voices still believing that this isn’t scientific enough. Guerrilla Test detractors think you are invading users' spaces, privacy and conformability. They think labs are neutral places and users will find more issues because they will take tasks as a job and finally get a gift card or compensation.

Once having compared these two testing methods, maybe you can now decide better in your next user test. Both of them have their limitations and strengths. So, which is your team? Street Testers or Laboratory Rats?