Expand your prototyping knowledge

The Value of User Testing – Q&A with Validately founder

The Value of User Testing – Q&A with Validately founder

Discussing the importance of user testing and research in today’s web and software development companies. Justinmind talks with Validately founder, Steven Cohn.

In any IT project, user testing is essential to get feedback on the overall user experience from random users. The use of high-fidelity interactive wireframes and prototypes in user testing is changing the way IT projects are carried out as it is now possible to know how the app works for users before having even started the development process – drastically cutting rework.

A good prototyping tool must be integrated with a good user testing platform, that’s why at the beginning of 2016, you’ll have great news from Justinmind: full integration with Validately, one of the best rapid user testing platforms for prototypes and active features.

Steven Cohn, the founder of Validately, gives some practical advice on how to conduct proper user testing.

Why should a product team allocate its limited time to user testing?

Let me hit you with some industry stats:

  • 50% of the features in a typical product are unused by its customers. That’s right, half of what you are designing and building right now, will not be used by customers.
  • 30% of a typical development team’s efforts are classified as “re-work”. Meaning, re-coding an existing feature to better reflect customer requirements or to make it more usable.

Those are massive numbers. Real needle moving stats and it is wide spread. The truth is,

It is impossible for a designer or a product manager to sit in a room and guess how a customer wants to use a feature/product.

What are the different testing options? 

I’ll explain user testing using this grid:

When you are testing a new feature design, you want to learn 2 things:

  1. UI Usability
  2. Real Life Usability

UI Usability is easy to understand. Can someone go from page 1 to page 2 to page 3 to page 4 without getting confused?

Real Life Usability is more complicated. This test type answers whether a user will actually use the feature in their real life. As simple as it sounds, there is a lot that goes into that answer.

Remember this one thing when testing Real Life Usability:

What a customer says they will do, what a customer says they did and what a customer actually does are three different things.

Some example questions for when testing Real Life Usability are:

  • How will they use the feature in their daily life? How frequently? In which circumstances?
  • What do they view as alternatives to that feature? What do they like about that alternative? What do they hate about that alternative? What will make them switch from that alternative?

I can go on and on about the questions that help predict the success of a feature design. But the main point is this:

Until you put a prototype of a new feature in front of a customer, there is NO WAY to determine whether it meets the customer’s requirements to actually use it.

Simply gathering “requirements” in interviews is not good enough. Customers have to see something visually and try it out to tell you if they would use it.

With this basic understanding of the goals of user testing, the next part is how.

Moderated Testing

Moderated Testing means testing with a customer in real time. It can be remote or in person, but you are live with the customer asking them to try a prototype or already built feature and asking them Real Life Usability questions (noted above). You can also learn UI Usability with Moderated testing.

The benefit of Moderated testing is that you learn deep insights on how a customer will use a feature design. You can follow up and ask questions when a user mentions something and dig deep. However, this type of testing is a bigger time commitment.

This type of test is best done using a prototype because it’s easy to make changes. You’ll need to recruit testers that meet your persona & schedule a time to do a live session; run the tester through a prototype and ask them to complete specific tasks; ask Real Life Usability questions, noted above; take notes and record the session to share important aspects with stakeholders. It would be good to speak with 5 testers per persona. If you don’t have time or budget for 5, then 3 is better than zero.

Download Justinmind today and get testing!


Unmoderated Testing

Unmoderated testing is asynchronous. You set up a test, send it out to people and record their interactions, and often their voices, as they attempt to complete certain tasks. You are not live with them.

The major benefit is that it’s quick because it is asynchronous. You can get large volumes of responses while you are working on something else. However, you can’t ask follow up questions and dig deeper while a user is being tested. Thus you can really only learn UI Usability. Unmoderated is NOT helpful for learning Real Life Usability.

This test is best done with a clickable prototype, because you are certainly going to learn and want to make changes. You’ll need to recruit testers that meet your persona from either a panel or on your own. You give test takers a task to complete on the clickable prototype and measure how many can complete the task within a reasonable time period. Often you want the participant to share their thoughts out loud as they attempt to complete a task. You can ask qualitative questions afterward such as the SUS scoring approach.

Tasks should not be overly prescriptive and should not use specific words that are used in the UI. For example:

  • Bad task: Go to the Dashboard and “Add A New Story” to the queue. [since there are buttons labeled Dashboard and Add A New Story in the UI]
  • Good task: Add a story to your backlog

There are two approaches with unmoderated testing in terms of number of testers:

  • High volume click tests: typically 100 respondents and you just look at click patterns and qualitative responses
  • Talk-alouds: this is when the users voice their thoughts outloud. You only need 5 testers per persona to learn 85% of the usability issues.

How does Validately help with User Testing?

When we speak with Product Managers, UX Designers and User Researchers about why they don’t conduct more User Research/Testing, they usually say the same thing…I don’t have the time! When we delve into this this is, it’s really because the process around the actual test sessions takes longer than the testing sessions. Specifically the following pain points:

  • Finding representative customers (Personas) to take tests
  • Scheduling these tests
  • Running these tests without fooling around with recording software
  • Producing post-test reports in a shareable manner for stakeholders

Validately takes the pain out of user research/testing by automating all of these steps for our customers. We attack these pain points to make it really easy to talk to your Persona and test prototypes or live code. We offer simple and affordable solutions for both Moderated and Unmoderated testing.

How does prototyping help with user testing? 

Waiting to run user tests on coded features is an extremely inefficient and costly approach. Here’s the data (IBM):

      • 30% of a typical development team’s time is spent on re-work, meaning re-doing already coded features.
      • It is 200x more costly to change a coded feature than to change a prototype.

These stats highlight why it is a best practice is to run your user tests on prototypes. But there is another reason too. Once a company codes a feature, they become beholden to it. But testing on prototypes, allows design teams the freedom to experiment with different approaches and learn the best approach before being locked into one design.

What are the features that make a prototyping tool a great user testing tool too? 

High-fidelity prototypes, like the ones in Justinmind, are always best for running your user tests. The interactions are realistic and so the user can get a feel for how the new feature will actually work. It’s also always best to run your tests in the user’s natural environment and on their normal device, which is another reason why Justinmind prototypes are perfect for user testing.

Justinmind and Validately will be joining forces starting from January next year. Stay tuned for further updates and in the meantime sign up for a Validately demo and download Justinmind to start improving your workflow!


Daniela Castrataro

About the Author

Daniela is Content Editor at JustInMind

Comments are closed.