You might have noticed, if you don’t have much more important things to be doing with your time, that the whole ‘is paper prototyping a valid use of time’ thing kicked off on #DesignTwitter recently – sparked as far as I can tell by the below tweet which shows someone using paper prototypes to teach kids about design. Innocuous enough, you’d assume.
However, there were some (now deleted as far as I can tell) comments from one account in particular arguing that low fidelity prototyping approaches (such as paper prototyping shown in the image) are a waste of time, becuase a higher fidelity prototype would gather more accurate usability results, citing aesthetic-usability effect as their part of their reason.
I don’t normally dive in on twitter storms (especially over the weekend when I’m with the kids…) but the comments got me thinking. I find how and why we prototype fascinating, so wanted to collate a bit more than a snarky twitter response to lay out some of my thoughts, which are based on both my own experience and academic research done by other people.
For now – lets leave aside that the fact that the comments are confusing ‘what should professional designers do to make the best products’ with ‘how can designers communicate process in an engaging way to school children’ and just pick up the question of whether its worth using paper prototyping and other similar lo-fidelity approaches in general.
Prototypes aren’t just for testing
The argument I saw levelled against paper prototyping was that it would get less accurate usability testing results.
Don’t forget that testing with users is only one of many reasons you might create a prototype.
Prototypes are a fundamental part of any design process and the specifics of how any what you prototype should be dictated by what you want to use the prototype for. Prototypes are learning tools, that learning can take the form of usability testing with representative users, or learning what is or isn’t technically feasible with whoever’s building the design. Prototypes are also communication tools, they are ways of building consensus across teams each with a different skillset, understanding of the solution and different priorities. A quote I think that sums this up well is commonly attributed to the Kelley brothers:
“If a picture is worth a thousand words, a prototype is worth a thousand meetings”
Tom and David Kelley
4 different kinds of prototyping – the Erichsen model
I’m not the only one who sees it this way. One of my favourite discoveries through researching prototyping in academic literature is this model proposed by a team of researchers looking at how prototyping is used within product manufacturing.
What they found was that effective design teams used prototypes in different ways at different parts of the process. They classified them as reflective, or affirmative tools, which are used internally or with external users. These are all valid parts of a design process (see model below – illustration mine)
You can read the whole paper on core.ac.uk, if you get the chance (and don’t find yourself pulling your hair out at the structure of academic literature) I’d recommend reading it all. Its a nice summary of how prototypes are so much more than a testing tool, but in fact enable teams to externalise, communicate and socialise design ideas, all with the purpose of achieving effective design outcomes. The fact that its focused on the automotive industry means that I think the details of how each of the 4 categories are described could do with tweaking to better apply to e.g. digital product development, but the model still stands. Maybe one day I’ll get around to writing up my thinking on how it should change…
Prototypes for communication are equally valid prototypes
This is all to say that even if a prototyping method might not get you the most accurate usability testing results possible, there is still potential value in it, if used in the right way. Prototypes are as much about internally reflecting and communicating effectively as they are for testing with users.
For us as designers, it may be that if we took the time to build paper prototypes collaboratively with stakeholders/users etc before launching straight into polished Figma files we might actually design the right thing, then can usability test the hi-fi thing to design to right.
Or maybe if we built a paper prototype and passed it around the table at that meeting, we could have surfaced everyone’s feedback on the concept as opposed to whether the buttons were the right colour, got alignment on direction and then ultimately ship the thing.
But what about the aesthetic-usability effect
Well I’m glad you asked, I wouldn’t be so hasty to use this as evidence that only hi-fidelity prototypes are worthwhile, even if we just consider usability testing.
The aesthetic-usability effect refers to users’ tendency to perceive attractive products as more usable. People tend to believe that things that look better will work better — even if they aren’t actually more effective or efficient.
In effect, the nicer your prototype looks, the more likely users are to overlook the usability issues with it. Which is the primary purpose of running usability testing, to identify usability issues. The NNG site adds more context, from the perspective of how this might play out in a live usability session:
It’s a familiar frustration to usability-test moderators: You watch a user struggle through a suboptimal UI, encountering many errors and obstacles. Then, when you ask the user to comment on her experience, all she can talk about is the site’s great color scheme.
So, by presenting a higher fidelity prototype you’re potentially increasing the risk that the user will comment primarily on superficial issues, glossing over fundamental usability issues. Which seems like a bad outcome from a usability test. I’ve seen this play out in testing over my experience too. But…lets look at the evidence. Some previous colleagues explored this very thing, and its a topic I looked into as part of my research too.
The impact of fidelity on usability testing feedback
The two main studies I’m aware of in this space are:
and
Both do a comparison study of prototypes of the same design at three fidelity levels to assess what kind of usability feedback they gather. They’re both from the early days of interactive devices (2000s) and I’d be interested to know if anyone’s published on it more recently than that, with all the new tools currently available.
However the summary findings for each paper respectively are:
- All prototypes achieved similar results for the performance test. There was in fact little difference in performance across the prototypes with different fidelity levels. This in itself is an important result showing that in the early stages of the design process, the fidelity level might not have a significant impact.
- In this research, we figured out that major usability issues such as unclear meanings of labels, icon/symbol/graphical representation issues, locating appropriate interface elements, mental model mismatch, and appearance/look of the product, were identified by all the three types of prototypes, namely, the fully-functional prototype, the computer-based low-fidelity prototype, and the paper-based low-fidelity prototype. However, some of the issues like physical handling and operation, comments on the concept itself, comparison with other similar products, and performance-related issues, were identified by either the fully-functional one or the computer-based one, which are both real-time interactive.
Reading the full conclusions of both is interesting, there are of course limitations on the role of paper prototypes within the development of ’embedded products’ in these cases. I’d be interested to see how much that has carried over into our world of ubiquitous touch screens.
It would appear that in fact, low fidelity prototypes are pretty effective at gathering accurate usability results. The paper prototyping methods had limitations when it came to gathering feedback on the physical device aspects, as you might expect.
So are paper prototypes a waste of time?
I’d suggest not. Of course, they’re not always the right approach. If you want to know how you should prototype at the point of the design process you’re at, I’d suggest asking yourself some questions like these:
- What do I need to learn?
- What skills do I/we have
- What access to tools do I/we have
- Who is the audience for the prototype?
- What fidelity is most beneficial to enable me to learn what I need?
Have you got any thoughts to add? Let me know on twitter.
alistairuff 
Leave a comment