May 13, 2010
Stakeholder alignment
I’m designing the UI for an application, and people here have different ideas about what this product should do. How am I supposed to design when I don’t know what I’m designing?
This is a common problem, especially when requirements haven’t been written down and agreed on by the project and business stakeholders. We recommend doing a stakeholder survey early in the project, to help align the stakeholders. The survey should ask each stakeholder to define:
- key business goals of the project (increase user adoption, reduce support costs, etc.)
- user experience metrics (efficiency, learnability, etc.)
- which personas or user types to target
- key use cases or user tasks
- the top features and functions
- relevant technologies that will affect the user interface
- branding guidelines
- etc.
Try to limit the number of stakeholders here to 6 people. Good luck with that.
Some of the stakeholders won’t have data or opinions about some of the questions, and that’s fine. They can skip those questions.
Analyze and summarize the data, and present the results to the stakeholder team. Get everybody in one meeting, and point out where there is agreement and disagreement. Where there’s disagreement (and there might be a lot), get the stakeholders to discuss, iron out the differences, and achieve alignment/Nirvana.
Often, stakeholders disagree because they rely too much on their personal opinions and guesses instead of on data from customers and end-users. You can make this whole process go smoother if you are armed with good data to inform the team’s decisions.
Filed under: Design, Process_and_Strategy | Permalink
January 11, 2010
Finding study participants
How can we find people to be in our studies?
Study participants can be found in many ways.
Start by defining the types of users you want for your study. A typical persona will not be enough here. For example, if you have a persona called “Amy”, a 27-year-old office manager who likes romantic comedies, etc., whom will you recruit for your study? People named Amy? 27-year-olds? Office managers? Romantic comedy fans?
Instead, for each user type (perhaps “Office Manager” in this case), you need to define the relevant set of demographic and psychographic criteria that represent the actual users you are targeting. This might include:
- an age range
- a minimum level of education
- a list of possible job titles (or, better yet, key job responsibilities)
- the gender breakdown (% female and % male)
- a minimum level of experience in the job
- a geography (for example, “U.S. only” or “evenly split between North America and Europe”)
- an income range
- specific attitudes, beliefs, motivations and interests that matter
- etc.
Next, determine how many participants from each user type should be included in your study. The number will be based on: whether you’re doing a survey, informal usability test, etc.; your budget; how confident you want to be that the study data are “correct”; and other factors.
Be sure to recruit in the proportions that match your target users—if 90% of your website’s targeted users are women, then 90% of your participants should be women. Also, the best data will come from people who are selected randomly from the population of people matching your criteria. So go beyond family, friends and co-workers.
The best ways to find participants will vary depending on the people you’re targeting, the type of study, and other factors. Sources for potential participants include:
- public places (the mall, the street, etc.)
- market research firms that recruit like this for a living
- referrals from friends, family and previous study participants
- trade shows and conferences (good for domain-expert users)
- your customer database
- newspaper or online ads you place
- religious or not-for-profit groups (who may help you recruit in exchange for a donation)
- social networking sites
- that User Group you so wisely created earlier for exactly this purpose.
When recruiting, have potential participants answer a questionnaire to see whether they are in fact good matches for your study.
Include surrogate users (people who know or claim to know the real users) only as a last resort, and interpret their data carefully. They might lead you down the wrong path.
Filed under: Usability, Process_and_Strategy | Permalink
October 28, 2009
Competitive Usability Testing
We’re redesigning our site, but there’s internal debate about whether the new design is any better than the current site. What’s the best way to determine which one is better?
Rather than continue to debate internally, you should get data from your end-users so you can make an informed decision. The guy down the hall has an opinion, and so do you, but your users’ responses matter a lot more.
Many techniques can help you figure out which design is better, but we’ll discuss two here. Both are types of “competitive benchmarking”, also known as “competitive usability testing” or “A/B testing”.
First, you could run a competitive test based on the think-aloud methodology. In this test, a user performs tasks with Version A of the design (or site, etc.) while thinking out loud. You make note of what the user liked and didn’t like, why they failed to complete certain tasks, where they got lost or confused, etc. Then the user does the same thing with Version B of the design while you observe. At the end, you ask the user which version they preferred and why. Do this with other representative users and look for patterns in the responses. The winning design usually emerges (“8 out of 10 users preferred Version A, and Version A was easier to use”), along with plenty of ideas on how to improve it.
Second, if the new design is already implemented, you could run a live test in which some percentage of visitors to your site get the new design while the rest see the usual design. You look at web analytics data and metrics you care about (for example, conversion) to decide whether the new design is “better”.
We recommend using both of these techniques together (the first during the design prototype phase and the second after implementation). But either technique is better than none.
Tips for competitive testing include: using the same tasks and the same metrics for both versions; asking users “why?” when possible; and alternating the presentation order across test participants if they see both versions (half will see Version A first, and the other half will see Version B first). It’s also good to have users compare designs at the same level of fidelity and interactivity—don’t test static wireframes against a live site and expect to get valid results. In this case, you could create static wireframe pages from your live site for the test.
Expero offers a training course on Competitive Usability Testing for people who want to learn the techniques in depth.
Filed under: Usability, Process_and_Strategy | Permalink
October 09, 2009
Hard data from think-aloud tests
We run think-aloud usability tests at my company, but some people here don’t pay attention to the results because we don’t get much “hard” data. Do we need to run different tests?
A well-executed think-aloud study can yield useful quantitative data (for example, rate of an occurrence or behavior) as well as qualitative data. We recommend that before you invest in more costly testing techniques, you try collecting and analyzing the quantitative data you may be missing from your think-aloud studies.
Start by counting how often important things happen during the study sessions (a user fails to complete a specific task, a user goes to the Help section, etc.). Ask users to rate the usefulness, ease of use, or appeal of particular features, as well as the overall user interface. All of a sudden, you have hard data to report: “Users rated the feature a 3.4 out of 5 on Usefulness”; “70% of the users tried to access Help when trying to create an account”.
A more advanced but very helpful step is to quantify what the users said during the study. This is “content analysis,” a technique from the communication and psychology fields. In content analysis, you categorize users’ comments (for example, as positive or negative), then you count. This yields even more hard data: “78% of users’ comments about creating an online account were negative, and only 22% were positive”; “Overall, 32% of users’ negative comments about the site focused on account creation, which suggests that this area needs a lot of improvement”.
Filed under: Usability | Permalink
June 05, 2008
Displaying the site map on every page
Someone on our website redesign team wants to include a list of 100+ links at the bottom of every page (to mirror Salesforce.com’s approach). Are a lot of companies including essentially their site map on every page?
Most sites don’t put their site maps on every page, and certainly not 100+ links. Doing so clutters each page and makes the content hard to scan and read. It also increases scrolling, which annoys some users.
Our advice instead is to perfect the information architecture, navigation and terminology in your site, and make the site map easy to access (if you are convinced you need a site map).
Why do some sites list a bunch of links on every page? For some companies, it appears to be search engine optimization (SEO). However, our experience shows that these lists often do not provide high value to users. If SEO is a big concern, then you might try it, but keep in mind that search engines are pretty sophisticated these days and don’t give much weight to extra “keywords” on the page.
Filed under: Design, Content | Permalink
Previous Questions |