6 Metrics for Managing UI Design
As part of a recent management summit at my company, we were asked to fill out an RMPT matrix for our departments (I head up Product Design). An RMPT matrix consists of (R)esponsibilities, (M)etrics, (P)rocesses, and (T)ools. I have been intending to develop better metrics for both measuring and guiding our design efforts, and this exercise served as a catalyst to get me started. Bear in mind that metrics help you focus your efforts and measure your progress, but you are also held accountable to them.
For (R)esponsibilities I specified the following:
1) Improve our products & innovate
2) Provide the UI design for new features/functions/products
3) Approve any UI design work done outside of Product Design
4) Validate our UI designs and explore user needs through user testing
Given those responsibilities (and that’s important because your metrics are linked to them) I then came up with the following metrics and met with my team who helped to refine them:
For a given period (e.g. a business quarter):
1) Number of layouts delivered
2) Number of interactive prototypes created
3) Percentage of product design requests completed by commit date
4) Number of users tested
5) Number of product improvements made
6) Number of product insights documented
Discussion:
1) We struggled with what to call this and how to word it specifically. We started with “Number of designs delivered”, moved to “Number of screens delivered” and settled on “Number of layouts delivered.” We chose layouts because designs seemed a little too broad and didn’t really give any indication of what we would actually be delivering.
2) We prototype when feasible and want to both push this effort as well as measure it. Again we struggled with the wording (i.e. “prototypes” vs. “designs”, etc.). We wanted to make it clear that we are measuring the number of interactive, non-static prototypes.
3) We recently implemented a Product Design Request System where we track the growing volume of design requests and deliverables to development and product management. We are holding ourselves accountable to deliver on these requests by our committed due date.
4) We test designs on users as much as possible and want to both measure this and push ourselves to do this as much as possible.
5) Here’s where it gets a little interesting. This one is admittedly qualitative and may need further refinement. With that said, it is easy for us to remain reactive and work day to day by responding to requests from product management and development without ever taking the initiative to improve our products by combining our research with our unique expertise in usability and design. This metric is meant to drive us to be proactive and take initiative to improve our products and to track those improvements over time to further illustrate our value. To make this metric more objective I plan to tie in efficiency measurements and satisfaction scores (SUS).
6) With our considerable user “face time” we are in a unique position to collect “insights” into product shortcomings, needs, and opportunities. We often take copious notes when doing user studies only to leave the notes sleeping silently in our Moleskines. We meet regularly with SME’s (subject matter experts), developers, users, and various other stakeholders. This metric is intended to remind us to document, publish, and take action on the “insights” that we collect.
My reason for publishing these metrics is to get feedback and to learn what metrics other designers work with. Let me know if you think an important metric is missing or if you disagree with these metrics and why.
On #2: If you’re trying to measure effort, not sure if this is detailed enough. I’ve seen interactive prototypes that take a day and others that take weeks. I suppose it depends on whether you have standards for what represents an “interactive prototype”
On #4: Is number of users tested as important as number of complete user testing cycles run?
Nielsen argued that 7 is the max number of test subjects needed per user testing pass. Theoretically your team could be wasting calendar time on a test of 20 or more subjects but still max out their User Testing metric. Changing it to user testing phases completed would require standards to be established for what constitutes a user test: hallway testing, formal tests with users, unit testing on specific features vs. full regression testing with users, etc.
You’re tackling a tough problem, one that took 2-3 graduate business classes to really get my mind around. It’s like chess: you fail if you only look one move ahead.
I’m concerned about #1. First, it’s not clear you specifically mean”page layouts”. So a complex project might have 50 and a simple one could have 4. Second, you have a risk of perverse incentives: a designer can double the number of layouts used for some application while only increasing her workload by 5% – the extra layouts could be necessary. Third, I’m not sure whether interim designs or design alternatives count – and those DO increase workload. If they don’t count, you reduce the incentive to do these valuable activities.
Agree with the comment above about #4.
For #5, You might add some measure of “that actually got implemented”. Else you’ll have change for change sake.
Given that you are doing a lot of testing and you are using the SUS, what about doing something a bit closer to the user? Increase task-based and system SUS scores, as measured with a statistically valid number of users? This approach ties to your four responsibilities directly, but does assume that many of your recommendations get implemented. You may not have sufficient control for that.
Most of these are inward-focused. Given that the goal of our work is to make the end user’s experience better, it may be useful to measure and track some outward-focused metrics.
Does your team have enough influence throughout the process to ensure that the designs they produce make it into the release reasonably intact? If so, metrics like customer satisfaction, time on task, and learnability may be useful.
If the existing dev process makes these types of measurements unfair (e.g., if development or product management routinely implements only small portions of the recommendations), then the initial step may be to improve the dev process itself, perhaps by inserting the UX folks into the QA/acceptance phase.
Very interesting post Russell, I’ve been thinking about it for the last hour or so and have a few thoughts of my own.
The responsibilities you listed are great and match my own, though are a much better articulation than anything I’ve put together. I’ll be using them as a foundation for my future communications within the company.
The metrics are where I’m having a hard time. I don’t think that tracking the amount of designs or layouts is a good metric. It seems that there are better metrics to track than general quantity. For example, would it make more sense to track traffic patterns and conversion in the projects you worked on instead of how many layouts were delivered and prototypes completed? This may complicate tracking beyond usefulness, but if it can be determined, it would be a more valuable stat in my book.
I’m also curious if there’s a better way to track insights. It seems to me that the value isn’t in documenting them, but in putting them to use. Is there a way to track UX/UI-initiated feature requests and their impact on the product(s) and bottom line? If so, that would be powerful.
Thanks for posting your metrics and kick-starting this conversation, I think a lot of us will benefits from it.
For any business process, the three things you classically want to measure are quality, cycle-time, and cost. So if you are trying to create metrics to measure the effectiveness of your design team, I’d ask how your metrics help you get at any of these. I think that most of your metrics seem to be looking at productivity, but measuring output is only half of the equation. You might also want to look at time and cost to deliver a prototype, for example.
Measuring quality of a design effort is always hard and you might think more on this. I note that you’re tracking product improvements, which is good. But want constitutes an improvement. Are there objective measures of user performance or customer satisfaction that you could collect?
Phil — Yes, we would have to provide more detail on what each of the elements is (e.g. “What constitutes an interactive prototype?”). Great point on #4 – will definitely consider that. Thanks!
Barbara — Great points also. We absolutely have to quantify what each of the deliverables is, and avoid perverse incentives. The challenge is controlling the complexity of the metrics we choose. I’m trying to balance “what I really want” with “what I know we will actually follow through on”. I’m fearful of creating a set of metrics that are great on paper but that never get implemented or kept up with. Finally, can you please elaborate on your last point?
Sarah — Can you provide some examples of “outward-focused” metrics? We do have checks and balances to ensure our design are implemented to spec (at least reasonably close), and I would like to use time on task (efficiency), but the challenge there is first establishing a baseline and then showing improvement over time. More work, but arguably worth it.
Alex — we are focused on web applications (I should have mentioned that) and so traffic patterns and conversions aren’t applicable. I’m with you on the insights; I definitely want to follow them through to results but wanted to start out with at least documenting them.
Matt — Good point – cost is not addressed. And yes, we will have to define what an “improvement” is (e.g. shorter time on task maybe) just as we will have to define what exactly a layout and an interactive prototype is.
Thanks to everyone so far for the great feedback and comments!!
One quick test of any metric is to spend 5 minutes trying to hack it – What
if you were your evil twin: how could you make evil happen while still
scoring well on these metrics? Better metrics make life harder for your evil
twin. Lousy metrics make it easy.
> 1) Number of layouts delivered
> 2) Number of interactive prototypes created
> 3) Percentage of product design requests completed by commit date
> 4) Number of users tested
> 5) Number of product improvements made
> 6) Number of products insights documented
One big assumption you’re making is higher numbers means better results. One
excellent prototype might do the work of 5 mediocre ones, but the designer
who tends to need 5 mediocre ones will score better here. Same for # of
users tested (you’re rewarding people with sloppy study designs, or who
can’t win basic arguments without going to the lab), etc. Volume is a very
poor measure for quality. But since measuring volume is easy and popular it
explains the dozens of organizations proud of their fancy metrics, but
somehow in denial of their lousy products. I’m really not a fan of
systematic metrics – it’s a favorite fuel for micromanagers.
You should also note there is nothing wrong with subjective metrics. Why
cant your team score itself 1 to 10 on team performance every month, or even
better, ask your clients & stakeholders to rate your performance. Then at
least you have a metric that is very difficult to manipulate. So what if
it’s not scientific: science is not a panacea. If the goal is to get a sense
of how you’re doing and focus team energy, qualitative measures can be just
as effective as quantitative ones. RMPT can work fine with subjective
measures.
Lastly thinking like a general manager, which I was most of my career, the
only metric I’d ever evaluate you on if I were your boss would be #5: number
of product improvements made. That’s the *only* metric that earns your team
its salary. A favorite scheme I’ve seen used for usability engineers is
simply this: # of usability issues found, # of recommendations made, # of
recommendations approved. You might need a different set for designers, but
you get the idea.
If you discover more layouts, more prototypes, more magic spells, lead to
more approved recommendations, you’ll be rewarded for it. And if those
things (layouts, protos, etc.) turn out to be a waste of time, you wont have
a team of people doing those things anyway just because there is a metric
that rewards it. (But do note that this is pretty much the only way to get
people to respect metrics: they must be tied to rewards).
And finally, I’d guess NetQos is a metric happy place give the business
you’re in, which is fine. But Creative work doesn’t fit metric schemes as
well as, say, performance testing does – creative work is inherently sloppy,
messy and wasteful – I’d seek out other creative groups, PR, Marketing,
Advertising, etc. and see how they’re handling fitting their creative work
into metrics. I suspect you’ll get better ideas from them than from the
engineering and Q&A orgs.
Everything Scott said and one more point: UI, Design, UX, IA and all the associated fields are qualitative fields. They cannot — BY DEFINITION — be measured. The closest you can come to measuring how well you do your job is to measure clients’/customers’ dissatisfaction…does your help desk get fewer calls on this problem than they used to? Are complaints lower than a similar app’s are (and good luck trying to get *that* data). Users seldom laud our work — the closer we are to “perfect”, the less they notice that it’s been done at all — so really, you’re stuck tracking the reverse.
Katie – not sure I agree with the fact that UI cannot be measured. I think it’s more difficult to come up with valid metrics, but I think it’s possible. And I would also argue that if you ever want to be taken seriously at the corporate level, you *better* come up with some sort of quantitative indicator of the value UI brings to the table… however fragile that indicator is. And to your point, maybe “tracking the reverse” is a method worth exploring?
Scott — I agree that volume without any sense of quality is questionable by itself. However I’m planning to evaluate the metrics in conjunction. If your team is cranking out quantity, but no improvements are being made, then you have a problem. Make sense? Volume is important when tied to quality. And I agree with you that qualitative metrics are valuable too. I’ve had mixed experience with having clients/stakeholders rate performance. We actually have that in place now (A, B, C…) and we score straight A’s… but what does that tell me? Where do I improve? I really like your idea on # of recommendations. Will definitely consider that. Lastly – the comment on seeking out other groups in the organization such as marketing — their chief metric (and yes they are evaluated by metrics) is how many leads they generate. It’s quantifiable and measurable over time. Easy for them…
I think it’s great to have metrics. However, after researching this thoroughly for large organizations for the last few years, I think these metrics will keep you busy, but probably not see much improvements in your products or position with within the organization.
Sarah is correct: these are completely inward focused. As Scott mentioned, the number of layouts is irrelevant. Having the *right* layouts is key.
Whenever there’s an issue with product design, it renders itself in some cost to the organization. Either customers aren’t buying (reduced sales), work is being redone (increased development costs), customer support calls are too high (increased overhead costs), or people spend a lot of time arguing over things they really have no information about (decreased productivity).
If it were me, based on what we’ve learned, I’d start with one or more of those. Where is the organization hurting because product design isn’t where it should be. Build metrics around that, preferably measured in dollars.
More on this in an article I wrote here:http://tinyurl.com/5pqkvb
Jared
Jared – with regard to relating metrics to dollars, “much easier said than done.” Did sales go up because of the changes in the sales department, the market, or competitive environment? If product design is give some credit to the change in sales, what percentage exactly? How do you quantify it?
I agree that the number of layouts taken in isolation “may be” irrelevant. But as I mentioned previously that is not my intention; the metrics should be considered in relation to each other. And admittedly I’m also taking into account the quality of my designers — I know my team won’t crank out inferior layouts for the sake of quantity.
I’m also not sure what the concern is regarding “inward focused” metrics. Can you comment? The intention of these metrics is to:
1) Measure our work and help us identify where we are spending our time
2) Guide us to spend our time in the right places
3) Help to communicate value outside of our department
Finally, one last point on cost. Ultimately I hope to get to a point where design output can be related to increased revenue and/or decreased costs. I’m not sure how to get there yet, and I haven’t read a single article on how to do it well either. With that said, to even start down that path requires the cooperation of several other departments that may not be easy to accomplish. You would need sales numbers, profit and loss statements, customer support metrics, and many other chunks of financial and operational data that may not be easy to acquire or to keep acquiring. Why should sales take on the burden of reporting certain data to me so that I can build my metrics? How can I get customer support to accurately account for and identify calls generated by poor usability versus some other reason? Trust me, I’ve been in executive positions for some time now and as straightforward and easy as this may appear, it’s not.
We need metrics that give us some insight and that we can generate relatively quickly and control as much as possible ourselves.
Actually, it’s very easy to equate dollars with design results. More importantly, it’s easy to equate dollars to improved customer and user experience.
Here’s where I’d start: Talk to your support folks. If your product design is not optimal, then they are spending time answering the phones (or handling emails, dealing with a support forum, or other activities to support customers).
You can quantify how much it costs for someone to answer the average question from the customer. You can project that if you could eliminate the need for customers to call with the top 10 questions, you’d save the company that much money.
Then do it.
That’s just one of a ton of ways to quantify customer experience improvements in terms of dollars.
You have as a responsibility to “3) Approve any UI design work done outside of Product Design”. We have a ton of research that shows this will likely stagnate your group, as it makes you a bottleneck and produces a substantial reduction in likely quality. You’re creating a review-and-approve organization and we have lots of data that says that doesn’t have long term success.
Instead, I’d take all 4 responsibilities and replace them with “Ensure that every design decision in the organization is made with up-to-date knowledge of customers, users, their goals, their needs, and the designs that will eliminate frustration and produce the most delight.” This would move from a review-and-approve approach to an educate-and-administrate approach (for former being reactive and the latter being proactive). This completely changes how you approach your work and will change you from a service that produces and approves designs to a resource that improves the customer’s experience.
Metrics that are in relation to each other don’t work because what happens when one goes up without the others. Do you discount it’s value?
As for getting data from sales: I’m betting they want the best product to sell. The easier the product is to sell, the easier they make their commissions. If you can promise them easier sales, you can get their cooperation on any number of fronts. Where are your metrics that ensure that sales will have an easier time selling your product than they do this year?
You want metrics that give you insight. Given that, I’d ditch the following: #1, #2, #3, #5, and #6. None of those are insight generating (even #5). You want metrics that give you insight, then measure the number of hours *all* employees spend watching real users/customers interact with your (and your competitor’s) designs. That’s your critical number. (You separate out the number of hours senior management spends. That’s actually more critical.)
Jared – first, thanks for taking the time to respond with such great insights. Take all my comments as someone who is very engaged and interested in this topic.
Your support example is easy to envision but very difficult to implement. Again, easier said than done. That doesn’t mean you shouldn’t go down this path, but it will be a lot of effort and coordination. Who educates the support people on which problems are related to product usability? Why should support take time out of their busy schedules to help *me* (being devil’s advocate) get the data I want? Getting support to change their processes is a political hurdle. Again, none of these are impossible, but they aren’t easy either. And aside from support, what are some of the other ways? I’m not trying to monopolize your attention on this post, but I find that many of these things are eloquently preached from the podium but after the speech, everyone has a dazed look in their eyes and no idea of how to actually follow through.
I agree with educate-and-administrate, but only to a point. People have different areas of expertise and natural talent. You could educate some on all the important concepts in usability and graphic design and they still would make bad choices and not have the capability to style. So do I educate? Absolutely! Do I still see the need for some approval process despite the obvious bottleneck? Unfortunately yes. Development does code reviews don’t they? With that said, I do firmly believe in giving others the tools to help them make the right decisions (we have implemented an in-house design library of template, patterns, etc. for that purpose specifically, although we still find the need to rework and tweak).
As for the comments on “insight”… I have several issues with that but will have to comment later due to travel.
Again, despite disagreeing on some points I really value your comments (and I’m sure others do too). Thanks so much.
So the benchmarking idea goes something like this:
Test current products, using both task based measures and SUS. This is your current benchmark level. New functions implemented are measured by meeting or exceeding the current benchmark. Indeed, you can add “target SUS” into the product development process, and measure by meeting that. The goal is to create quality new designs, and also test them. The target is to allow you to acknowledge that you might be building something very new or very complex, which will get a lower score naturally. It also helps you allocate your resources and negotiate with your peers.
You can simplify it out as % of new products meeting or exceeding target SUS, whether internally or externally designed.
Then you want to improve your current product/service. Over the quarter or year, your measure can be twofold: increase in system SUS, increase in specific task scores.
Just a niggling comment here: You note that subjective metrics are “not
scientific” but in fact there is a great deal of research into “subjective
metrics” like customer satisfaction. There are different definitions of
“science” and science can involve both qualitative and quantitative methods
and systematic data collection and analysis.
When I managed a usability group, I invited clients to give quarterly
feedback on how well the team was doing (ratings and qualitative comments
that were then coded by similarity). One issue that emerged from this
“client satisfaction” survey was that the type of report desired differed by
stakeholders. From that I learned to give clients examples of our reports
and ask them for feedback about how well the format would support their
needs (requirements input, detailed UI design, high-level consistency
issues, etc.).
[...] 6 Metrics for Managing UI Design “As part of a recent management summit at my company, we were asked to fill out an RMPT matrix for our departments (I head up Product Design). An RMPT matrix consists of (R)esponsibilities, (M)etrics, (P)rocesses, and (T)ools. I have been intending to develop better metrics for both measuring and guiding our design efforts, and this exercise served as a catalyst to get me started. “ [...]
Hi Russell. As others have commented, your metric focus on numbers,
and not quality.
I think you have two primary sets of people to address. The first is
the people you work with (presumably developers and people generating
business requirements/product owners). They will have their own ideas
as to what they see as success from you, which will include whether
you deliver designs on time, and whether they trust your professional
judgement. You could derive some metrics from discussions with them.
The second audience is the users of the applications that you are
developing for. You can survey and talk to users, and get measures of
overall satisfaction, or of satisfaction with key modules. You can
also measure in a number of ways whether people can successfully
complete tasks, and how long it takes, and where they get stuck. You
can measure how many problems you identify with the usability of live
applications, and how many you have design solutions for.
Those are the sorts of things I’d be more interested in, as they
relate more directly to business success.
[...] a first pass at choosing a set of metrics for managing UI design (see “6 Metrics for Managing UI Design), followed by some great feedback and many internal meetings, we finally decided on the following [...]
[...] Full Article here. “As part of a recent management summit at my company, we were asked to fill out an RMPT matrix for our departments (I head up Product Design). An RMPT matrix consists of (R)esponsibilities, (M)etrics, (P)rocesses, and (T)ools. I have been intending to develop better metrics for both measuring and guiding our design efforts, and this exercise served as a catalyst to get me started. Bear in mind that metrics help you focus your efforts and measure your progress, but you are also held accountable to them. [...]
See follow-on post with our final choice for our first metric set (based largely on feedback to this post) here: http://www.dexodesign.com/2008/10/22/3-not-6-metrics-for-managing-ui-design/
entry level resume template…
I personally agree with your comments, but there will always be some people who may not feel the same….