Categories
Best Practice Design Design Thinking Google Innovation Microsoft Usability User Experience

How to measure the quality of user experience and goals of product

If you can’t measure it, you can’t improve it – Peter Drucker

Peter Drucker wrote 39 books on modern business management.  He is widely regarded as the greatest management thinker of all time.  He is credited with two of the most important quotes in business management.  The first one – If you can’t measure it, you can’t improve it.  This quote is of tremendous value. 

If you can’t measure the effectiveness of user experience, you can’t manage or grow it.  Analyze how users are using your product or service.  Compare different user interfaces in A/B tests.   Collect the right set of data and make informed decisions based on it.

Measuring user experience
Image Credit: Interaction Design Foundation

Peter Drucker’s second quote – Management is doing things right.  Leadership is doing the right things.   It’s tempting to brainstorm a long list of metrics, but it will quickly get unwieldy and hard to prioritize.  Focus on a small set of key metrics that everyone on the team cares about.  Choose the right metrics to measure the effectiveness of user experience

UX researchers at Google have developed couple of useful methods to define metrics appropriate to measure the effectiveness and impact of user experience.  It’s called the HEART framework and Goals-Signals-Metrics (GSM) process.  Let’s look at the framework and process.

How to measure the quality of user experience

The HEART framework measures the quality of user experience.  The Goals-Signals-Metrics process measure the goals of your product or service.

The HEART framework

The HEART framework measures user experience in five categories –  Happiness, Engagement, Adoption, Retention, Task success. 

The HEART Framework and Goals-Signals-Metrics (GSM) Process
The HEART Framework and Goals-Signals-Metrics (GSM) Process

The HEART framework can be applied at different levels – product level, feature level, etc.  For example, you may measure the adoption of new subscription service, but also its specific features.

Happiness

Happiness measures user satisfaction, ease of use, net promoter score, etc.  User satisfaction and ease of use can be measured via surveys.  Net promoter score is how likely users are spreading a positive message about your product.  How does your product’s net promoter score compare with the industry ?

Engagement

Engagement measures the frequency of user involvement.  Metrics include number of visits per user per day, number of photos and videos uploaded per user per day, number of product downloads per user per day, etc.

Adoption

Adoption measures number of new users adopting your product.  Metrics include number of new user sign ups in last month, etc.

Retention

Retention measures the rate at which existing users return.  Metrics include the number of users still active within a given time period.  Distinguish new users from returning users by measuring adoption and retention.  You can measure how quickly your user base is growing.

Task Success

Task success measures the efficiency and effectiveness.  Metrics include time to complete a task, percent of tasks completed, error percentage, etc.

Recommendation

Not all of these may be equally important for your product or service.  Choose the most meaningful ones.  In an enterprise, the “heart” of HEART has minimal value, if any. Happiness and Task Success (HT) retain obvious value, but Engagement, Adoption, and Retention (EAR) become almost moot as users have no choice but to use the product. You may still measure what percentage are using key features of the product.  Bolster the HT with more EAR … or perhaps something else more important to enterprise product users and business owners ?

How to measure the goals of your product

There’s no magical HEART dashboard that can be applied to any product.  The most useful metrics are likely to be specific to your product.  The Goals-Signals-Metrics (GSM) process measures the goals of your product.

Goals

Identify goals so you choose metrics to measure progress towards those goals –  

Vimeo VideosGoal
Engagement CategoryYou want users to enjoy the videos they watch and keep discovering more videos
Vimeo SearchGoal
Task-Success CategoryWhen a user enters a search, you want them to quickly and easily find the videos or channels that are most relevant

Another question to ponder – are you interested in increasing the engagement of existing users, in attracting new users, both?  Different members of your team may have different ideas about the goals of your product.  This is an opportunity to build consensus about the direction for your product.  You can’t design for a disagreement.

Signals

Map your goals to lower-level signals.  There are usually a large number of useful signals for a goal.  Research and analyze which ones to choose.  How easy or difficult is each signal to log and track ?

Vimeo VideosGoalSignal
Engagement CategoryYou want users to enjoy the videos they watch and keep discovering more videosAmount of time they spend watching the videos
Vimeo SearchGoalSignal
Task-Success CategoryWhen a user enters a search, you want them to quickly and easily find the videos or channels that are most relevantEntering a query but not clicking on any of the results

Metrics

Vimeo VideosGoalSignalMetric
Engagement CategoryYou want users to enjoy the videos they watch and keep discovering more videosAmount of time they spend watching the videosAverage number of minutes spent watching the videos per user per day

Recommendation

The GSM process leads to prioritization of the metrics.  Track the metrics related to your top goals and that help you make decision.  Use the drawing worksheet like the one above to help you work through the steps.  Don’t include all the HEART categories if some don’t make sense for your product.  Feel free to add more categories of your own, if needed.

What has most value

At Microsoft Engineering India, we are a team of user researchers and designers who create effective and deep research-driven innovative design solutions for Microsoft internal line of business applications in domains like HR, Finance, Legal, etc.

Our goal is to create, inspire, and support compelling and effective experiences through deep research driven user understanding and innovative design, best practices and continuous improvement in quality in use.

Create, inspire and support compelling and effective experiences
Create, inspire and support compelling and effective experiences

We have developed a method to measure the quality of the applications we design by measuring –

  • Satisfaction
  • Effectiveness
  • Ease of use

We do this during design work (to predict success) and after major releases (to benchmark).  We want to understand –

  • How good are our existing apps ?
  • The effectiveness, efficiency and satisfaction
  • Measure scenario health
  • Cost reduction- fewer priority 1 UX bugs, less rework, etc.
  • How well do we do our work ?
  • What are the benefits of having a UX team ?
  • How does UCD improve the user experience ?
  • How do we extend UX across IT ?
  • ROI – to grow the responsibility for UX across IT ?
  • What training, guidelines, level of understanding are needed ?

Below slide shows the quality in use improving for Microsoft’s internal Careers Web site by adopting user centered design methodology. Read more about How to improve usability by leveraging design best practices.

Measuring the quality in use for Microsoft internal applications
Measuring the quality in use for Microsoft internal applications

How do you measure the effectiveness of user experience and goals for your product ?

What metrics do you track ?

Do share in the comments!

Like this article ? How about giving it a like and share ? Thank you!

By Ramesh Chandak

Principal Content Developer & Technical Program Manager

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s