Learning Metrics and Measurements – highlights from our podcast interview with Dr. John R. Mattox

Collecting data is easier now than it ever has been. Both the level of access and the sheer volume of data it’s possible to collect is astounding. It seems we can measure everything. However, analyzing the data we collect to form strategies that aid in meeting company goals is up to us. This is a very important topic that seldom garners the attention it deserves.
Dr. John R. Mattox, a Managing Consultant with the Metrics That Matter team at Gartner, Co-Author of Learning Analytics, and Collaborator on the book Predictive Analytics for Human Resources, is an expert in this space. In our most recent episode of The Hive Podcast, we had the opportunity to chat with John about the manner in which we classify and subdivide metrics in order to analyze and read them. Please see highlights below.
Divide and Conquer
Sometimes when we talk about learning metrics, we talk about it like a big cloud of things, but what are we really talking about?
John says that, rather than talking about frameworks like Kirpatrick’s Four Levels of Evaluation or Philip’s ROI Methodology, we should break down metrics into three different groups:

1. 1. Efficiency

1. 1. Effectiveness

1. Outcomes

Dividing metrics up into these three groups makes it easier to understand what types of information is being collected.  This is increasingly necessary to communicate the value of L&D.
Efficiency tends to center around activity and cost.  
How many people were there?  How much time was spent? How much did it cost?

• These are important metrics to track, especially considering their importance in the next step, but by themselves only carry so much weight.
• Effectiveness metrics are the ones that start to show the value in learning.

What’s the quality of training?  Was it delivered well? Did participants gain knowledge and skills?

• Outcomes are the effect we see the product of learning.

Did the company achieve its KPI’s?  Is performance up? What are the predictive measures?

• Things such as sales, revenue, customer and employee satisfaction is where L&D should be able to say, “Our programs influence these.”

Technology Does Not Equal Strategy
The “data nerd” in John prompts him to measure everything. However, he cautions others on reporting all the data that is collected, especially before there has been ample analysis to understand what’s in there. Measure everything, but don’t report everything.  Your strategic goals should inform what tools you’re using to collect, manage, and analyze the data.   Your CFO, CEO, etc. don’t want to be inundated with massive amounts of data.  It’s overwhelming, indistinct, and can be more confusing than enlightening. The difficulty becomes knowing what we’re measuring.

Figure out where you are in your journey and begin measuring. Some organizations buy the tools and begin to collect the data, but then the information just sits in a repository and no one really does anything with it. It’s a waste of time and money.
Measure, Monitor, Manage
What are we going to do with all this data?  Is it useful? Business leaders have begun to put more pressure on learning leaders to show measurable impact from learning activities. Leadership wants to see positive business outcomes. John encourages us to do this through a measure, monitor, and manage approach:
Measure your KPI’s. Monitor whether your goals for the year are achieving those KPI’s. And if not, Manage your business so you can achieve those goals.

John also offer some advice to those who are working on updating their learning strategy:
Find some experts that you trust.
If you don’t have people in your organization that you consider experts, find some.  They’re all over the place and can save you a lot of time and hardship by providing valuable insight.
Find a good organizational development model.
Metrics That Matter is one such model.  There aren’t a lot of tools on the market for measuring effectiveness and outcomes.  Stick with one that works for your strategy and needs.
Develop a long term strategy that’s not going to fail.
As we’ve mentioned before, technology does not equal strategy.  Without first having a plan of attack, data collection and associated tools can become a waste of time.
Align yourself with the business.
We see L&D leaders who aren’t aligned well enough with the business to drive what’s coming down the pike, or what learning needs to look like. Moving yourself into alignment with business goals will help you drive business for your company. Ultimately, anyone can collect data.  It’s what you can determine from what you collect that is the differentiator.  
To download or stream Dr. Mattox’s podcast, or to listen to any of our other guests, check out The HIVE: Perspectives on Innovation in Talent Development podcast series.