The Challenge of Training Measurement

More than 240 books on Amazon.com and 90 articles on the American Society of Training and Development (ASTD) website are devoted to the measurement and evaluation of training.   There are also more than 2,000 websites featuring consultants and tools to help the measurement process.  Yet, despite all these extensive resources, most training managers do not yet have a complete and actionable measurement program.

Earlier this year, we asked our research members what their top two challenges were.  Ninety-two percent of respondents rated measurement as the top challenge.

The Gap between Desire and Reality

Despite the emphasis on measurement among many practitioners, there is still an alarming gap between expectations and reality.  In 2004 and 2006, we surveyed training managers on what measures they believed were most important and what measures they were capturing consistently.  We found a significant gap.  What most managers do measure are enrollments, completions, satisfaction levels, and volumes.  What they most want to measure are job impact, business impact, and ROI:

In addition to showing a huge gap between the measures most managers want and those that they actually obtain, research also showed no significant change in measures between 2004 and 2006.  We have made little progress in the last two years:  training organizations are still failing at measuring the things they want to measure the most.

Are We Going About Measurement the Wrong Way?

The Limitations and Constraints of Traditional Measurement Models

Why is this?  After more than three years of research in this area (and discussions with many organizations doing an excellent job of measuring training) we have concluded that one reason organizations fall short in meeting their own expectations is that many organizations tend to approach the measurement problem the wrong way .   Rather than establish a measurement process which delivers actionable information to help them improve the effectiveness and efficiency of their operation, they overly focus on measuring the four levels specified by the Kirkpatrick model (1=satisfaction, 2=learning, 3=job impact, 4=business impact).

While the Kirkpatrick model is a good taxonomy to help understand the basics concepts of learning measurement, it is limiting.  Organizations which implement the most effective and complete measurement programs do not overly dwell on the measurement of job and business impact:  they focus on capturing actionable, operational information which is relevant to their business and their key training programs.  They set operational targets through key performance indicators and they measure them consistently and regularly.

To help learning & development organizations better implement such a process, we have developed a new measurement model which we call the Learning Impact Measurement Framework®.  This model and framework is described in detail in our detailed research report: High Impact Learning Measurement®.  This framework is designed to help training managers implement a measurement process (not a project) which will deliver actionable information to monitor and improve both training operations and the programs themselves.  It offers a set of easy-to-understand, nine (9) measurement areas you can use to build a complete measurement program.  And it is built upon an underlying model which fits the processes you use to build training programs and ultimately drive business impact.

Training is an Operational Support Function

As we talked with dozens of companies in this research, we found an important principle which helps guide the measurement process.  Corporate training is not an university or educational enterprise:  it is a business support function.  The training organization exists to support direct line of business functions – functions which manufacture products, sell them, service them, and support them.  In this way, training is very similar to other support functions in the business:  HR, IT, Finance, Marketing, and Facilities.

Consider the diagram below.  The learning & development organization crosses many lines of business – yet generates no direct revenue (except for customer education of course) or profit itself.

As a support function, training’s value to the business should be measured by how well it aligns with these business initiatives — not how well your training programs themselves generate revenue or profit.  (This is one of the missing concepts in the Kirkpatrick model.)  The training strategy, programs, and processes must be built and delivered in direct alignment with the product, business, process, and financial initiatives of these line operations.

For example, if your sales and product organization has objectives to gain market share by 30% through introduction of new product A, your training goals to support these objectives would be as follows:

Figure 3:  Measuring training as a support function

In a sense, all of your programs, initiatives, and investments should be made this way.  There is no corporate learning strategy unless it aligns and contributes to the corporate business strategy.  If you align well (and “alignment” is a very important measure in our framework), then you should be able to capture indicators which directly measure how well you are supporting specific business initiatives.

This is where the Kirkpatrick model is misleading.  It leads you to measure each training program as if should generate direct business impact on its own.  In many cases (e.g. compliance training, ERP application training, customer support training) the program’s value is only meaningful in the way it supports a line-of-business process or initiative.  Rather than try to compute “job impact” and “business impact” directly, training managers should be directly measuring how well they are meeting the needs of these business initiatives.  Performance consulting (or needs analysis) plays a critical role in this alignment process:

Training is a means to an end.  No satisfaction, learning, or job impact measures are meaningful if they are not correctly applied to the right urgent and pressing business problems.  And if we know what these are, then we can measure things that are far more specific to those initiatives.

A Generic Focus on Job Impact is Difficult We recently had a conversation with a training manager who was trying to measure the job impact of an application software training program for customer service representatives.  She was struggling to come up with metrics which define job impact.  I asked her to go back to the original problem she was trying to solve and ask the service managers how they would define improvements in that area.  There were some very specific problems they wanted to improve.  Once these were re-clarified, we encouraged her to measure improvements in these specific, business-driven areas.  In most cases, you will find that these measures are already being captured.  We call this “individual performance improvement” measures in the framework.

When you align yourself in this way another important thing happens:  you can use “other people’s data” (OPD).  Operational businesses, as shown in Figure 2, already have business and customer measures.  Hospitals know, for example, how many needle-sticks, slips and falls, or other operational problems are taking place at the nursing stations.  Sales VPs already know what products are selling and not selling and which sales reps are behind on quota.  By attaching yourself to these managers’ improvement initiatives and helping them to quantify the level of improvement possible your “individual performance improvement” and “organizational performance improvement” measures are already there to use.

Alignment:  An Important Measure

Our research illustrates how important it is to measure alignment.  Alignment means, quite simply, how well are your training programs specifically contributing to your organization’s most pressing business needs.  Alignment means working on the right programs, having line managers and executives committed to the programs, and making sure your learning, performance, and behavioral objectives are completely aligned with the objectives from the line of business.

One trick to measuring such alignment (which is described in detail in our research) is to ask line managers to fill out a “request for training” form.  This form should ask for specific information about the problem to be solved, the underlying cause of the problem, the parameters of the training (required length, media, cost), and other customer objectives.  (Yes I used the word “customer” – because as a support organization the line managers are your customers.)

Such a form provides alignment measurement in many ways:  it helps you quantify the magnitude of the problem and the level of resource to commit;  it gives you details about requirements for the program; it helps you set priorities against other programs, and it gives you the customer’s objectives for success.  In a sense, this type of alignment process tells you precisely what to measure to understand how successful you are.

How Alignment makes Impact Measurement EasierWe recently reviewed an outstanding onboarding program developed by one of the most successful premium womens’ clothing retailers.  This particular company measures the sales performance of each sales person in each store every week.  They publish a weekly report which ranks the sales of each person by store, district, and company-wide.  You can believe this is a “performance-driven” culture.  The Senior VP of HR demanded 100% alignment between the onboarding program and the store operations managers.  So as they rolled out the program they looked at the week by week sales performance of those who took the program and those who did not.  Believe me, this program was very well aligned — and it worked very well. They then looked at the “quality of hires” going into this program, and found that individuals who were rigorously pre-assessed before being hired had similarly higher sales performance.  In fact, they found that “pre-assessment” gained almost the same sales improvement as those who went through the onboarding process, indicating that they could get even higher impact by using both processes.The lesson in this example is that the training manager knew specifically their goals up front and established a measurement metric which was already being captured and was fully aligned with the goals of the program.  Through this measurement exercise they learned many things about the nature of the program itself (actionable information) and they regularly validate this impact on employees “before” and “after” training.  In our research report, we detail a similar measurement process in place at Randstad, one of the world’s largest staffing companies.

What if you are worried about computing ROI?   Rather than computing ROI at the end of a program, for example, we recommend that you compute ROI up front.  By using some of the techniques in our research you can compute the potential level of business contribution before you design the program — and then use this measurement information to make actionable decisions:  how much investment does this program warrant?  What levels of adoption and learning are desirable?  Should this program be blended in a complex way or purchased off the shelf?  What are the desirable outcomes to expect?  What are the measures of true success?

A New Approach:  The Impact Measurement Framework®

Considering the business-focused principles above, our research identifies nine (9) measurement areas to consider in an integrated measurement program.

Measurement Area	Questions it should Answer
Adoption	How well did you reach the desired audience?  Do you know who they are?  What are the obstacles preventing the right level of adoption?
Utility	How well did this program address the learners’ defined workplace needs?  Would they recommend it to their peers?  How “useful” was it?
Efficiency (Financial measures)	How cost-effectively did you develop and deliver the program?  Was it built in the most cost-effective way?  How does its cost per learner and per hour compare to other programs?   Financial measures (e.g. cost to develop, cost per learner, etc.) are very actionable benchmarks, and can be compared against industry averages using The Corporate Learning Factbook or other sources of industry data.
Alignment	How well were the business priorities defined?  How well do business managers (customers) agree with the priorities, design, and objectives of the program?  Do you understand the total quantification of business justification and do you understand the root causes of that problem?
Attainment (of customer objectives)	How well do you meet the specific program and budgetary objectives of your customer?  Do these include deliver on-time? on-budget? within certain time constraints? with certain quality goals?  These are six-sigma “voice of the customer” measures.
Satisfaction	How well did the learners like the content, delivery, experience, and nature of the program?  This can be a very powerful set of measures, far more than “smile sheets.”  How do these satisfaction measures compare to existing skills, motivation, and attitude (see the model below)?
Learning	How well did the desired learning take place?  Does this learning directly correlate to the root cause of the business problem?  Is this learning relevant?
Individual Performance	How well are the individual performance objectives, defined up front, met by the program?  (Based on existing performance measures in the line of business, typically uncovered early in the problem definition.)
Organizational Performance	How well are general business or HR measures affected by the program?  (These are likely to be based on existing measures being captured by the line of business – e.g. sales per employee, etc.)

Figure 4: Impact Framework Measurement Areas

These nine measurement areas were developed through our research.  But rather than blindly trying to understand each independently and applying them to your organization, we have applied these elements to the actual underlying training processes which drive business impact.  We call this the Business Impact Model®.  The model, shown below, decomposes each phase of the business impact process into training processes and elements which impact results.  Many of these elements are not in direct control of the training organization – but you can influence and work with them.  The nine measurement areas (shown above) are then specifically applied to these red and orange areas of the Impact Model to help training managers decide what they want to measure.

Business Impact Model®:  How Training Drives Impact

 

 

Figure 5:  Business Impact Model®

For example, if you work for a large retail organization and one of the biggest training investments you manage is in-store training for safety and loss prevention, your biggest impact issues will be targeting and marketing. Can you get all the right people to take the courses?  So in your case, the framework would point you toward measuring “adoption.”  Adoption measures would tell you who is supposed to take the training and help you understand who is and is not trained, why they are not trained, and what motivational, marketing, or management issues are necessary to drive high adoption.  Such a measure, while simple, can be profoundly important in certain programs.

Bringing it Together:  The Impact Measurement Framework®

 

 

Figure 5:  Business Impact Measurement Framework®

The green boxes are the measurements which monitor, track, and evaluate these phases into the impact model.  Together you can see the power of the whole measurement framework.  You can decide which elements of your training process or impact you believe need the most attention and put in place a measurement program to monitor and evaluate these elements.

As you start to think through these measurement areas (in green above) you will see that they fall into two categories:  those indicators (KPI’s) which can be used to indicate your alignment and impact, and those evaluative measures which actually try to directly evaluate effectiveness.  We strongly urge you to focus in both areas.  Training managers often spend too much time on training evaluation and not enough time on key performance indicators.  The latter are measures which you can use to help you make operational decisions – they, in essence, make sure that you are “keeping the lights turned on” through your training programs.

 

 

Figure 6:  Business Impact Measurement Framework Areas by Type

Summary:  Measurement is a Journey

Measurement of training is a journey, not a destination.  And it is a critical one:  with organizations spending 1-4% of their total payroll on training, it is important to know that these dollars are being well spent.  Our High Impact Learning Organization research showed that training organizations with an integrated planning and measurement process are up to 15-23% more effective and efficient than those without such a process.

Don’t be afraid to start small.  Focus on measurement as a process not a project.  As your business changes and your processes mature, your measurement processes will evolve.  We hope that this article and our underlying research will help you make measurement an easier, more business-focused, and more valuable process than ever before.  We look forward to hearing from you with feedback, comments, and examples of your measurement challenges and successes.