Written by Patti P. Phillips and Jack J. Phillips and published by Chief Learning Officer in 21 September 2021
When thinking about ROI, most people imagine complex calculations, finance and accounting challenges, and misguided measures. In reality, the ROI calculation is a relatively fundamental calculation, comparing benefits with costs in financial terms. The ROI calculation is not needed for most programs. In most situations, other measures reflect the major purpose of why you are implementing a program. However, in some situations, ROI is needed, and sometimes it’s required.
ROI is part of a chain of value, as shown in Figure 1. This simple value chain is essential to understanding how value is delivered from any learning and talent development program. While each level of outcome provides a perspective of value, ROI answers the question: Is the program worth it? Did it provide enough monetary benefits (which are usually cost savings or cost avoidance) to cover the cost of the program? All of us make ROI decisions almost daily as we purchase items or services. As you think about buying something, you ask, is it worth this much money? The question reflects a common concern. It shouldn’t come as a surprise when executives, who fund learning and talent development, want to know the ROI of a significant program requiring a large budget and many resources.
With that backdrop, the remainder of this article will address the four questions we often hear about the use of ROI:
- Does everything come down to money?
- If the ROI is calculated, won’t that become the basis for the program’s future?
- Is a negative ROI a death sentence for my program?
- I work in a nonprofit. Is ROI even appropriate?
1. Does everything come down to money?
The short answer is no. Sometimes behavior change is the ultimate purpose of a program or process. If so, your evaluation can stop there. The same is true with impact. Not all impact measures need to reflect monetary gains or savings. Impact without the money represents intangible measures in the value chain, but it is not necessarily less important than ROI.
For example, a large technology company established a crash course to teach employees how to work remotely when COVID-19 hit the United States. Ensuring that employees knew how to work remotely was necessary, and the executives who requested the course wanted to make sure the remote working solution was effective. They didn’t need to see the ROI; they wanted to know if individuals were working as a team (teamwork), with good coordination and communication (collaboration), and getting their work done properly (productivity).
The team tracked employees’ reaction to working remotely, their knowledge of processes and ability to best work remotely, and their success with working remotely in the organization (Levels 1, 2 and 3). To understand the impact, they asked employees the extent to which this program helped them improve teamwork, collaboration and productivity using a five-point scale. Data was collected from employees and their managers. Executives wanted the managers’ perspective to ensure that teamwork and collaboration were successful and employees were productive. This approach provided executives with data showing the value of this program. There was no need to push the evaluation to the ROI level because Level 4, Intangible Impact, was all they needed.
2. If the ROI is calculated, won’t that become the basis for the program’s future?
The answer is no. While there are five levels of outcome data, these levels include six types of data — reaction, learning, application, tangible impact, intangibles and ROI. They are all important data sets, and ROI is only one of them.
For example, a religiously affiliated health care system was experiencing too many complaints of sexual harassment. It appeared that some physicians and leaders were making comments that employees (particularly the nursing team) considered offensive. The complaints of sexual harassment were made to the HR director, who, by policy, had to investigate the complaints and resolve the issue. The root cause of most of the complaints was a lack of knowledge of what constitutes sexual harassment in the organization. Essentially, sexual harassment is in the eye of the beholder. If a person finds another person’s actions or comments offensive, then it constitutes harassment. The organization conducted a learning program on sexual harassment awareness designed to prevent these complaints. (See “Value for Money: Measuring the Return on Non-Capital Investments,” Chapter 1: Measuring ROI in Sexual Harassment Prevention.)
Reaction data to the program was collected. A pre-/post- test was administered to measure knowledge of the organization’s policy and of what constitutes sexual harassment. An anonymous survey to a 25 percent sample of employees was conducted to measure application. The survey revealed that, for the most part, the sexual harassment activities ceased to exist. Consequently, the sexual harassment complaints declined dramatically, showing a significant cost avoidance. When the cost avoidance was compared with the cost of the program, the ROI was calculated, and the program yielded a very positive ROI.
Was the ROI necessary? No. The purpose of the program was to change the harassing behaviors. Data from the anonymous survey provided evidence that the goal was achieved. So, why take the evaluation to ROI?
The HR director wanted to remind executives that preventive programs can work. Essentially, the sexual harassment concern was a compliance issue. Individuals need to comply with all the rules and regulations regarding sexual harassment. The HR director wanted to show that tackling this issue was a good business decision. It saved the organization costs, and when you compare the cost of the program with the cost savings, it becomes clear it was a good business decision. In this case, the measure of most importance was Level 3, Application. ROI added the finishing touch, or, as some people say, it was the icing on the cake.
The main point is that ROI is just one of six types of outcome data and should not be the sole basis for a program’s future.
3. Is a negative ROI a death sentence for my program?
The short answer is no. A program would only be discontinued if it’s the wrong solution. A negative ROI is not necessarily an indicator a program is the wrong solution. Negative ROIs often merely indicate a breakdown occurred along the way. Programs can break at any point — from reaction to learning to application or even impact — if the program isn’t clearly aligned to the business in the beginning.
Kaycee Buckley, global commercial training manager for a large health care company, conducted an ROI study on a coaching program provided to sales managers who coached the sales team. (See “Proving the Value of Soft Skills: Measuring Impact and Calculating ROI,” Chapter 15: Measuring ROI in Coaching for Sales Managers.) After conducting the study, she realized the coaching was not as successful as anticipated or desired; therefore, the ROI was negative. When she presented the results to the management group, she included data showing where the program broke down. Essentially, the sales managers were not following through with the coaching. The sales managers realized the coaching was important; they just were not doing their part to make it successful.
This study showed clearly that the program was costing more than it was benefiting the company because sales managers were not using what they were learning. The organization changed the approach to solution implementation, and the managers undertook a renewed effort to make it work and deliver a positive ROI on the next offering. While the ROI was negative, the executive group was impressed with the kind of data the L&D team provided to them. It helped them see a more complete picture of what was happening.
This type of problem often surfaces when you wait for executives to ask for the ROI on a program. Some L&D managers say, “We don’t have the request for ROI, so we are not pursuing it.” That’s a risky mindset. When a request for ROI comes your way, three things happen. First, you have a short timeline to provide data when you often need much more time to do so. Second, you are now defending the investment decision after the fact. Third, ROI is now on an executive’s agenda.
The key is to be proactive. Play offense and position ROI on your agenda. This positions you to drive the initiative on your timeline. When you discover a program is not working, as Buckley did, show what needs to change to make it work. This approach puts you in a much better position to avoid the ill effects of a negative study.
4. I work in a nonprofit. Is ROI even appropriate?
Yes, ROI is appropriate. Consider this situation: Janet Simmons is CEO of HOPE Ministries in Baton Rouge, Louisiana. Among other services, her company provides training and coaching to organizations, helping them recruit and train unemployed or underemployed individuals. She’s helping serve a vital segment in the community.
A major foundation is funding some of her efforts to help companies reach out and hire the “unemployable,” as these individuals are sometimes labeled. The companies that work with HOPE Ministries are interested in the services provided and the individuals who need help. At the same time, they invest money and resources into the process and would like to see the impact and even ROI of these efforts. And HOPE Ministries wants to see the ROI because they can use this data to convince other organizations to become involved. The foundation doesn’t need the ROI; they just want to make sure these individuals are employed, retained and successful in the organization. That’s Level 4, Impact.
We need to think about different perspectives. In this case, the client companies and the nonprofit need the ROI. The foundation funding the ministry doesn’t. ROI depends on the stakeholder’s perspective.
Evaluation studies can be very helpful for nonprofits. At present, Simmons is measuring the success of a particular training and coaching program to help an organization with a retention issue, as their turnover is far too high. She is measuring this program through the value chain, including the ROI, and using the ROI Methodology as shown in Figure 2. This system is a way of designing for and delivering success and measuring the success of the program.
ROI is an important measure. It answers the question: Is the program worth it? When to use it depends on your perspective and the purpose of your program.