You Can’t Measure That…
posted by David Sward on November 13, 2006
I hear this all the time from IT managers. Most of the time I hear it when people just don’t want to put in the effort to determine the ROI of an IT solution. If I had a fraction of the ROI I’ve quantified after being told it was too hard to measure, I’d be sitting on a beach in Barbados drinking beer for the rest of my life. Everything is measurable; it is more a question of the reliability and validity of what you decide to measure.
What this typically comes down to is getting the right metrics identified and operationally defining them. We use metrics called business value dials to document the value of our IT solutions. Value dials (e.g., employee productivity, days of inventory, factory uptime, etc.) represent a standard set of metrics; each has a definition and standard calculation. I’ll cover this topic in my next post.
Let’s talk about the concept of operational definitions in more detail; it is a core element of measuring business value. It is the step that translates the concept of business value into some type of measurement. For example, if eight people are asked to measure aggression and report their findings, the likelihood that we would see agreement among them is very low, as each person will define and measure aggression in their own way.
The solution, use an operational definition. Give the same eight people the following instructions: “Today you are going to measure aggression. To do this you will go to Washington Elementary School, from 1-2 pm, go the sandbox in the northwest corner, and, using this form, count the number of times one child either strikes or pushes another child.”
When the reports are in, there will be some discrepancies, a concept known as inter-rater reliability, but overall the data will be similar. Operational definitions are necessary to measure business value. Without operational definitions in place, everyone may be measuring, but it is highly unlikely they are measuring the same thing. Operational definitions make measurement independent of a person or group, repeatable by others, and standardize results.
Comments (8)
tagged: business value dials, metrics, operational definition
Comments
Nov 14 | Ray M. said:
I think it’s great that progress is being made in estimating business value of IT investments. Some of this value is predictable and easily measurable, and in these cases, management can use it to make changes with confidence of positive impact. But I also believe there’s a risk that moving too large a percentage of IT investment through purely data-driven decision making processes can constrain an IT department’s ability to deliver less measurable strategic benefits such as agility and enablement of future innovation.
The push to deliver IT capabilities under an SOA model to support flexible integration going forward is a good example. If I’m building a service, and have the opportunity to build or buy a secure, web-services based mangement interface for the service at a 20% premium in the cost of the project, or to skip that premium and go with a solution with it’s own administrator-facing (web/GUI) mangement interface, my ‘data-driven’ decision will be influenced by the presence of potential consumers of the management interfaces.
Suppose this project is being justified based on an ROI calculated out for five years, but no potential consumers of the SOA management interface exist at T -90 days when ROI is being calculated. The SOA option ‘loses’ because it adds no measurable business value, despite the likely case that one year after deployment, another valuable project will materialize that will be slowed down by the absence of this interface, and eventually drive expensive rework to our imaginary IT service. The end result: higher costs in the first service, slower deployment and higher costs in the dependent project (which may as a result fail to demonstrate enough value to proceed), and a slower, more expensive business.
No one can question the value of data-driven decision making. But one can question the value of their data to drive decision making. We expect the sky to be blue in two years, but we don’t know what the bleeding edge of software usage and capabilities will look like next week. As long as this is true, IT decision makers need to have the vision to attach value to the flexibility and adaptability of systems we choose implement, otherwise data will them lead to predictably mediocre decisions.
Nov 16 | David Sward said:
Ray, you make a number of very good points. I did simplify this topic, but did it on purpose. We do use more than just ROI in making investment decisions. The following paper is good summary of the approach http://www.intel.com/it/pdf/it-investment-value.pdf. The Business Value Index covered in the whitepaper reviews an investment priority methodology; ROI is one factor that contributes to the final result.
What I was trying to say is that you need to take a systemic and objective approach to forecasting and measuring ROI, even when it is hard. When you don’t drive this level of thinking into the organization, what I have found is that all projects start to become “strategic” in nature, so as to avoid the valuation question. You want to get everyone in the organization using the same approach to IT valuation, so that data you feed into an investment decision making process is as credible as possible. It changes the types of arguments people have, but they will still argue.
Getting the right balance between the types of investments you make is important. I would also suggest that tracking your IT investments in a portfolio (e.g., a business value portfolio that tracks forecast and value delivered), aggregating the value delivered, and communicating that value back to your customers in language they understand is equally important. It also helps drive alignment with the business units. An objective approach to measuring the business value of IT makes this easier.
In a perfect world we would be able to describe all intangible business benefits in an objective manner. I don’t work in a perfect world. One advantage of employing a systematic approach to IT value management is that as the organization gets better at doing it, you can start driving out some of the softness.
Nov 22 | n.tava said:
Sorry, it’s not a comment but a question. In one good Semico study, it was said: SOI=Big ROI. My question is simple, what do you think about this? Will Intel need to switch soon or later to SOI?
Feb 08 | Markus S. said:
What I don’t understand in the whole ITBVM-Program is how all these terms are conntected together: BVI, Value Dials, ITE, Business Value etc.
So:
The Business Value Index measures the BV of a whole IT project portfolio, by weighting each project in this portfolio with criteria (=scoring model).
Value Dials measure the value of a certain project. Some of them can be translated to a monetary value, while some are “only” KPI’s. From the Value Dials there is a forecast ROI model derived (=financial attractiveness).
But how is the value matrix built up (“Win-Win-Project”)? How are the components fit together to get the Business Value of: a) a single IT project and b) the whole IT function?
Feb 14 | David Sward said:
Hi Markus,
Your question covers a lot of ground; you can get a graphic at this url: http://www.intel.com/intelpress/BVM_business_value_concepts_v1.pdf. It will help illustrate the relationship between the concepts.
An IT business value (ITBV) program is primarily focused on getting systematic and objective processes around forecasting and measuring the value of IT using KPIs, business value dials, etc. that are defined by the customer and end-user. This information is tracked and updated in a business value portfolio to understand the larger picture and to help the IT organization make better decisions on spending the budget.
The Business Value Index (BVI) is a tool used to prioritize potential IT investments against a set of defined criteria; ROI is only one of the items in the list. Work done in an ITBV program would feed the ROI part of the BVI. I suggest using the BVI as a starting point; the approach, just like the business value dials needs to be modified to align with the business your IT organization supports.
Oct 15 | Andy said:
Hi,
Nice article, but the hardest thing is actually getting consensus on defining the “right” metrcis. For a number of clients I have worked with, senior management is reluctant to tie themselves to hard metrics due to 1) Internal Political issues, 2) Percieved lack of control over IT to deliver solutions and benefits.
However I like the maturity shown at Intel and the fact that you have a clear model. I have started a blog to discuss my experiences in the Benefits realization space, so may reference your valuable insights.
Andy http://benefitsrealization.blogspot.com/
Oct 22 | Nick Spanos said:
I would be interested in seeing a working version. Otherwise, it is simply one more philosophical discussion that cannot be applied to the real world.
Jan 29 | Susan Joslyn said:
Hi David (and other posters), There is great information and perspective here. In the original blog and in everyone’s comments!
I am preparing a presentation and would like to reference this blog and you (David) by name. Could you extend permission via email for that?
Susan