The Right Mix of Performance Metrics

My three-variable framework leads to a three-step approach to developing measures of operational risk: First, a company must establish its external risk profile by assessing the market's overall complexity and volatility. Then it should determine its internal risk profile based on its chosen business model. Finally, it needs to evaluate its execution efficiency in each core business process and in aggregate.

Many components of this approach are in use today. The challenge is that much of the information is assembled in an ad hoc manner, and there is no unifying process to provide a complete operational risk picture. For example, assessments of market structure lie at the heart of companies' analysis of possible entry into a new market or determination of the right investment allocation in a particular sector as part of the portfolio management process. However, the tools and methods for these assessments vary widely, and many of the conclusions drawn are subjective. Organizations need to move to a series of systematic, fact-based metrics.

They must start by ensuring that they're collecting the right data. This requires developing a complete and balanced picture of key business measures across three dimensions: operational vs. financial, internal vs. external, and leading vs. lagging. Operational data describes the activity within the organization. It includes measures of volumes, cycle times, resource utilization, productivity, and quality. Financial information translates operational metrics into results such as sales, expenses, and profit. Internal information relates to all activities and events occurring within an organization. External information includes measures relating to customers; suppliers; competitors; and more macro factors such as regulation, politics, and the economy. Leading, or predictive, information estimates future measurement values and can be developed for operational or financial, internal or external measures. Lagging, or historical, information reports actual results for a current or previous period.

The fact that most information available today is internal, financial, and lagging is the main impediment to effective management of operational risk. Instead of just-in-time warnings of risks or opportunities, organizations get information just too late to do anything about it.

To overcome this information deficit, companies should effectively combine operational and financial information to form a more complete and timely picture of operational risk than historical financial reports can provide. Operational measures are excellent leading indicators of future financial results: orders predict sales, returns predict credits, complaints predict customer turnover, quality of production processes predicts cost of goods sold. If, for example, a manager sees that the level of calls coming into the telesales department is declining, there is a good chance sales will decline unless the company sees a compensating increase in the proportion of calls that result in a sale. By tracking call center activity and the ratio of sales to calls, managers can detect problems sooner and take immediate corrective action -- for instance, they might increase their promotional spend to minimize the impact of the drop-off in call volume before the problem manifests itself in a revenue shortfall.

Companies should seek out leading indicators aggressively and prize them above all other measures. However, identifying good ones requires both creativity and analysis. An organization should assemble a list of possible predictive measures for its business, then subject those options to statistical testing to determine the strength of the correlation and the time lag between changes in the indicator and the financial result. Pinpointing which indicators offer the most relevant information for decision-making might take one or two iterations. It is difficult to assess a metric's value until managers make real decisions based upon real information, so after determining which measures are predictive, decision-makers should develop a series of alternative reports using candidate metrics and test them with different audiences to isolate those that are most valuable. Then they can develop the appropriate reporting, modeling, planning, and decision-making processes.

Ultimately, operational risk measures can be built into the target-setting process that guides business planning; however, companies should continuously evaluate the relevance of each measure over time, as the indicators themselves are subject to the same forces that increase or diminish the importance of the factors they measure. Not that long ago, phone companies touted the quality and availability of their dial tone as a competitive feature. Now it is a competitive necessity and has little value as a measure of differentiated service.

At the same time businesses are working to expand their stable of leading and operational indicators, they should reconsider the balance of internal and external information they rely on. Organizations can accumulate internal information more eas- ily than external information. Internal systems contain large amounts of data about internal processes and operations (albeit sometimes of questionable quality). But companies have very unstructured processes for collecting and analyzing external data. External information is usually collected on an ad hoc basis and is not effectively integrated into the overall management reporting process. Separate reports describe customer satisfaction, competitive positioning, and market evolution. Often, a manager seeking to understand the implications of external information on internal operations needs to undertake manual analysis to combine external data with internal data; this takes time and slows decision-making.

Battery manufacturer Duracell has long used one leading external indicator to help guide international growth; it looks at changes in a country's GDP to predict future demand for batteries. Over time, the company has seen a strong correlation between growth in GDP and sales of portable electronic devices -- and, hence, battery consumption.

Who Benefits From Operational Risk Management?

Systematically assessing a company's operational risk profile benefits many different constituencies. First, investors, lenders, and regulators can better understand the drivers of the company's financial performance and can derive insight into potential risks or opportunities that may lie ahead. For example, if an organization is about to embark upon a major acquisition, knowledge that the acquiring company does not have a stable, scalable, and standardized IT environment should alert investors of potential integration problems if the deal is consummated. Such knowledge could have been very valuable to investors at the time of USA Waste's acquisition of Waste Management in 1998. Subsequent accounting irregularities and major system failures cost investors dearly only two years later.

For investors and lenders, good operational risk information can answer several key questions: Do the underlying operational processes support the financial results and measures -- and projections of future financial performance? Does the organization's strategy effectively recognize the various types of market structure it will encounter, and is management taking adequate steps to ensure the appropriate skills are available? Is my portfolio adequately balanced from an operational risk perspective? (For example, what proportion of your total portfolio is exposed to significant downside risk should the Internet fail or prove unstable for any period of time?)

Board members also benefit from an increased understanding of the organization's operational processes. Serving on a corporate board of directors used to be a fairly benign obligation. Attained as a confirmation of one's standing in the community, the job offered lavish hospitality and required little onerous work. That has changed. In addition to exercising fiscal stewardship over the organization, directors now must seek assurance that the company's operational processes are effective in capitalizing on new opportunities to create wealth while retaining an acceptable level of risk and control.

Understanding an organization's operational risk profile is valuable for employees, customers, and suppliers, as well. Operational risk factors can often help answer questions such as "Should I work for this company?" and "Should I do business with this organization?"

Finally, insight into operational risk factors offers huge benefits to managers allocating resources and making other decisions. The Hackett Group's benchmarks show that over half of professional finance staff time at an average organization is spent collecting data rather than analyzing it, while only 12 percent of staff time is spent on data collection at a world-class company. This fact suggests several interesting conclusions: The quantity and quality of analysis at an average company must fall short of that at a world-class company. The cost of the analytical process is likely higher at an average company because more analysts are required to reach the same level of understanding. And an average company probably experiences higher turnover of its best analysts because those people must spend so much time digging for data rather than using their analytical skills.

Effective operational risk management is no longer an option. I have little doubt that operational risk management will eventually become a core component of the overall business management process and that we will see the development of more systematic and objective measurement tools, for use by business managers and other stakeholders. (Exhibit 5 illustrates some of the major attributes of an organization that is effectively integrating operational risk into its overall management process.) Any improvement in the ability to predict future performance, whether expectations are for positive or negative change, gives managers the most valuable commodity of all -- time to think and act before it's too late.

Top 10 Reasons Why Budgets Fail What is budgeted is rarely what matters. Most budgets tell you how much will be spent on office supplies but do not tell you how much will be invested in customer retention. Failure to match the desire for detail with predictive capabilities. (How do you estimate the number of red cars that will be sold in Spain next November?) Disconnect between strategies, business plans, and budgets. Strategies talk about innovation, customers, and talent -- while budgets tell you how much will be spent on facilities. Confusion over whether budgets are control or planning processes. Mismatch between accountability and influence. Variances are used to apportion blame for past mistakes, not to make better decisions in the future. Executive management fails to set clear targets up front, leaving managers to develop detailed budgets in a vacuum. Executives issue top-down mandates at the very end of the process when they don't like the result -- thereby invalidating all prior work. Bias toward financial budgeting with little understanding of the true drivers of financial results. Slow and misdirected management reporting averaging 10 days from accounting close to final distribution of all reports.

David Axson was a co-founder of The Hackett Group and is the author of "Best Practices in Planning and Management Reporting" (John Wiley & Sons, 2003).

Previous 1 2 3 4