Software Metrics: Ten Traps to Avoid

As software development gradually evolves from art toward engineering, more and more developers appreciate the importance of measuring the work they do. While software metrics can help you understand and improve your work, implementing a metrics program is a challenge. Both the technical and the human aspects of software measurement are difficult to manage.

According to metrics guru Howard Rubin, up to 80 percent of software metrics initiatives fail [Rubin, 1991]. This article identifies ten traps that can sabotage the unsuspecting metrics practitioner. Several symptoms of each trap are described, along with some possible solutions. By being aware of these common risks, you can chart a course toward successful measurement of your organization's software development activities. However, it is important to realize that none of the solutions presented here are likely to be helpful if you are dealing with unreasonable people.

Objectives of Software Measurement

As you design, implement, and adjust your software metrics program (and it will need adjusting as time goes on), it's important to keep these four primary objectives of software measurement in sight:

1. To collect objective information about the current state of a software product, project, or process.
2. To allow managers and practitioners to make timely, data-driven decisions.
3. To track your organization's progress toward its improvement goals.
4. To assess the impact of process changes.

It is easy to lose sight of these objectives and collect data just to say you have it, or just because you are able to measure it. Emphasize the business and technical results you are trying to achieve, link the metrics program to your software process improvement activities, and make sure the data you collect gets used for constructive purposes.

Trap #1: Lack of Management Commitment

Symptoms: As with most improvement initiatives, management commitment is essential for a metrics effort to succeed. The most obvious symptom that commitment is lacking is when your management actively opposes measurement. More frequently, management claims to support measurement, and effort is devoted to designing a program, but practitioners do not collect data because management hasn't explicitly required it of them.

Another clue that managers aren't fully committed is that they charter a metrics program and planning team, but then do not assist with deploying the program into practice. Managers who are not committed to software measurement will not use the available data to help them do a better job, and they won't share the data trends with the developers in the group.

How can you distinguish true commitment from lip service? First, look for allocation of resources, including capable people (not just whoever happens to be free at the moment) and money for tools. Necessary tools include metrics analysis software for source code, a database to store the collected data, charting and reporting software, and scripts to automate the data collection and analysis processes. A committed manager will also issue a policy to the organization, clearly stating the objectives of the metrics program, emphasizing his personal interest in the program, and stating his expectations of participation. A committed manager will help the program succeed by overcoming the resistance that mid-managers and project leaders may exhibit to the measurement initiative. This is virtually impossible to accomplish from the bottom up, so the drive to succeed must come from senior management.

Solutions: To persuade managers of the value of software measurement, educate them! Good resources to start with are Software Metrics: Establishing a Company-Wide Program Practical Software Metrics for Project Management [Grady, 1987] and Process Improvement [Grady, 1992]. Tie the metrics program to the managers' business goals, so they see that good data provides the only way to tell if the organization is becoming more effective. You also need management's input to help design the metrics program, to ensure that it will meet their needs. Those selecting the data items to collect should attempt to hear the "voice of the customer" from the managers at various levels who constitute primary audiences for the program.

If you cannot obtain commitment from senior management, turn your attention to the project and individual practitioner level. There are many valuable metrics that developers and project teams can use to understand and improve their work, so focus your energy at those who are willing to try. As with any improvement initiative, grass roots efforts can be effective locally, and you can use positive results to encourage a broader level of awareness and commitment. However, expect to run into a glass ceiling that will limit the potential organizational breadth of the metrics program unless managers at various levels help make it happen.

Trap #2: Measuring Too Much, Too Soon

Symptoms: Hundreds of aspects of software products, projects, and processes can be measured. It is easy to select too many different data items to be collected when beginning a metrics program. You may not be able to properly analyze the data as fast as it pours, so the excess data is simply wasted. Those who receive your summary charts and reports may be overwhelmed by the volume and tune out.

Until a measurement mindset is established in the organization, expect resistance both to the concept of measuring software, and to the time required to collect, report, and interpret the data. Developers who are new to software measurement may not believe that you really need all the data items you are requesting. A long list of metrics can scare off some of the managers and practitioners whose participation you need for the program to succeed.

Solutions: Begin growing your measurement culture by selecting a fairly small, balanced set of software metrics. By balanced, I mean that you are measuring several complementary aspects of your work, such as quality, complexity, and schedule. As your team members learn what the metrics program is all about, and how the data will (and will not) be used, you can gradually expand the suite of metrics being collected. Start simple and build on your successes.

The participants in the metrics program must understand why the requested metrics are valuable before they'll want to do their part. For each metric you propose, ask, "What can we do differently if we have this data?" If you can't come up with an answer, perhaps you don't need to measure that particular item right now. Once the participants are in the habit of using the data they collect to help them understand their work and make better decisions, you can expand the program.

Trap #3: Measuring Too Little, Too Late

Symptoms: Some programs start by collecting just one or two data items, which may not provide enough useful information to let people understand what's going on and make better decisions. This can lead stakeholders to conclude that the metrics effort is not worthwhile, so they terminate it prematurely.

Another obstacle to getting adequate and timely data is the resistance many software people exhibit toward metrics. Participants who are more comfortable working undercover may drag their feet on measurement. They may report only a few of the requested data items, report only data points that make them look good, or turn in their data long after it is due. The result is that managers and project leaders don't get the data they need in a timely way.

A metrics program has the potential to do actual damage if too few dimensions of your work are being measured. People are tempted to change their behavior in reaction to what is being measured, which can have unfortunate and unanticipated side effects. For example, if you are measuring productivity but not quality, some people may change their programming style to generate more lines of code and therefore look more productive. I can write code very fast if the quality is not important.

Solutions: As with Trap #2, the balanced set of metrics is essential. Measure several aspects of your product size, work effort, project status, quality of product, or customer satisfaction. You don't need to start with all of these at once, but select a small suite of key measures that will help you understand your group's work better, and begin collecting them right away. Since software metrics are usually a lagging indicator of what is going on, the later you start, the farther off-track your project might stray before you realize it. Avoid choosing metrics that might tempt program participants to optimize one aspect of their performance at the expense of others. Make participation in the metrics program a job expectation.

Trap #4: Measuring the Wrong Things

Symptoms: Do the data items being collected clearly relate to the key success strategies for your business? Are your managers obtaining the timely information they need to do a better job of managing their projects and people? Can you tell from the data whether the process changes you have made are working? If not, it's time to reevaluate your metrics suite. Another symptom of this trap is that inappropriate surrogate measures are being used. One example is attempting to measure actual project work effort using an accounting system that insists upon 40 labor hours per week per employee. Force-fitting an existing solution to a specialized problem will not necessarily give you the high-quality information you need to really understand how your organization is performing its software work.

Solutions: Select measures that will help you steer your process improvement activities, by showing whether process changes are having the desired effect. For example, if you're taking steps to reduce the backlog of change requests, measure the total number of requests submitted, the number open each week, and the average days each request is open. To evaluate your quality control processes, count the number of defects found in each test and inspection stage, as well as the defects reported by customers.

Make sure you know who the audience is for the metrics data, and make sure the metrics you collect will accurately answer their questions. As you design the program, leverage from what individuals or project teams are already measuring. The goal/question/metric (GQM) paradigm works well for selecting those metrics that will let you answer specific questions associated with organizational or project goals. With GQM, you first determine your improvement or business goals. Next, you identify the questions you'll have to answer to determine if you are making progress toward those goals. Finally, you select the metrics that will give you the necessary data to answer those questions. Several companies, including Hewlett-Packard and Motorola, have successfully applied GQM [Daskalantonakis, 1992].

Trap #5: Imprecise Metrics Definitions

Symptoms: Vague or ambiguous metric definitions allow every practitioner to interpret them differently. One person counts an unnecessary software feature as a defect, while someone else does not. Time spent fixing a bug found by testing is classified as test effort by one person, coding effort by another, and rework by a third. Trends in metrics tracked over time may show erratic behavior because individuals are not measuring, reporting, or charting their results in the same way.

You'll have a clue that your metrics definitions are inadequate if participants are frequently puzzled about what exactly they are being expected to measure. If you keep getting questions like, "Do I count unpaid overtime in the total work effort?" you may be falling into this trap.

Solutions: A complete and consistent set of definitions for the things you are trying to measure is essential if you wish to combine data from several individuals or projects. For example, the definition of a line of code is by no means standard even for a single programming language. Much of the published metrics literature doesn't even define what a line of code meant to the organization whose experience is being described. Standardize on a single tool for collecting metrics based on source code; PC-METRIC from SET Laboratories is a good choice (http://www.molalla.net/~setlabs). Versions are available for both the DOS/Windows and Unix environments, for many programming languages. Automate the measurement process where possible, and use standard calibration files to make sure all participants have configured their tools correctly.

A few metrics, such as function points, do have standard definitions available, which makes it easier to compare data collected by your organization with that of other companies for external benchmarking. However, there are even different kinds of function points, so make sure everyone involved in the data collection is applying the same definitions. A variety of metrics have been defined for use with object-oriented software, also [Lorenz, 1994].

Those designing the metrics program must create a precise definition for each data item being collected, as well as for other metrics computed from combinations of these data items. This is much harder than you might suspect. Plan to spend considerable time agreeing on definitions, documenting them as clearly as possible, and writing procedures to assist practitioners with collecting the data items easily and accurately. Pilot projects are an effective way to test your metrics definitions and collection procedures with a representative sample of metrics program participants.

Trap #6: Using Metrics Data to Evaluate Individuals

Symptoms: The kiss of death for a software metrics initiative is to use the data as input into an individual's performance appraisal. Using metrics data for either reward or punishment, such as rank-ordering programmers based on their lines of code generated per day, is completely inappropriate. When someone knows that the numbers she reports might be held against her, she'll either stop reporting numbers at all, or only report numbers that make her look good. Fear of the consequences of reporting honest data is a root of many metrics program failures.

Solutions: Management must make it clear that the purpose of the metrics program is to understand how software is being built, to permit informed decisions to be made, and to assess the impact of process changes on the software work. The purpose is NOT to evaluate individual team members. Control the scope of visibility of different kinds of data; if individual names are not attached to the numbers, no individual evaluations can be made. However, it is appropriate to include the activity of collecting and using accurate (and expected) data in an individual's performance evaluation.

Certain metrics should be private to the individual; an example is the number of defects found by unit testing or code review. Others should remain private to a project team, such as the percentage of requirements successfully tested and the number of requirements changes. Some metrics should have management visibility beyond the project, including actual versus estimated schedule and budget, and the number of reported and open customer-reported defects. The best results are achieved when individuals use their private metrics data to judge and correct themselves, thereby improving their personal software process.

Trap #7: Using Metrics to Motivate, Rather than to Understand

Symptoms: When managers attempt to use a measurement program as a tool for motivating desired behaviors, they may reward people or projects based on their performance with regard to just one or two metrics. Public tracking charts may be pointed out as showing desirable or undesirable results. This can cause practitioners who are using the charts to understand what's up with their software to hide their data, thereby avoiding the risk of public management scrutiny. Managers may focus on getting "the numbers" where they want them to be, instead of really hearing what the data is telling them. As with Trap #3, the behavioral changes stimulated by motivational measurement may not be the ones you really want.

Solutions: Metrics data is intrinsically neither virtuous nor evil, simply informative. Using metrics to motivate rather than to learn has the potential of leading to dysfunctional behavior, in which the results obtained are not consistent with the goals intended by the motivator [Austin, 1996]. Metrics dysfunction can include inappropriately optimizing one software dimension at the expense of others, or reporting fabricated data to tell managers what they want to hear.

Stress to the participants that we must have accurate data if we are to understand the current reality and take appropriate actions. Use the data to understand discrepancies between your quality and productivity goals and the current reality, so you can improve your processes accordingly. Use the process improvement program as the tool to drive desired behaviors, and use the metrics program to see if you are getting the results you want. If you still choose to use measurement to help motivate desired behaviors, be very careful.

Trap #8: Collecting Data That Is Not Used

Symptoms: The members of your organization may diligently collect the data and report it as requested, yet they never see evidence that the data is being used for anything. People may grumble that they spend precious time measuring their work, but they don't have any idea why this is necessary. The required data is submitted to some collection center, which stores them it a write-only database. Your management doesn't seem to care whether data is reported or not (related to Trap #1).

Solutions: Software engineering should be a data-driven profession, but it cannot be if the available data disappears from sight. Project leaders and upper management need to close the loop and share results with the team. They need to relate the benefits of having the data available, and describe how the information helped managers make good decisions and take appropriate actions. Selected public metrics trends must be made visible to all stakeholders, so they can share in the successes and choose how to address the shortcomings.

Today's current data becomes tomorrow's historical data, and future projects can use previous results to improve their estimating capability. Make sure your team members know how the data is being used. Give them access to the public metrics repository so they can view it-and use it-themselves. Remember to protect the privacy of individuals, though. One team member should never be able to view the personal data of another, although project- and organization-level data should be visible to all members of the project team.

Trap #9: Lack of Communication and Training

Symptoms: You may be falling into this trap if the participants in the metrics program don't understand what is expected of them, or if you hear a lot of opposition to the program. Fear of measurement is a classic sign that the objectives and intent of the program need to be better communicated. If people do not understand the measurements and have not been trained in how to perform them, they won't collect reliable data at the right times.

Solutions: Create a short training class to provide some basic background on software metrics, describe your program, and clarify each participant's role. Explain the individual data items being collected and how they will be used. Defuse the fear of measurement by stressing that individuals will not be evaluated on the basis of any software metrics data. Develop a handbook and web site with detailed definitions and procedures for each of the requested data items. Top-down communication from management should stress the need for data-driven decision-making, and the need to create a measurement-friendly culture.

Trap #10: Misinterpreting Metrics Data

Symptoms: A metric trend that jumps in an undesirable direction can stimulate a knee-jerk response to take some action to get the metric back on track. Conversely, metrics trends that warn of serious problems may be ignored by those who don't want to hear bad news. For example, if your defect densities increase despite quality improvement efforts, you might conclude the "improvements" are doing more harm than good, and be tempted to revert to old ways of working. In reality, improved testing might well find a larger fraction of the defects that are present-this is good!

Solutions: Monitor the trends that key metrics exhibit over time, and don't overreact to single data points. Make sure you understand the error bars around each of your measures, so you can tell whether a trend reversal is significant. If you can figure out why, say, your post-release bug correction effort has increased in two successive calendar quarters, you can decide whether corrective action is required. Allow time for the data you're collecting to settle into trends, and make sure you understand what the data is telling you before you change your course of action.

Requirements for an Effective Metrics Program

While individuals and project teams can easily apply a variety of metrics to track and improve they way they work, a successful organization-wide metrics program has several additional prerequisites. Most important is management commitment to creating a measurement-driven culture. The metrics planners must select a manageable, balanced set of relevant metrics and write clear definitions of the various data items being collected. Both the expectations and the results of the program must be clearly communicated to the participants, using training to explain the program and defuse the fear of measurement that is so common among software developers. Integrate measurement with your software process improvement program, using the latter to drive the desired behavioral changes, while relying on the metrics program to monitor results. Finally, you can evaluate individuals on their willingness to participate in the program, but never on the basis of the data they report.

Many of us struggle with how to implement a sensible metrics program that gives us the information we need to manage our projects and organizations more effectively. Staying alert to the ten risks described here can increase the chance of successfully implementing a software metrics initiative in your organization.

Bibliography

Austin, Robert D. Measuring and Managing Performance in Organizations. New York: Dorset House Publishing, 1996.

Daskalantonakis, Michael K. "A Practical View of Software Measurement and Implementation Experiences Within Motorola," IEEE Transactions on Software Engineering, vol. 18, no. 11 (November 1992), pp. 998-1010.

DeMarco, Tom. Controlling Software Projects. Englewood Cliffs, N.J.: Yourdon Press/Prentice-Hall, 1982.

Grady, Robert B., and Deborah L. Caswell. Software Metrics: Establishing a Company-Wide Program. Englewood Cliffs, NJ: Prentice-Hall, 1987.

Grady, Robert B. Practical Software Metrics for Project Management and Process Improvement. Englewood Cliffs, N.J.: PTR Prentice-Hall, 1992.

Hetzel, Bill. Making Software Measurement Work. Wellesley, Mass.: QED Publishing Group, 1993.

Lorenz, Mark, and Jeff Kidd. Object-Oriented Software Metrics. Englewood Cliffs, NJ: PTR Prentice-Hall, 1994.

Rubin, Howard. "Measuring 'Rigor' and Putting Measurement into Action," American Programmer, vol. 4, no. 9 (September 1991), pp. 9-23.

Wiegers, Karl E. Creating a Software Engineering Culture. New York: Dorset House Publishing, 1996.

Acknowledgment

This paper was originally published in Software Development, October 1997. It is reprinted (with modifications) with permission from Software Development magazine.