How do you know how effective your product development process is? Are the changes you making to your development process improving effectiveness? How do you measure effectiveness? Reduced time to market? Reduced development cost? Improved product performance? When goals may conflict with one another, how do we measure performance? These are the questions on manager’s minds as they consider how to measure the performance of their product development process.
There is the old adage that you can only manage what you can measure. But what you measure drives people’s behavior. And sometimes it drives behavior in unintended ways. Further, as managers strive to measure different aspects of performance, they can establish a plethora of metrics that require significant effort to capture information and measure. And if we focus measurement on one facet of performance, other areas can get neglected. Finally, what is easy to measure may not be important or relevant to measure.
For example, if we measure designers based on hours per drawing and this measure is then used as the basis for performance reviews, compensation decisions, bonuses, or promotions, what happens? This metric becomes the basis for optimizing what they perform to. Designers may spend less time considering appropriate tolerances, including all needed notes, considering how a part may be inspected, determining the most appropriate and lowest cost material, considering whether are features are manufacturable, and considering whether the part function might be consolidated with another part to simplify assembly. In other words, they deal only with those factors that improve the metric of focus and at least to some degree ignore the other elements of their job. They may even go so far as to manipulate performance measures or fudge data.
So, consider the following factors.
- What is metric used for? It’s important to distinguish between data used for purposes of internal monitoring of performance by development personnel themselves versus data to be used by managers for reward and punishment. Metrics are most useful for internal analysis by development personnel rather than for external evaluation by managers who may not fully understand their limits. Such metrics can be used to inform personnel of their performance and offering recognition to those who have excelled and aid those who have fallen behind. To the extent that they are used to determine pay and promotion, metrics will be subject to gaming the data.
- How useful is the information? Because an activity is easy to measure does not make it worth measuring. The ease of measuring may be inversely proportional to the importance of what is measured. Is what you are measuring representative of what you really want to know? If the information is not very useful or not representative for what you’re trying to achieve, you may be better off not measuring it.
- What do you want to measure? When the items to be measured are influenced by the process of measurement, measurement becomes less reliable. Therefore, the more that the measure relates to people or people and their process, the less reliable the measure becomes. This is because people react to the measurement in ways that can distort the results and the overall objective. On the other hand, if the metric relates to an object (rather than a person) such as a product or manufacturing process, it is more likely it is to be measurable.
- What are the costs of getting the data? Information is never free, and often it is expensive in ways that don’t occur to those who want more of it. Collecting, processing and analyzing data takes time, and a large part of this expense lies in the opportunity costs of the time put into developing the metrics versus developing the product. Every moment that you or your colleagues or employees devote to producing metrics is time not devoted to the activities being measured. Even if the performance measurements are worth having, their worth may be less than the costs of obtaining them.
- Who develops the measurement?Accountability metrics are less likely to be effective when they are imposed from above, using standardized formulas developed by those far from active engagement with the activity being measured. Measurements are more likely to be meaningful when they are developed from the bottom up, with input from development personnel. This means asking those with the tacit knowledge that comes from direct experience to provide suggestions about how to develop appropriate performance metrics. Try to involve a representative group of those who will have a stake in the outcomes. In the best case, they should continue to be part of the process of evaluating the measured data. A system of measured performance will work to the extent that the people being measured believe in its worth.
- Does the measurement create perverse incentives? As individuals work to maximize their own interests, there are inevitable drawbacks to all schemes of measured reward. Consider how the metric could negatively affect their behavior.
Just because performance measures often have some negative outcomes doesn’t mean that they should be abandoned. They are still worth using, despite these potential issues. It’s a matter of trade-offs and judgment related to the trade-offs. Not everything can be improved by measurement, nor can everything that can be measured be improved.