Editor's note: This is the first part in a multi-part series examining the pitfalls of sustainability measurements that draws on examples from outside the business world. Part 2 will offer examples from criminal justice and financial services, and future installments will bring in other fields.
It’s a great irony that the best-intended measurement programs can create perverse incentives or unintended consequences, like cheating scandals born from school testing initiatives.
Deloitte recently reported that two-thirds of CFOs now are driving sustainability activities, putting quantifiable metrics reporting front and center. While we (mostly) support expanding sustainable business measures and work like GRI’s new G4 framework guidelines, we also caution against increased risks of reaching incomplete or only partially correct conclusions.
Why the “hazards ahead” sign? Mainly because recent history shows us that rational, objective measurement designs are significantly challenged by pervasive human idiosyncrasies, biases and blind spots.
In this series, we’ll look at some of these measurement pitfalls from largely outside the business world. By taking a meta-look at the “metrics” of metrics, we can learn from these examples to avoid or course-correct problems on the way to improved sustainable business measurements.
We start off with a major pitfall from education.
Pitfall 1: Counting what’s easy to count rather than what’s important
Einstein famously said, “Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted.” This pitfall is the realm of overly emphasized, inappropriate numbers, or numbers without sufficient context and explanation to make them meaningful.
Further, the very act of establishing a metric focuses attention spans on it, and can divert attention away from other aspects of a topic that have not been “metric-fied.” It’s true that creating metrics for these other aspects, or “intangibles,” sometimes may be difficult or feel flat-out impossible. But much like the smaller, quieter sibling in the bird nest, there are consequences when non-quantifiables are out-competed and left off the measurement dashboard. Since we don’t yet fully understand which elements will prove to be significant for sustainability measurement, disregarding “difficult to measure” markers creates risk.
Here’s a real-world illustration: Billions of dollars have been spent in the last two decades to measure teacher effectiveness and student achievement in U.S. public schools, with the unintended consequences of cheating scandals, testing failure cover-ups and a disheartening lack of convincing evidence that all that money has done any good.
High-stakes standardized testing, as it exists today in the wake of the 2001 No Child Left Behind Act (NCLB), hasn’t worked as it was intended. The Atlanta Journal-Constitution does a great job of laying this out in its 2012 “Cheating Our Kids” investigative series. Under the pressure to produce improving scores, with money and jobs at stake, the cheating scandals and cover-ups we keep seeing were practically inevitable.
Next page: Important but less tangible indicators