Finding the Right Metric for News
The project our Knight-Mozilla fellow will help tackle was hatched in January during a bus ride to the Austin airport with news brainiac (and karaokaholic) Greg Linch.
He had just written a terrific post on his blog, The Linchpen, about the need for more sophisticated metrics to measure the success or failure of journalism online. I’d been thinking about the same problem, but Greg crystalized the challenge and the opportunity perfectly.
In his words: “So, what if we measured journalism by its impact?”
It sounded to both of us to be an ideal project for someone to sink his or her teeth into. After all, the benchmarks we use now are so ill suited. They are the simplistic, one-dimensional metrics we all know: pageviews, time on site, uniques. We use them largely because they are there and because they are easy — even though we all know they’re a lousy way to measure impact.
We know readers know the difference between a cotton-candy piece about the latest Kardashian kerfuffle and, say, David Barstow’s policy-changing Wal-Mart investigation. One will change laws and lives. The other will be forgotten in a day. Are pageviews or uniques really the right measure? Of course not.
What we do not have are ways of measuring how a piece of journalism changes the way people think or act. We don’t have a metric for impact.
This is not a new problem. The metrics newsrooms have traditionally used tended to be fairly imprecise: Did a law change? Did the bad guy go to jail? Were dangers revealed? Were lives saved? Or least significant of all, did it win an award?
But the math changes in the digital environment. We are awash in metrics, and we have the ability to engage with readers at scale in ways that would have been impossible (or impossibly expensive) in an analog world.
The problem now is figuring out which data to pay attention to and which to ignore. It is about setting up frameworks for testing, analysis and interpretation that are both scalable and replicable.
It’s about finding that clear signal among the white noise that tells us whether our journalism is resonating or not, whether it is having the impact we believe it should. Helping us clear away the noise is the goal of our proposal to host a Knight-Mozilla fellow.
Now, we are under no illusion that in 10 short months we will emerge with a single granite tablet on which all the answers will be found. This is an incredibly complex question, starting with what we even mean by the word impact.
What should emerge from this fellowship is something far more enduring and impactful: A framework, methodology and tool set for newsrooms to study and answer these questions for themselves.
The ideal outcome would be a suite of open-source tools, techniques and best practices that, in aggregate, help all of us understand readers better and enhance the impact of our journalism. At a bare minimum, we hope to start asking the right questions.
If you’re an analytics nerd, a news junkie and think it would be neat to spend some time working on a problem like this using The New York Times newsroom as your laboratory, we’d like to hear from you. The deadline is August 11th.