> quotes

The Numbers Game
By: Evantheia Schibsted 
Issue: June 2000

The shocking truth is that statistics are only as credible as the sources that produce them.

When New Economy research firms speak, everyone listens. The statistics they produce make their way into the press; into the pitches of public relations folks; into the hands of startup planners attempting to woo investors. And even though a lot is riding on the reassuringly precise numbers produced by firms such as IDC, GartnerGroup, The Yankee Group, Forrester Research, and Jupiter Communications, few question the validity of their statistics–despite the billion-dollar disparities that often exist among competing studies.

Consider the numbers estimated for U.S. consumer ecommerce listed in the chart below. For 2003, the lowest and highest estimates are separated by $87 billion. You’d be better off throwing darts at a spreadsheet.

Why the wild discrepancies? There are a number of reasons. For example, research houses define "ecommerce" in different ways. Forrester, Jupiter, and Yankee Group count only purchases transacted online, while Cyber Dialogue and IDC include purchases researched online but transacted elsewhere.

Firms also gather data from different sources. Cyber Dialogue and IDC use demand-side data–information gathered by asking consumers what they plan to spend–while Giga Information Group and ActivMedia use supply-side data–estimates of future demand based on past sales. Some, like Forrester, Jupiter, and Yankee Group, consider both kinds of data.

Next, even the most well-designed studies carry a margin of error. "If we’re plus or minus 20 percent I consider that a bull’s eye," says Evan Cohen, vice president of Jupiter data research.

There’s also bias. "Let’s rid ourselves of this mythology that somehow a researcher is completely objective," says Geoffrey Ramsey, statsmaster for eMarketer, a firm that aggregates and analyzes statistics from hundreds of research houses. "It’s like saying that a truly objective journalist exists. We all come to it with a point of view. We’re human."

Like bias, conflict of interest creeps into even the most fastidious studies. Many research firms sell their reports to the same companies they interview for data. "You have to remember research firms have clients and they’re doing research for those clients," says Evans Witt, president of Princeton Survey Research Associates. "Research is done for a reason and you need to be aware of that reason. Find out who paid for the survey."

Researchers also succumb to "publication bias"–the tendency to release reports with favorable results. "There’s nothing wrong with it. That’s their prerogative," says Witt. "You just need to be aware of it."

Such issues have impeded researchers for years. But Net Economy firms have the additional challenge of predicting the future of markets with no past. Old school researchers projecting next year’s demand for crude oil have decades of historical data–and generally straight trend lines–from which to extrapolate. Projections are trickier for firms measuring and projecting online markets, which have less than 10 years of historical data and are growing exponentially. Those hockey-stick curves can’t continue forever; the tough part is figuring out when–not if–they will level out.

Human error often undermines otherwise valid studies. The New York Times recently reported a widely circulated 1985 statistic botched by the compilers. The stat reported that the average recently divorced woman’s standard of living fell 73 percent, while the average recently divorced man’s rose 43 percent. More accurate assessments list the drop for women at 27 percent and the rise for men at 10 percent. The error wasn’t discovered until 1996, after the shocking statistic triggered hundreds of articles, influenced child support and property division laws, and was even cited by the U. S. Supreme Court and President Bill Clinton.

"Just because an analyst says it doesn’t mean it’s true," cautions Sree Sreenivasan, a professor of new media at Columbia University School of Journalism. "But they are more likely to have access to better information than an individual businessman. So you take it. But you’ve got to look at any statistics with a wary eye. And if something sounds off to you, there’s a good chance it is off."

 Return to Web Version