Statistical noise

Dr. Ben Goldacre is correct about statistical noise to a certain degree, but not as exemplified. “Statistical noise” is useful as a sociopolitical tool to motivate politicians and government to do something positive; for example with high unemployment figures being reported by the BBC news media.

The same could be said of the recent riots in England, where social networking was used to bring communities together to clean up effected areas. However with many in the media focusing and reporting in a knee-jerk manner (BBC News 24 coverage over the period of disorder for example), important messages get lost in statistical noise.

We know that their have been about 1,632 arrests, with around 1,002 resulting in prosecution, to date. The Guardian’s editorial reports that 53 million people currently live in England. The latest dataset shows there are 1002 riot related defendants in Magistrates Courts, August 18th, 2011, so far.

Taking the above Magistrates Courts report, broadcast news reports on the number of people in groups, with as many as 400 persons being reported in one place during the disturbances, the number of cities effected… I would estimate that a maximum of about 5,000 people were involved. This figure is guess-work based on provided figures and inferred information from various reports, with a statistical deviation of about 3,000 people.

With a maximum of 5,000 people and the current recorded population figure of 53,000,000 people living in England, one can look at the figures in a positive light. The media could report a story that approximately only 0.5% of the population were involved in the recent disturbances. Or to put it another way, for headline purposes, approximately 99.5% of the population in England were not involved in the recent disturbances!

I have not seen this positive statistical news reported by any of the media outlets. I have seen many stories with people being frightened, feeling disgust and even some on BBC news feeling shame and revolution at being British due to the recent disturbances. These are not positive stories. It is time to report that 99.5% to 99.8% (with a ± factor of 0.3%) of people living in England were not involved in the recent disturbances.

With any statistical analysis noise creeps in. The same can be said with someone’s ability to cover news stories on a particular topic. I may have missed a positive report with many negative reports swamping radio, TV and newspaper pages, however the chances of a positive spin being put by the “British media” is a small deviation; less than 0.01%?

Like with all statistical noise, the real hidden meanings and causes are not straight forward but often complex; we live in a complex society. Dr. Goldacre’s assumption that many numbers “amount to noting more than statistical noise” with “no pattern” is misleading.

We are barraged, every day, with a vast quantity of numerical data, presented with absolute certainty and fetishistic precision. In reality, many of these numbers amount to nothing more than statistical noise, the gentle static fuzz of random variation and sampling error, making figures drift up and down, following no pattern at all, like the changing roll of a dice.

Fuzzy numbers are more than mere statistical noise. While “many numbers” may seem to amount to nothing, one can always find meaning hidden in “junk” information. The recent discoveries on DNA with the human Gnome project should be a window into this world; not all “junk DNA” is junk. “The gentle static fuzz of random variation and sampling error” is not so much static noise, like a ghost from the big-bang (CMBR), but rather an indication that chaos, mathematically at least, is at work.