prior probability

Thus far, we have identified several common forms of “data fraud,” including cherry picking, data dredging, and the false cause fallacy. Yet all of these myriad forms of data fraud might be mere symptoms of a larger problem: publication bias. Just as TV and print media compete to report on the most salient or salacious events that will grab their viewers’ or readers’ attention (“If it bleeds, it leads”), scientific journals likewise compete to publish studies with the most exciting, novel, or “sexy” findings. But the problem with this fetish for novelty or salience is that it generates a scholarly market failure, one resulting in the overproduction of sexy studies, or in the words of the good folks at Geckoboard (a UK-based consulting firm), “For every study that shows statistically significant results, there may have been many similar tests that were inconclusive…. Not knowing how many ‘boring’ studies were filed away impacts our ability to judge the validity of the results we read about. When a company claims a certain activity had a major positive impact on growth, other companies may have tried the same thing without success, so they don’t talk about it.” That is why both the news media and the most prestigious scholarly journals often end up presenting such a distorted picture of reality.

Credit: Franco, et al.

Thus far we have seen the related statistical sins of cherry picking and data dredging. Today, let’s talk about the false cause fallacy (or “false causality” for short), which occurs when you observe two events that appear together and then leap to the conclusion that one event must have caused the other. (Here is a mundane example. The video below presents many more.) In reality, just because two events occur together does not mean that one caused the other. The causation may run in the opposite direction or some unobserved third factor might be the underlying cause of both events or there might be no direct or indirect causation at all!

Let’s proceed with our parade of fraudulent data practices, shall we? Next up is data dredging (a/k/a “p-hacking”), a more sophisticated (and less transparent) form of cherry picking. In the words of Wikipedia: “The process of data dredging involves automatically testing huge numbers of hypotheses about a single data set by exhaustively searching … for combinations of variables that might show a correlation ….” This form of data fraud thus occurs when researchers perform multiple statistical tests on a single set of data and then selectively publish only those results that satisfy some test of statistical significance. Such ex post results, however, are often just spurious correlations. The lesson here is this: beware of so-called “statistically significant” results. To avoid perpetrating this form of data fraud (and reduce positive-results bias to boot), some journals and funding organizations are now requiring researchers to preregister their clinical trials, stating in advance what hypotheses they are going to be testing.

We presented a collection of fraudulent data practices in our previous post. Now, let’s consider each fraudulent technique in turn, beginning with the “Texas sharpshooter fallacy” or cherry picking: the practice of selecting results that fit your claim and excluding those that don’t. According to the good folks at Geckoboard (a London-based consulting firm), this practice is “[t]he worst and most harmful example of being dishonest with data. When making a case, …. people often only highlight data that backs their case, rather than the entire body of results. It’s prevalent in public debate and politics, where two sides can both present data that backs their position. Cherry picking can be deliberate or accidental. Commonly, when you’re receiving data second hand, there’s an opportunity for someone choosing what data to share to distort the truth to whatever opinion they’re peddling. When on the receiving end of data, it’s important to ask yourself: ‘What am I not being told?’”

That is the title of this well-reasoned essay by Joanna J. Bryson, a computer science professor at the University of Bath. (Also, check out her TEDxCERN Talk below on the question, “Is A.I. changing us?”) Here is an extended excerpt from Dr Bryson’s excellent essay:

There is in fact no question about whether we own robots. We design, manufacture, own, and operate robots. They are entirely our responsibility. We determine their goals and behaviour, either directly or indirectly through specifying their intelligence, or even more indirectly by specifying how they acquire their own intelligence. But at the end of every indirection lies the fact that there would be no robots on this planet if it weren’t for deliberate human decisions to create them.

The principal question is whether robots should be considered strictly as servants—as objects subordinate to our own goals that are built with the intention of improving our lives. Others in this volume argue that artificial companions should play roles more often reserved for a friend or peer. My argument is this: given the inevitability of our ownership of robots, neglecting that they are essentially in our service would be unhealthy and inefficient. More importantly, it invites inappropriate decisions such as misassignations of responsibility or misappropriations of resources.

Shout out to Armin Chosnama, who introduced us to Tyler Vigen’s wonderful spurious correlations website. Below is just a small subsample of over 30,000 such correlations. File under “bullshit statistics.”

Did you know that landlocked Oklahoma is bordered by six separate States? The hand-painted map of Oklahoma pictured below, which was drawn by artist and fellow polymath Jerry M. Wilson, is the first installment of a series. Check out more of his beautiful hand-painted maps via Reddit here and his homepage via Tumblr here.

Image credit: Jerry M. Wilson (u/ironandredwood)

Inspired by Leonard Bogdonoff‘s project to scrape Instagram to create a universal and searchable database of street art from around the world, we conclude our series on Cuban graffiti artists by including the following non-Instagram compendia of Havana street art:

1. Brendan Sainsbury’s list via Lonely Planet (five images).
2. Sarah Laskow’s list via Atlas Obscura (seven images)
3. Barbara Maseda’s list via culturetrip (ten images).
4. Public Art Havana via the Public Art App (ten images).
5. Havana street art walk via triposo (14 images).
6. Cuban street art via Pinterest (55 images).

Take a look before this art disappears. All of Havana’s street artists are now threatened by a new censorship law called “Decree 349” …

Photo credit: Karim Amar

Did you know that March 25 used to mark the beginning of the new year in Britain and in her North American colonies? Rebecca Onion explains why in The Boston Globe. Here is an excerpt from her fascinating essay:

This March 25 will likely pass quietly, another chilly Tuesday in early spring. But in the Boston of 300 years ago, the day would have been very noteworthy indeed: It marked the start of the new calendar year. The Colonists, as Britain had for centuries, celebrated the change of the year in late March—the Feast of the Annunciation, or Lady Day. Rents were due, contracts began, and obligations renewed on March 25, the “New Year.” *** Under the Julian system, New Year’s tended to vary from country to country; Britain preferred to mark it on March 25, a Christian holiday, rather than the original Roman New Year’s Day of Jan. 1. (March 25 was traditionally the date when Mary found out about her pregnancy, and familiarly known as “Lady Day.”) So through the 1600s and into the 1700s, continental Europeans had agreed to turn their calendars over on Jan. 1, but the British world continued to mark the beginning of a civil and legal year three months later.