prior probability

The problem of priors

Posted on June 28, 2014 by F. E. Guerra-Pujol

Note: this is the fifth and final installment of our review of the paper “Bayesian reasoning in science” by Colin Howson and Peter Urbach.

We now come to the “main event”: the problem of priors.

That is, where are you supposed to get your Bayesian priors from? (To return to the coin-toss experiment we discussed in our previous post, for example, what prior should the experimenter assign to the probability of the coin being fair?) Aren’t all persons’ priors ultimately subjective and thus non-scientific, for as Howson and Urbach acknowledge on p. 374 of their paper: “at some point … prior probabilities will have to be used which merely reflect [subjective] opinion.”

In the last part of their paper, Howson and Urbach describe several valiant attempts to generate “explicit, ‘objective’ rules for calculating priors” yet end up conceding that “there seems to be no way of ‘objectively’ defining prior probabilities.” So, what is to be done, then? How do Howson and Urbach, in particular, deal with the problem of priors?

In brief, the authors argue that subjectivity is good, that subjectivity is universal, and that subjectivity is irrelevant.

First, they argue that the subjective nature of Bayesian priors is a strength, not a weakness: “… our argument has all along been that this is really no weakness: it allows expert opinion due weight …” [Time out #1: what if the “experts” themselves disagree with each other? Whose prior wins out?]

Next, they take a direct swipe at Fisher and his sundry disciples, arguing that all science is inherently subjective [time out #2: really?] and that Bayesian methods are at least open and honest about their subjectivity. In the pull-no-punches words of Howson and Urbach:

[A subjective prior] is a candid admission of the personal element which is there in all scientific work. The inventors of ‘objective’ methodologies [ i.e. Fisher and his ilk] … merely sweep the personal element under the carpet.

Lastly, they argue that the subjective nature of Bayesian priors is irrelevant, since persons with different prior beliefs should converge in their posterior beliefs as data and evidence accumulate (assuming those persons are good Bayesians, of course). [Time out #3: so why do we see so little convergence in so many different domains, such as politics, economics, and philosophy?]

Are you persuaded by any of these arguments? Are they consistent with each other? Let us know what you think …

Keep calm and update your priors.

Posted in Uncategorized | Tagged Bayesian reasoning, priors | Leave a comment

Lies, damned lies, and … statistics

Posted on June 27, 2014 by F. E. Guerra-Pujol

Note: this is the fourth part of our review of the paper “Bayesian reasoning in science” by Colin Howson and Peter Urbach. (The fifth and final installment of our review shall appear on 28 June.)

Let us return to Howson and Urbach’s Bayesian paper today. After presenting Bayes’s rule and the Bayesian approach to truth on pp. 371-372 of their paper (see our Bayesian blog posts of 25-26 June for reference), Howson and Urbach concede that the Bayesian approach to truth “has been widely criticized because it is based on personal, hence subjective, probabilities [cf. the problem of priors we talked about in our post of 26 June titled “Beliefs are like gambles”]. Scientific inference, critics say, should be perfectly objective.” Howson and Urbach thus spend the rest of their paper comparing and contrasting the Bayesian approach to truth with its leading challenger, what they refer to as the “classical statistical inference” model, an alternative approach to truth associated with the work of such giants as R. A. Fisher, Jerzy Neyman, and Egon Pearson (all of whom Howson & Urbach lump together as “classical statisticians”).

In brief, Howson and Urbach begin the second part of their paper by noting that the “classical” or non-Bayesian approach to truth “has two principal parts, the first relating to the testing of hypothesis (using significance tests) and the second to estimating the values of unknown parameters.” (In this post, we shall focus on Howson and Urbach’s critique of Fisherian hypothesis testing and the related idea of “significance”.) The authors then take a simple example to illustrate the Fisherian approach: an experimenter tossing a coin 20 times and counting the number of times the coin lands “heads” in order to test whether the coin is fair or not. “There are 21 possibilities,” they write, “ranging from no heads and 20 tails to 20 heads and no tails.” But how does the experimenter in this simple example know whether the coin is fair, i.e. how does he actually “test” his hypothesis in this case? If he is a Fisherian, he must perform a secondary “significance test”; that is, he must now proceed to “test” his results from the 20 previous coin tosses (though not the coin itself).

You will find the splendid details of Howson and Urbach’s critique of significance testing on pp. 372-373 of their paper, but their main point, as we understand it, is this: whether the experimenter’s coin-toss results in the example above are “significant” in a statistical sense at some predetermined level (such as 0.05) tells us nothing about the actual coin being tested! Why? Because a significance test is not a direct test of truth; it is simply a secondary or subsidiary test of one’s experimental data. (By way of analogy, consider the difference between a historical or legal investigation into the actual contents of a document versus an investigation of the way in which that document was made.) There is thus no necessary or logical relation between the “significance” of a given statistical test and the truth of the hypothesis being tested.

Worse yet, Howson and Urbach note that significance results are easy to manipulate and are super-sensitive to experimental design. In particular, they present this additional critique of significance testing on p. 373 of their paper — the stopping-rule problem:

In our earlier example, it was assumed … that because the coin was tossed 20 times, all of the possible outcomes would exhibit [some combination of] 20 heads and/or tails. But these are the possible outcomes only if the experimenter has a premeditated plan to throw the coin 20 times. Had the plan been to stop the experiment when, say six heads appeared, he could have got just the result he did, but with a different list of unrealized, possible outcomes.

So what? Here’s what:

Because significance is calculated by reference to these [unrealized, possible] outcomes, a result could be significant if the experimenter had had one plan (or stopping rule in mind), but not significant if it was another.

In short, in the eloquent words of Howson and Urbach: “This dependence of significance tests … on the subjective, possibly unconscious intentions of the experimenter is an astonishing thing to discover at the heart of supposedly objective methodologies. It is also a most inappropriate thing to find any methodology, for the plausibility, or cognitive value, of a hypothesis … should not depend on the experimenter’s mind.” (Ouch!)

But hold on in a minute … what about the problem of subjective priors (which we noted in our post “Beliefs are like gambles” below)? Does the subjective Bayesian approach to truth fare any better than standard Fisherian methods? Stay tuned …

Posted in Uncategorized | Tagged Bayesian reasoning, Statistics, Truth | Leave a comment

Did Kurt Gödel really discover a loophole in the Constitution?

Posted on June 26, 2014 by F. E. Guerra-Pujol

Our 2013 paper Gödel’s loophole considers two related questions: why have so few scholars taken Gödel’s alleged discovery seriously, and what was this possible logical contradiction in the Constitution? (Hint: it probably has to do something with recursion.) There is also some recent discussion of our thesis at Hacker News (Y Combinator) here. (Note: We will return to our review of “Bayesian reasoning in science” in our next post.)

Posted in Uncategorized | Tagged Kurt Gödel, Law, Recursion | Leave a comment

Beliefs are like gambles …

Posted on June 26, 2014 by F. E. Guerra-Pujol

Note: this is the third installment of our review of the paper “Bayesian reasoning in science” by Colin Howson and Peter Urbach.

Thus far, we have seen how Howson and Urbach briefly consider the relation between probability and truth (see previous blog post), and we also compared and contrasted ordinary gambling odds, like the familiar system of odds used in sports betting, with the more elegant probability scale (ranging from 0 to 1, inclusive) used in probability theory. In the next part of their paper (pp. 371-372), Howson and Urbach summarize the seminal contributions of two important probability theorists — Frank Ramsey and Bruno de Finetti — and then derive Bayes’ theorem from the standard axioms of probability theory for good measure. We won’t rehash all the technical mathematical details of Bayes’s rule here, or of Ramsey and de Finetti’s theoretical work either. Instead, we shall summarize (in words) two fundamental qualitative insights from this part of their paper:

Insight #1: beliefs are like gambles — The task of assigning a probability value from 0 to 1 to a future event (or to the truth value of a hypothesis) is ultimately based on subjective personal beliefs and can thus be quantified by a wager, the same way sports bettors wager on the outcome of sporting events. Why? Because a person’s degree of belief in something, although entirely subjective and personal, can be measured objectively or “translated” (so to speak) by the amount of money he is actually willing to bet on his beliefs. (For those of you keeping “intellectual score” at home, we owe this important insight to Frank Ramsey and Bruno de Finetti.)

Insight #2: gambles must be updated — What is Bayes’s rule really all about? Stated informally, it’s ultimately about “updating” (as we like to say) your gambles, i.e. revising your subjective prior beliefs regarding the truth value of a given hypothesis h (cf. insight #1 above) after you are able to review some amount of evidence e relevant to h. That is, would you be willing to bet more money, or less money, on your beliefs after evaluating e? The evidence may consist of an empirical test of h (as in science), or testimony from a witness (as in law), or a scouting report (as in sports). Whatever the case might be, a good Bayesian should assign some weight to e and update his priors in light of e. (Thank you, Rev. Bayes!)

But now we must contend with a new and perhaps insurmountable problem posed by the Bayesian approach to truth (in addition to the pesky problem we mentioned in our previous post regarding the subjectivity of Bayesian methods): where do you get your priors from? It is no exaggeration to say that this question re: priors has stirred up the most controversy in scientific and philosophical circles. (Note the motto of this blog.) For their part, Howson and Urbach devote most of their paper responding to this question (and the related problem re: subjectivity in science), so rest assured, we shall continue our review of their paper “Bayesian reasoning in science” in future posts …

Don’t we all!

Posted in Uncategorized | Tagged Bayesian reasoning | Leave a comment

Truth and probability

Posted on June 25, 2014 by F. E. Guerra-Pujol

Note: this is the second installment of our review of the paper “Bayesian reasoning in science” by Colin Howson and Peter Urbach.

Following their short introduction on gambling odds (see post below for a summary), Howson and Urbach present the basic laws or “axioms” of probability theory. (You can read more about the laws of probability here. See also the formal paper by Russian mathematician S. S. Vallander below.) More importantly, they note the relation between probability and truth: “Suppose h is some scientific hypothesis,” they write. “Experimental data can never conclusively prove that h is true, even if it is true. [A reference to Karl Popper would have been nice here.] So you are never absolutely certain of h’s truth, only more or less. The inductive inference [therefore] consists in assessing the degree of certainty warranted by the evidence.” [Emphasis added by us.]

We quote Mssrs Howson and Urbach at length here because we believe that these three crisp sentences not only reveal an important insight about the underlying nature of hypothesis testing in science; the Bayesian approach to truth also tells us a lot about litigation and the legal process generally (of which we shall have much more to say in future blog posts). In short, ultimate truth is not an “all or nothing” affair like religion or politics. Truth is more like a horse race or the World Cup (or “Copa do Mundo” for those of you in Brazil and Portugal) — for she is subject to the same vagaries of uncertainty as the outcome of a horse race or a football tournament.

For now, however, notice how this Bayesian view of truth poses a potential paradox. Again, in the words of Howson and Urbach, “if the probability of a hypothesis merely reflects our own personal degree of belief in h, how can an objective logic of inductive inference be based on such probabilities?” A good portion of Howson & Urbach’s paper is devoted to this fundamental question, so “stay tuned” … we shall continue our review in future posts …

Posted in Uncategorized | Tagged Bayesian reasoning, Probability, Truth | Leave a comment

“Bayesian reasoning in science”

Posted on June 25, 2014 by F. E. Guerra-Pujol

That is the title of this commentary by Colin Howson and Peter Urbach published in the journal Nature on 4 April 1991. (Howson and Urbach also published a book with the title “Scientific Reasoning: The Bayesian Approach”; see the image of their book cover below.) Their dense 1991 paper offers a concise overview of Bayesian methods, provides a powerful critique of alternative statistical methods, and has shaped our work as well (in which we apply Bayesian methods to litigation). We shall thus review the main points and insights of “Bayesian reasoning in science” in this and the next few blog posts.

Let’s begin at the beginning, shall we? Howson and Urbach start out by acknowledging that “ours is uncertain world” and by noting how gamblers use odds to measure numerically the likely outcomes of future events. (This method of expressing probabilities is especially common in sports betting. For example, prior to the running of this year’s Kentucky Derby, the odds that California Chrome would win the race were 5 to 2, meaning that a $2 wager on this horse finishing in first place would pay out a total of $7 in winnings–i.e. a profit of $5, plus the bettor’s original $2 wager.) The authors also identify a major problem with this familiar system of gambling odds: “Because odds are ratios the odds scale starts at 0 and is unbounded to the right (infinite odds).” The solution to this problem is to transform the odds scale into a finite probability scale (ranging from 0 to 1) by restating the probability of an event p using the formula p = odds divided by 1 + odds. In other words, we want to express probabilities using a uniform and finite space (i.e. from 0 to 1) in order to make probability problems tractable and easier to solve. “Stay tuned” … for we shall review the remaining parts of this important paper over the next few days.

Posted in Uncategorized | Tagged Bayesian reasoning | 2 Comments

Science bounty hunters?

Posted on June 24, 2014 by F. E. Guerra-Pujol

Dr Christopher Keating says via his global warming website that he is willing to pay a bounty or reward of $10,000 USD to anyone who is able to disprove global warming. The official rules of this contest are thus:

1. Dr Keating will award $10,000 USD to anyone who can prove, via the scientific method, that man-made global climate change is not occurring;

2. There is no entry fee;

3. You must be 18 years old or older to enter;

4. Entries do not have to be original, they only need to be first;

5. Dr Keating is the final judge of all entries, but he will provide comments on why any entry fails to prove the point.

Time out … Does anyone see any problems with Rule #5?

Incentives matter!

Posted in Uncategorized | Tagged Bounties, Science | Leave a comment

How many World Cup matches are fixed?

Posted on June 23, 2014 by F. E. Guerra-Pujol

Forty-eight matches will played during the first stage of the World Cup in the nation of Brasil. But how many, pray tell, of those “first 48” matches will be fixed or rigged in some form, e.g. by means of payoffs to players, trainers, or refs? (By the way, we don’t mean to pick on FIFA; after all, the same question can be asked about the NBA, the NCAA, and other sports leagues.) For our part, we would like to believe that the number is very close to zero (since the glory of advancing to the next stage is “priceless”), but we are not so sure about the pre-World Cup qualifying rounds. Addendum: see our post for 1 July.