We recently discovered Ron Allen and Mike Pardo’s new paper on “Relative plausibility and its critics” via Larry Solum’s Legal Theory Blog. Because of our fascination with all things Bayesian and with the role of probability theory in legal trials, this 71-page, single-spaced paper is a must-read for us. In their paper, Allen (pictured below, left) and Pardo (right) criticize probabilistic theories of juridical proof and then present an alternative “relative plausibility” theory of legal proof. In this post, we will summarize Allen and Pardo’s main criticisms of probability theory in law:
1. The reference class problem. According to Allen and Pardo, an objective theory of legal probability is a non-starter because of the reference class problem. In brief, the reference class problem refers to the difficulty of deciding what class or group of events to use when calculating the frequencies or objective probabilities applicable to a particular event. This predicament is unavoidable because any event can be a member of a wide variety of classes or groups, and the frequency of the event may differ significantly depending on how the class or group is defined. (By way of example, to estimate the probability of an aviation disaster, one might use the frequency of crashes of all aircraft, the frequency of crashes of a particular model or type of aircraft, or the frequency of crashes of aircraft flown by x firm in the last five, ten, or y years.) We will delve into the reference class problem in a future reply post. For now, suffice it to say that we agree with Allen and Pardo on this score.
2. The subjective probability problem. Allen and Pardo are also highly skeptical of subjective theories of probability, such as those advanced by our intellectual heroes Frank Ramsey and Bruno de Finetti, because a judge or juror’s subjective probabilities “need not be constrained in any way by the quality of the evidence [presented] at trial” and thus have “no necessary relationship to advancing accurate outcomes” (p. 10). But how accurate (let alone fair) is the existing system of “binary verdicts” in which a defendant is either “guilty” or “not guilty”? All methods of proof must cope with uncertainty (contested facts, untruthful witnesses, etc.) and are thus prone to error, so we will push back against Allen and Pardo’s critique of subjective probability in a future post.
3. The theory-versus-practice problem. Allen and Pardo also emphasize the chasm between probabilistic theory and actual legal practice. In their words (p. 10): “the probabilistic account (in both objective and subjective forms) is inconsistent with how [judges and jurors] process and reason with evidence, which occurs holistically and not in the item-by-item fashion envisioned by [probability theory].” That said, the irony of this critique does not escape us, especially considering the prevalence of subjective probability in all walks of life, including law. We will thus address the theory-versus-practice problem in a future post.
4. The conjunction problem. This problem occurs because as Allen and Pardo rightly note (p. 11), “legal doctrine and jury instructions typically dictate that the standard of proof applies to the individual elements [of a case], not to cases as a whole.” (Put aside, for now, the possible contradiction between this observation and Critique #3 above.) So, in a civil case consisting of two legal elements, if the plaintiff proves each to 0.6, most commentators would agree that the plaintiff should win under the preponderance of the evidence or “more likely than not” standard. But if these two elements are independent of each other, then the probability of plaintiff’s claim is only 0.36, via the multiplication rule of probability for independent events, thus falling well below the preponderance standard. For what it’s worth, the conjunction problem appears in other settings as well, so we will delve into this dilemma in a future post.
5. The unknown probability problem. Allen and Pardo present the following stylized example to illustrate this problem (p. 12): “suppose the plaintiff presents a version of the facts that is 0.4 probable, [while] the defendant presents an alternative version of the facts that is 0.2 probable.” Should the plaintiff win (under a preponderance standard) because its version is twice as likely to be true as the alternative, or should the defendant win because the plaintiff’s version is still below the preponderance threshold of 0.5? Since probabilities must add up to 1, in this example what should we do with other 0.4 probability? Stay tuned. We will address all five of these objections (and assess Allen and Pardo’s alternative “relative plausibility” theory of evidence) next week …