prior probability

Note (1/4): The second half of this post has been significantly revised.

We have been highlighting some of the main ideas contained in Tetlock and Gardner’s 2015 “superforecasting” book in our previous two blog posts. Yesterday, for example, I presented a step-by-step overview of their Bayesian approach to forecasting. In my next two posts, I will restate Tetlock and Gardner’s Bayesian methodology in formal mathematical notation (see image below) and present a concrete example relevant to my domain of expertise (constitutional law).

In particular, I would love to be able to forecast when the Supreme Court of the United States (SCOTUS) is going to overturn a precedent or when it is going to declare a law unconstitutional. In order to engage in Bayesian reasoning in this domain, i.e. in order to make a Bayesian forecast, we will need three pieces of information: (1) the “base rate” or the historical frequency p(H) in which precedents have been overturned in the past or in which federal laws have previously been declared unconstitutional; (2) some piece of evidence E, such as a large number of amicus briefs submitted to SCOTUS, that we are likely to see when a law or precedent is destined to be overturned, or p(E|H); and last but not least, (3) the probability or likelihood of seeing this same piece of evidence, i.e. a large number of amicus briefs, in those cases in which a precedent is upheld, or p(E|not H).

(Before proceeding, you should be asking, Why should the number of amicus briefs be considered a relevant piece of evidence when we are engaged in judicial forecasting? In brief (pun intended), it turns out there is a large scholarly literature attempting to measure the influence of amicus briefs on judicial decision making. I will review this fascinating literature in a separate post. For now, however, the fact that such an amicus practice exists and that there is such a large literature about this practice are, by themselves, relevant pieces of evidence that amicus briefs are important.)

How would we begin to engage in Bayesian reasoning in this domain? First, let’s start with the base rate. Since SCOTUS was created in 1789, it has heard over 28,000 cases! (Today, SCOTUS hears about 80 cases per term, with each Supreme Court Term commencing on the first Monday of October and concluding on the last week of June.) During this 230-year span, SCOTUS has overruled itself 236 times (here is a comprehensive list of “Supreme Court decisions overruled by subsequent decision“) and has found 182 federal laws to be unconstitutional (here is a list of “Acts of Congress held unconstitutional in whole or in part by SCOTUS“). These initial observations thus appear to indicate an extremely low base rate of 0.015 (rounded up), or 236 + 182 = 418 divided by 28,000.

Next, since the use of amicus briefs has become a regular practice only recently, let’s tinker with our historical base rate to make it more accurate. Specifically, let’s consider limiting our evidence to the recent past, say the period from the 2005-06 Term, when John Roberts replaced the late William Rehnquist as Chief Justice of SCOTUS, to the end of the 2017-18 term last year. (By 2005, the practice of amicus briefs in SCOTUS had become quite common.) During this shorter span of time (2005 to 2018), SCOTUS heard about 1000 cases, and by my count, the Roberts Court has decided 13 cases in which it overruled one or more of its precedents and has struck down 26 federal laws, either in whole or in part, or an average of two federal laws struck down per year. (Shout out to Professor Jonathan H. Adler for compiling these data!) These observations give us a revised base rate of 0.04, or 40 divided by 1000–still small, but almost three times as large as our previous base rate.

Notice that the base rate, standing alone, indicates that the reversal of a precedent or a judicial declaration that a law is unconstitutional is an extremely rare event, but its probability is not zero. In fact, the base rate could even be higher, since we should exclude the vast majority of cases in which none of the parties are asking SCOTUS to change the jurisprudential status quo. After all, asking SCOTUS to take such dramatic action as overruling a precedent or striking down a federal law is a long shot. Given this reality of appellate practice, let’s assume that the parties have asked SCOTUS to change the status quo in only 100 of the 1000 cases SCOTUS has heard from 2005 to 2018. (Note: this is just a simplifying assumption on my part. I intend on applying for a research grant to actually count up the number of times since 1789 that a lawyer has requested SCOTUS to change the jurisprudential status quo.)

This simplifying assumption will thus cause our revised base rate to increase up to 40 percent (!), or 0.4 (40 divided 100). This increase makes sense, since we have narrowed down the base rate to the smaller subset of cases in which one of the parties is asking SCOTUS to change the status quo by overruling a precedent or by striking down a federal law. But at the same time, the base rate is too high because it lumps together two types of cases: (1) cases in which a precedent is overruled, and (2) cases in which a federal law is declared unconstitutional. For the remainder of this post, let’s focus on the first type of case only. (Full disclosure: the stability of SCOTUS’s precedents is what motivated to engage in this research in the first place.)

Since there are only 13 Roberts Court cases in which a precedent was overruled, our revised base rate is 0.13, or 13 divided by 100. Now that we have a plausible base rate, we must next proceed to revise or “update” our historical base rate upon the arrival of new evidence. When SCOTUS agrees to hear a new case, for example, many government agencies, interest groups, and other non-parties will have an opportunity to present amicus briefs to the Court. In fact, it turns out that the median or average number of amicus briefs submitted to SCOTUS since 2010 is about 11.75 per case (see here, for example), but we should expect a greater number of amicus briefs the more important the case, i.e. the more likely the case has national or even international implications. In my next post, we will use this information (the number of amicus briefs) to predict the posterior probability that a precedent will be overturned.

Image credit: Kara Lilly, via mawer.com

In our previous post, we summarized some of the main ideas we stole from Tetlock and Gardner’s fascinating book on “superforecasting.” Their book is really useful because it is a kind of “how to” manual for Bayesian thinking. In fact, it is the first book I’ve read that shows you how to “be like Bayes”–how to apply Bayesian methods to real world decision problems. For the benefit of my loyal followers, I will summarize Tetlock and Gardner’s Bayesian methodology below.

Step 1. Formulate a testable problem. The problem must have an answer, and I will henceforth refer to the answer of the problem as “x” because the answer is unknown until the event occurs.

Step 2. Think probabilistically about x, the answer to the problem. Think of x as a prediction or as a “scorable” guess. Your answer may be correct (or incorrect) with some positive probability between 0 and 1. In plain English, “maybe” is usually the most correct answer as to whether x is true or not, but as Tetlock and Gardner note, there are “many grades of maybe” (p. 141) or “units of doubt” (p. 168), so you should not only make a guess about x, you should also observe how strong or weak your belief in x is.

Step 3. Test your guess. How does one go about testing a guess? We will need to subdivide this step into smaller ones as follows:

• Step 3a. Figure out the base rate. The base rate is “how common something is within a broader class.” (Tetlock & Gardner, 2015, pp. 117-118.) So, to think probabilistically about a problem (see step 2 above), ask yourself: “How often do things of this sort happen in situations of this sort?” (Ibid., p. 279.) Congratulations! That is your “prior”!
• Step 3b. Restate the problem in Bayesian terms by asking more questions. In the words of Tetlock & Gardner (p. 111): “What information would allow me to answer the question?” In other words, what evidence do you need to see for x to be true, or how likely is it that x would be true if I saw such-and-such evidence? (I will elaborate on this crucial question-asking procedure in my next post.)
• Step 3c. Make plausible guesses about the correct answers to these other questions in Step 3b and then revise your guesses whenever you obtain new evidence or information that is relevant to any of these questions.

I will elaborate on and express all these steps in more formal Bayesian parlance in my next post.

Happy Birthday Sydjia! (Image credit: Tim Cocks)

That is my favorite quote from “Superforecasting: the art and science of prediction” by Philip Tetlock and Dan Gardner, which I finally got around to reading during this Christmas holiday. Along with Sharon McGrayne’s 2012 book “The theory that would not die” and Nate Silver’s 2015 book “The signal and the noise,” Philip Tetlock and Dan Gardner’s excellent Superforecasting book are must reads for anyone interested in how to improve hypothesis-testing in science or how to improve decision-making in such fields as business, politics, and law. (This trifecta of Bayesian tomes is pictured below.) Below are some of the main ideas I have stolen from Tetlock and Gardner’s beautiful book:

1. Science is not about certainty or facts; science is about probability. Or about “testability,” to be more precise: “… in science, the best evidence that a hypothesis is true is often an experiment designed to prove that the hypothesis is false …. Scientists must be able to answer the question: ‘What would convince me I am wrong?’” (p. 38) By contrast, “Fuzzy thinking can never be proven wrong.” (p. 252)
2. Beliefs should be testable. In the eloquent words of Tetlock and Gardner, “Beliefs are hypotheses to be tested, not treasures to be guarded.” This is why so many legal and political debates are such a supreme waste of time, or in the words of Tetlock and Gardner (p. 268): “These are accomplished people debating pressing issues, but nobody seems to have learned anything beyond how to defend their original position.” In short, beliefs must be tested because even the firmest belief may be wrong.
3. Bayesian methods can be used to test beliefs. As Tetlock and Gardner show, so-called “superforecasters” are just Bayesians by another name. They test their beliefs by restating those beliefs in the form of predictions and they attach a numerical value to express their degrees of belief in their predictions. These Bayesian ideas are so important and so useful in so many different domains, I will devote my next few posts expounding on the Bayesian ideas in Tetlock and Gardner’s book.

To my dearest Sydjia: Happy Birthday Eve!

This video shows how the musician Calvin Harris created his lovely song “Slide”. What will you create in the new year?

Latin Americans like to celebrate Christmas Eve (Nochebuena) with their family and closest friends. That’s why 24 December is my favorite holiday tradition and my favorite day of the year.

It’s time to size up what we have been working on during this past year. In addition to my regular teaching duties, I worked on the following research projects:

1. Along with my colleagues Sean Melvin (Elizabethtown College) and David Orozco (Florida State), I contributed 12 chapters to a new textbook on “Business Law and Strategy.” (McGraw Hill Education, in press). Our textbook presents the standard topics of the business law curriculum–such as torts, contracts, corporate governance, etc.–from a strategic and game-theoretic perspective.

2. I attended the 2018 Fall Altheimer Symposium commemorating the 60th anniversary of Cooper v. Aaron, a leading school desegregation case. My talk explored the legality of President Eisenhower’s decision to send troops to Arkansas during the 1957 Little Rock Crisis. The symposium was hosted by the William H. Bowen School of Law of the University of Arkansas on 28 Sept. 2018 and I very much enjoyed my stay in Little Rock, especially my visit to Central High School. After the symposium, I wrote up a paper titled “Domestic Constitutional Violence” (CDV), which will be published in a symposium edition of the UALR Law Review. (I will be blogging more about my CDV paper soon.)

3. Earlier this year, I wrote up a formal review of Randy Kozel’s excellent book Settled Versus Right: A Theory of Precedent (Cambridge University Press, 2017). My review, which is titled “The Problem with Precedent,” will be published overseas in The Vienna Journal on International Constitutional Law. In the meantime, you can read an ungated version of my review here. (I also blogged about Kozel’s thought-provoking book earlier this year.)

4. During the summer months I also wrote up a law review article in which I present a Bayesian or probabilistic theory of stare decisis. Kozel’s book (see item #3 above) and the U.S. Supreme Court’s 5-4 decision in South Dakota v. Wayfair in June of 2018 are what motivated me to write this paper, which is tentatively titled “Bitcoin, the Commerce Clause, and Bayesian Stare Decisis.” The paper will be published in a symposium edition of the Chapman University Law Review next year, and I will present my “Bayesian Stare Decisis” model of judging at the 21st Annual Federalist Society Faculty Conference, which is scheduled to take place on January 3 and 4 in New Orleans.

5. In the summer I was also scheduled to attend the XVIII International Hemingway Conference at the American University in Paris. Although I was unable to attend, in honor of this occasion I drafted a short essay titled “The Bayesian Man and the Sea.” My paper, which applies Bayesian probability to Ernest Hemingway’s novella The Old Man and the Sea, explores the logic of Santiago’s “decision calculus,” his fateful decision to set sail after going 84 days without catching a single fish, from a probabilistic perspective.

6. In honor of what would have been the philosopher Robert Nozick’s birthday 80th birthday on 16 Nov. 2018, I wrote up an extended review of Nozick’s classic book Anarchy, State, and Utopia (ASU). In particular, beginning in November 2017 and extending into the spring and summer of 2018 (with various interruptions), I eventually ended up blogging about every single page of ASU. Type in the word “Nozick” in this blog’s search function if you wish to peruse any of my takes on Nozick’s masterpiece.

7. I also had the opportunity to talk about my 2013 paper “Gödel’s Loophole” with Professor Brian Frye on his scholarly podcast series “Ipse Dixit.” My Gödel paper, which has been downloaded over 5000 times (!), explores the possibility of a logical contradiction in the U.S. Constitution. It is available here (via SSRN), and you can listen to Prof Frye and I retell the story of Kurt Gödel’s discovery of this contradiction here.

Image credit: kinsta.com

Check out these beautiful mathematical pudding bowls. Each bowl contains the proof of a theorem, such as Gauss’s proof by induction. The proof is in the pudding bowls! (You can order a set of four bowls here. Hat tip: Cliff Pickover.)

Source: http://www.presentindicative.com

That is the subject of this beautiful paper by Kevin Toh, a philosopher of law at University College London. (The full title of his scholarly paper is “Authenticity, Ontology, and Natural History: Some Reflections on Musical and Legal Interpretation.”) In addition, Michael B. Coenen, a law professor at LSU in Baton Rouge, has written an excellent review of Professor Toh’s work. As Professor Coenen correctly notes, Toh is not the first to explore the relation between musical and legal interpretation; past and present preeminent legal scholars such as Jerome Frank (“Words and music: some remarks on statutory interpretation“), Richard Posner (“Bork and Beethoven“), and Jack Balkin (“Verdi’s High C“), among many others, have commented on the connection between music and law. But Toh’s extensive research draws from a wide variety of scholars, including literary theorists, philosophers of art, social psychologists, evolutionary biologists, as well as legal theorists. Below the fold is an extended excerpt from Prof Coenen’s masterful review: Continue reading →

Check out this incredible and creative collection of some of my favorite college graduation caps from this fall’s commencement ceremonies at the University of Central Florida. (All photos by Carly McCarthy, via UCF Today.) #ChargeOn