Note (1/4): This post has been significantly revised.
In our previous post, we painstakingly estimated the base rate or the historical frequency in which a precedent is overturned by the Supreme Court of the United States (SCOTUS) in those cases in which a party is asking SCOTUS to take such an action. In short, since 2005, the year John Roberts was appointed and confirmed as Chief Justice, we estimated a 13% or 0.13 prior probability that SCOTUS will change the “jurisprudential status quo” by overturning a precedent in the smaller subset of cases in which a party is asking SCOTUS to overrule one or more of its precedents.
Now that we have established our base rate, we can forecast how likely it is that SCOTUS will overrule one of its own decisions in any particular case this term. By way of example, let’s consider the aptly-named Gamble v. United States, docket number 17-646, which was heard by SCOTUS on 6 December 2018. As of today (30 December), a decision in this case is still pending. In summary, the attorneys for the petitioner, Mr Terance Gamble, are asking SCOTUS to overrule the “separate sovereigns” exception to the Double Jeopardy Clause, which would require SCOTUS to overrule a line of precedents culminating in Abbate v. United States, 359 U.S. 187 (1959). How likely is it that SCOTUS will overrule this doctrine?
Historically speaking, there is only a 0.13 chance such a dramatic event will occur. Given such a modest base rate, without any additional information we might be tempted to conclude that it is unlikely that SCOTUS will depart from its previous precedents when it decides Gamble. Nevertheless, it turns out that a substantial number of amicus briefs were submitted in this case: 12 in all, or just a shade more than the historical average of 11.75 amicus briefs per case since 2010. Given this new piece of information, we can now use Bayesian reasoning to update our prior! Specifically, the Bayesian approach to forecasting requires us to estimate two sets of conditional probabilities. One is the hit rate or p(E|H): the likelihood of seeing a high number of amicus briefs (with “high” defined as any number greater than the historical average of 11.75 amicus briefs per case) when SCOTUS decides to change the status quo by overturning a precedent or declaring a federal law unconstitutional. The other is the miss rate or p(E|not H): the likelihood of seeing a high number of amicus briefs even when SCOTUS decides to uphold the status quo, i.e. when SCOTUS does not overturn a precedent or does not strike down a federal law.
Next, let’s estimate these two sets of probabilities for the Roberts Court era: 2005 to the present. (Note: I am just going to guess what these probabilities are for now. As I mentioned in my previous post, I intend on applying for a research grant so I can comb SCOTUS’s records and determine what these probabilities are.) Let’s start with p(E|H). Surprisingly, not all cases that end up changing the jurisprudential status quo have generated a high number of amicus briefs. By way of example, there were only three amicus briefs in Montejo v. Louisiana, 557 U.S. 778 (2009), which overruled Michigan v. Jackson, 475 U.S. 625 (1986), and similarly, there were only four amicus briefs in Johnson v. United States, 556 U.S. ___ (2015), which struck down a portion of the federal Armed Career Criminal Act. Nevertheless, most cases that end up departing from the jurisprudential status quo do tend to generate a high number of amicus briefs, so let’s assume that 20 out of the 25 Roberts Court cases that did change the status quo also generated a high number of briefs. (Stated formally, p(E|H) = 0.8.)
But at the same time, we should also expect to see a high number of amicus briefs even when SCOTUS decides to uphold the status quo, so let’s assume that only one of every five cases with high numbers of amicus briefs (i.e. greater than 11.75) will depart from the jurisprudential status quo, i.e. will involve a decision overturning a precedent or striking down a federal law. (Stated formally, p(E|not H) = 0.2.) Now all that remains for us to do is to plug these three probability estimates (base rate, the hit rate, and miss rate) into Bayes’ formula, which is pictured below. In plain English, the updated probability p(H|E) that SCOTUS will change the status quo when it decides Gamble is the hit rate times base rate divided by the hit rate times base rate plus the miss rate times one minus the base rate. When we plug our guesses into Bayes’ formula, we see there is now a 37% probability that SCOTUS will overrule the “separate sovereigns” exception to the Double Jeopardy Clause. Although 37% is still a modest probability, it is almost three times as high than our historical base rate of 13%!
But how can we know whether this forecast is a good one or not? In my next post, I will steal another idea from Tetlock and Gardner’s 2015 excellent “superforecasting” book to answer this question: the idea of the Brier score. The basic idea is this: once we make a large number of SCOTUS forecasts, we will be able to score the overall accuracy of my simple forecasting model.
For any fellow math geeks out there, below are the details of my Bayesian updating:
p(H|E) = [p(E|H) * p(H)] divided by [p(E|H) * p(H) + p(E|not H) * p(not H)]
p(H|E) = [0.8 * 0.13] divided by [0.8 * 0.13 + 0.2 * 0.87]
p(H|E) = 0.104 divided by 0.104 + 0.174
p(H|E) = 0.104 divided by 0.278
p(H|E) = 0.374