A public interest group based in Las Vegas — the Coalition for the Protection of Marriage — recently filed a petition alleging the non-random assignment of judges in a subset of same-sex marriage cases decided by the Ninth Circuit Court of Appeals in San Francisco. (In federal cases, appeals on the merits are decided by three-judge panels and judges are supposed to be assigned randomly to these panels.) When we first heard of these allegations, we were intrigued by the possibility of “panel packing” or non-randomness in the assignment of judges, since we suspect that the assignment of judges to high-profile cases is anything but random. But, a few days ago, when we finally read the statistical study on which these allegations are based, we detected a possible reference class problem. (For the record, the report was prepared by Dr James H. Matis, a retired statistics professor.)

In summary, Dr Matis’s study singles out two federal judges in particular, Judge Marsha S. Berzon and Judge Stephen Reinhardt, noting that of the 11 same-sex marriage cases decided by three-judge panels from the Ninth Circuit since Jan. 1, 2010, Judge Berzon was assigned to five of the panels, while Judge Reinhardt sat on a total of four panels. Thus, on a bench consisting of 29 judges, the odds of the same two judges being assigned to that many panels on a particular legal issue (same-sex marriage) are 441-to-1. Dr Matis’s statistical study, however, appears to be fundamentally flawed due to the reference class problem.* As the lawyers representing Beverly Sevcik (the plaintiff in the Nevada same-sex marriage case) explain in their opposition to the petition:

“In order to obtain [his] desired statistical results, [Dr Matis] has narrowly defined the parameters of [his] search—to only cases decided after 2009 involving the federal constitutional rights of lesbian and gay people. But that federal constitutional limitation on subject matter is arbitrary * * * The reality is that Ninth Circuit jurisprudence implicating the rights of lesbian and gay people is far more expansive than the meager 11 cases that [Dr Matis] has cherry-picked. There has been a litany of appeals implicating the rights of lesbian and gay people in which neither accused judge was assigned to the panel. This includes appeals related to immigration, military service, public and private employment, schools, prisons, jury selection, free speech, privacy, and even marriage. The addition of these cases, which are far from exhaustive, more than quadruples the number of purportedly relevant cases identified by [Dr Matis]. It is hardly surprising that in some 50-odd cases or more [** see Addendum #2 below], the same judges would be assigned across at least a handful of panels (pp. 12-14, footnotes omitted).”

But this critique of Dr Matis’s report begs an important question: how should we define the reference class of equality cases in order to measure whether the assignment of judges to these cases is random or not?** Also, how should we define “random” in light of this reference class problem? Any takers?

*Addendum (Nov. 20): Let’s expand the reference class to include the “50-odd cases or more” involving gay rights in one form or another. We still have to determine whether Judge Berzon or Judge Reinhardt sat on any of the panels in this new and expanded reference class of cases. If so, Dr Matis’s  may be right, after all! We will be looking into this and reporting back soon …

**Addenudum #2 (Nov. 26): We just noticed that the lawyers representing Beverly Sevcik cited an additional 36 federal cases from the Ninth Circuit involving gay rights, which would increase the size of the relevant reference class from 11 cases to 47 cases in total. We still have to look up whether Berzon or Reinhardt sat on any of the panels in these additional 36 cases. For now, though, let’s assume that they did not. Let’s also assume (for mathematical convenience) that there are 30 (instead of 29) judges on the Ninth Circuit.

Given these simplifying assumptions, we must now conclude that Dr Matis’s allegation of panel packing on the Ninth Circuit is wrong. Since each case is decided by a panel of judges, and since there are three judges per panel, all other things being equal we would expect each judge on the Ninth Circuit to appear on at least one out of ten panels on average. Working with our expanded reference class of 47 cases (i.e. 47 separate panels), Judge Berzon’s assignment to five of those cases and Judge Reinhardt’s role in four of those cases does not raise any statistical red flags …