Comey and McCabe Audits: How Likely It Was a Coincidence?

The New York Times has reported that the Internal Revenue Service gave one of its most rigorous audit rates to James B. Comey, the former director of the FBI, and Andrew G. McCabe, his former deputy.

This has raised many perfectly reasonable questions, most of them variants of: What are the odds? As the article pointed out, the chances that two high-ranking political enemies of President Donald J. Trump were audited by sheer coincidence are slim.

But minuscule is not zero.

If we wanted to believe this was a coincidence, how unlikely would we say it was? Here, we try to estimate that probability as seriously as we can.

Facts first: Both men were chosen for audits under the National Investigation Program (NRP), a small subset of all the audits the IRS conducts each year. These audits analyze a sample of returns to collect data on tax compliance.

According to the IRS, there were about 5,000 such audits in 2017, 4,000 in 2018 and 8,000 in 2019, chosen from approximately 154 million individual tax returns each year. Mr. Comey’s audit was for his 2017 tax return; Mr. McCabe’s was for his 2019 comeback.

Many aspects of the NRP complicate our calculations, including the IRS auditors’ sampling methodology and the different years of the audits themselves. We will return to these topics later. For now, we will assume that all taxpayers have an equal chance of being audited and that both men were audited in 2017.

If this problem appeared in a probability textbook, it might read like this:

If there are 154 million marbles (the approximate number of tax returns filed each year) in a giant ballot box, and some of them are red (representing Mr. Comey and Mr. McCabe between them), what are the chances that you will draw two or more red marbles if you randomly draw a few thousand from the urn (the number of audits in that year)?

It may sound complicated, but it is a relatively well-studied problem, something that many math or statistics students would encounter in their college courses. People have already derived equations to estimate these probabilities, with names like the hypergeometric distribution, which has applications like election auditing and card counting.

We can simply input our estimates for the total number of marbles, the number of red marbles, and the number of draws, and we’ll get a probability. If we believe that there are only two red marbles, that is, if we limit the exercise to only Mr. McCabe and Mr. Comey: This equation gives a probability of about one in 950 million.

Those are odds considerably higher than your chances of winning the Powerball. It is also an almost meaningless result. At best, it’s the right answer to the wrong question.

Understanding why requires acknowledging an absurdity inherent in our exercise: to better estimate the probability of an unlikely event, we must set aside the fact that we know it has already happened. (The probability of that happening is 100 percent.)

Jordan Ellenberg, a professor at the University of Wisconsin who has written books on math and reasoning, put it this way: “In some counterfactual universe, what is the probability of this happening that has already happened in our universe?”

It may seem strange, but the same problems arise even in probabilistic exercises as basic as tossing a coin.

If you tossed a coin 20 times in a row, your specific sequence of heads and tails is extraordinarily rare, about one in a million, but it happened. Y some flip sequence will always happen. It’s an amazing match only if that’s the sequence you set out to get before flipping.

Likewise, it is wrong to restrict our search to just Mr. Comey and Mr. McCabe, because we would likely be examining these probabilities if we knew that two other Notable political enemies of an administration were audited in place of these two men.

A better question is: What is the probability that two or more people I like it Would Mr. Comey and Mr. McCabe be audited during this period?

Should this group of people include two top FBI officials? Some high-ranking Justice Department official? It is this framework, a subjective rather than factual decision, that drives any probability estimate more than any choice of statistical distribution or sampling weights.

Here is a plot of the probability our equation throws at different options for the number of red marbles, ranging from two (Mr. Comey and Mr. McCabe and no one else) to 400 (a conservative estimate of the number of Americans that Mr. Trump insulted by name on Twitter since he began his run for the presidency).

The probability increases dramatically with the choice of who should be considered a red marble next to Comey and McCabe.

The point is not to decide on a number, but to recognize that our choice of group size is what drives our response. Although some guesses are certainly better than others, many choices are defensible.

Now let’s try to boil something down a bit more realistically, and get back to some of the things we ignored in our simple interpretation of this problem.

First, the two men were not audited for the same year. By expanding our scope to cover the three-year period from 2017 to 2019, our resulting odds increase significantly. This is simple: if a person has a certain probability of being audited in a given year, more years means more opportunities to be audited.

Second, we are only interested in the probability that at least two people are chosen. We will not consider the probability that the same person will be chosen twice; seems unlikely given that audits can drag on for a year, according to Comey’s account. Note that we are looking at the probability that at least two people are selected, not precisely two, since it would also be significant if three or more individuals were chosen from a group.

Finally, the IRS does not select people in a truly random manner. Instead, the agency tends to select some types of taxpayers, including high-income ones, more often than others. For fiscal year 2001, the NRP sample included rebates from people around the 90th percentile of income at about 1.7 times the rate one would expect if rebates were chosen regardless of earnings. That rate skyrocketed through the top income ranges, so that people with incomes in the top 0.5 percent were more than 10 times more likely to be in the sample than someone closer to the median income.

We can probably assume that any group of Trump enemies would gain more than a random sample of Americans. But we cannot realistically estimate the entire income of everyone in our group in each year. We also know that the IRS has considered other factors in its sampling, such as the type of returns taxpayers file, and that sampling methods can change from year to year. This leaves us with little guidance on how to combine the IRS methods. As such, we will leave our estimates unweighted by revenue. As a practical exercise, if you are concerned about how income affects these outcomes, you can double the resulting probability if you think members of a group have very high incomes, and multiply it by 10 if you think you are extraordinarily wealthy.

Incorporating those options, the following table provides some estimated probabilities based on the size of the group being considered..

Alternatively, if our choices are not satisfactory, we create a simple calculator to make its own odds:

So which estimate is “correct”?

The most realistic results of this equation could be accurately described as “very rare” or even “extraordinarily rare,” but neither is proof of irregularities.

“It’s a bit like the irresistible force and the immovable object,” said Andrew Gelman, a professor of statistics and political science at Columbia University, when told about this exercise in the abstract. “On the one hand, you’re saying it’s completely random. On the other hand, you suspect that it is not.

Gelman, like every other statistician who spoke to The Times about this problem, said the biggest hurdle wasn’t the details, but defining the question itself.

When we try to calculate the probability of a given event because we suspect it may not be random, we end up in the tricky position of trying to figure out how we would have predicted the probability of the event prior to it happened, said David Spiegelhalter. He directs the Winton Center for Risk and Evidence Communication at the University of Cambridge, an organization dedicated to improving the way quantitative evidence is used in society.

The math is easy, he said, but formulating the question is tricky, bordering on “nonsense,” in large part because of how hard it is to pin down the group we care about.

“’What is the probability of this happening?’ it’s an easy statement to make,” she said. “It’s a family statement to make. But it’s actually a very difficult question to answer.”

Mathematics has its limits. The point of trying to estimate a probability like this, Gelman said, is not to make too much of the numbers, but rather to let the result prod you to find out more.

In this case, the best question is not one that has an answer you can look up in a statistics textbook.

Instead, Gelman said, the question to ask is, “What’s going on?”

matthew cullen contributed report.

Leave a Comment

Your email address will not be published.