Konstantin and I have a recurring conversation. One of us says "Wait! I think we're wrong about the evidence for Planet Nine. Here's why..." and proceeds to spin a hypothesis about how we could have been misled by the data or that we are predicting something that is not happening, and then we will carefully go through the entire Planet Nine analysis over again before convincing ourselves that, no, all is normal in the outer solar system (where by normal we mean that Planet Nine is happily orbiting the sun waiting to be seen). So far, we have not found a reason to discard the Planet Nine hypothesis. Will one some day? Perhaps. If the evidence for the existence of Planet Nine were to unravel we would be sad, but we would have to give up the idea. We have been prepared for that moment since the day we first proposed Planet Nine.
Has the moment now come? A new paper by Napier et al. appeared on the archive the other day with the fairly conclusive title of No Evidence for Orbital Clustering in the Extreme Trans-Neptunian Objects, which, well, sounds pretty bad for Planet Nine. There was a quick flurry of requests for comments on the paper, but it seemed like it was better to carefully read and go over the analysis than to simply glibly comment. It's a substantial paper with a lot buried in it, so it took a while, but I think I have now fully digested the paper and can comment on what is going on.
The central idea of the paper is pretty simple: do a careful analysis of where three separate surveys pointed their telescopes and use that analysis to examine whether or not their is any clear sign of orbital clustering in the outer solar system. The specific method is different, but the paper follows the general prescription of our paper from 2019 in trying to determine if the orbits of distant KBOs are aligned in the same general direction and if the orbits are tilted in the same direction. These effects are precisely what is predicted by the Planet Nine hypothesis. In our 2019 paper, we analyzed the 14 objects known as of that time that have semimajor axis (average distance from the sun) beyond 230 AU. We found that we could rule out a non-clustered outer solar system at the 99.8% confidence level. Pretty good right?
Our 2019 paper performed one other critical analysis (this will matter below!). We showed that the 4 objects found by the OSSOS survey were too few to be able to detect the clustering. If you think of the measure of clustering as a single number (which it isn't; it's a 4-dimensional point, but we'll pretend), you could say that we measured a clustering of 4+/-1, for example. But the OSSOS survey could not detect clustering because they measured 0+/6. The uncertainty on their measurement was larger that the expected effect. All was happy in Planet Nine land.
OK, so back to the Napier et al. analysis. They look at 14 objects that have been discovered since our original 2015 paper. After much work they conclude that they can rule out a non-clustered outer solar system at only the 83% confidence level. Bad news for Planet Nine! Taken at face value it seems that the evidence for clustering has weakened or gone away entirely.
Statistically, it is weird that a 99.8% significant result would fade so quickly with just a little new data. Interestingly, when Napier et al. looked only at the objects which are identical between their analysis and our analysis, they see clustering at the 99.5% significance level in excellent agreement with our original analysis, so we all agree that those 2019 results appear on firm ground. So: what is up with the 7 new objects in their analysis that suddenly drops the significance? Let's find out who they are and what surveys they came from.
First, let's look at the measure of clustering from our 2019 paper:
What you are looking at are measures of two parameters for each of the 14 distant objects in our analysis. x,y shows, basically, the direction that the orbit points (with some additional complications). p,q shows the direction that the orbit tilts (with some additional complications). As you can see, the orbits of 11 of the 14 objects point towards one quadrant and, on average, they are tilted in a common direction. Napier et al. followed our prescription here, so you can see their versions of these same plots in their Figure 4. The big red dots are the average position of the black dots and show the strength of the clustering. (To complete the analysis we must determine if, when observational biases are included, the red dots are far enough away from zero [unclustered] to be significant; they are, at the 99.8% confidence level).
Let's add the new Napier et al. objects to these plots:
The green points are from the spatially concentrated (thus highly biased) DES survey, the purple from the much broader survey of Sheppard and Trujillo (there is one additional object that Napier et al. include that was never reported to the Minor Planet Center; we restrict our analysis to objects whose detection history we can track; that one unreported object is down in the lower left of the lefthand plot). According to Napier et al., the inclusion of these points which are clearly consistent with the previous measurement of clustering into the analysis makes the evidence for clustering inconclusive. WHAT?
It took my a long time to understand this strange-looking behavior (again, instant commentary is not always the best commentary). But I think I get it. I believe it is all about the measurement uncertainty that we talked about above. Our paper says that the clustering statistic is 4+/-1. Their paper says the clustering statistic is consistent with zero. But I believe that their clustering statistics is something more like 4+/-6. Thus consistent with zero, as they correctly claim, but not good enough to detect the clustering that exists. Do I know this for sure? No, because they did not publish their uncertainties (an interesting question would be how did whoever refereed the paper let them get away without publishing their uncertainties!).
OK, but wait. They have about the same number of objects in their analysis as we did in our 2019 analysis, so why would our uncertainties be smaller? I think that the answer is primarily due to the fact that DES, which only looked in one direction in the sky, happened to look right in the direction of the clustering. Why? Dumb luck. Their survey was designed long before there was an inkling of Planet Nine. And the Planet Nine clustering is independent of any DES data.
Why is that a problem, though? Imagine a situation where I look around for a few nights and notice that the sun is setting in the west the modest number of times I happened to notice the sunset. You then decide to studiously look west. You see sunsets, but only in the west. You have performed a very biased survey, so when you do your statistically correctly you state that you cannot confirm that the sun sets exclusively in the west because your large survey would only see western sunsets thus the direction of the setting sun is statistically conclusive with being in all directions. It seems weird, but adding in highly biased data that is biased in the precise direction as the signal for which you are looking makes it harder to confirm the signal in the first place. But there is a solution. That solution? Publish your uncertainties.
Napier et al. get it right, in the end:
Ah ha! Sadly, they don't check for consistency with the previously measured clustering, or they would see that, indeed, the ETNOs were already known to be clustered precisely where DES has looked.
I think this solves the mystery of how adding in objects which appear quite clustered makes the significance of your clustering appear to go away.
I think that the right conclusion is that the highly-biased DES data is consistent with the previous measurements of clustering, but that the bias from DES is strong enough that we should probably not be surprised by this. In the end, the previously measured clustering from our 2019 paper is still valid (and has actual uncertainties published), and the conclusions of that paper remain. The clustering of distant Kuiper belt objects is highly significant. It's hard to imagine a process other than Planet Nine that could make these patterns. The search continues.