In today’s paper, Vickers and colleagues argue for a method to convey confidence level in communicating evidence of extraterrestrial life, noting that “inconceived alternatives” (i.e., unexplored false positives) have always reperesented an important issue, both in past and present discoveries.

With new technological advances, such as NASA’s Perseverance or the James Webb Space Telescope, the problem of communicating to the public about evidence of potential extraterrestrial life becomes more pressing. In this paper, Vickers and colleagues investigate the issue of confidence levels regarding various claims that can be made concerning such evidence.

While much work has been done on understanding possible biosignatures, assessing uncertainty has so far only been briefly addressed in the literature. False positives always remain a possibility for instance, and characterizing these alternative explanations is a task of prime importance in investigating evidence of extraterrestrial life.

Furthermore, the authors describe several “cautionary tales” taken from the history of astrobiology where claims have been made about signs of life, and which were later contested by subsequent discoveries. One first example is that of the accumulation of oxygen in the atmosphere of a planet: we now know of several abiotic processes that can explain this phenomenon, but the fact that such a discovery could have taken place before these “unconceived” processes were known should give us pause.

Several similar situations have taken place in the history of astrobiology. Vickers et al. mention Sinton’s interpretation in 1957 of a curious martian absorption spectrum which would have “very likely” indicated organic molecules or “vegetation”. Several years later, Shirk and colleagues described an abiotic explanation of the phenomena, and soon after it was discovered that this “biosignature” was in fact due to deuterium in Earth’s atmosphere.

More recent examples include the martian meteorite ALH84001, which was once thought to display evidence of past microbial life on Mars (1996)—a claim that was dismantled in 2022—and that of the existence of stromatolites in Greenland in 2016, which would have extended the fossil record by 200 million years, for which an abiotic explanation was found in 2018. The fact that these abiotic explanations were first described after the evidence was first discovered is precisely what Vickers and colleagues have in mind when talking about “unconceived alternatives”. This should thus serve as a cautionary tale when considering current biosignatures, such as strong oxygen-methane disequilibrium in the atmosphere, for which we have no alternative abiotic explanations given current knowledge.

One key concept in judging how excited we should be about these biosignatures is, according to Vickers and colleagues, the extent to which we have explored the possibility space of false positives. But given the fact that astrobiology is still a young science, the search for abiotic processes is still in its infancy.

A few attempts have been made at quantifying confidence in claims made about evidence of extraterrestrial life. One first attempt is the “Confidence of Life Detection” (CoLD) scale, which is linear, presented in Green et al. (2021). The scale runs from 1 to 7, where 1 indicates low confidence and 7 refers to very high confidence, and is put forward is a communication for clear communication with the public (Fig. 1).

Observing Fig. 1, we can see that Level 4 would indicate that sources of abiotic processes have been shown to be implausible. Given the above discussion, the authors state that this is difficult to quantitfy, and that Level 4 could both correspond to situations where confidence about evidence of extraterrestrial life is low, and to situations where it is high. Consequently, this would obviously be a source of confusion for the general public.

Another assessment of the problem comes from the “Standards of Evidence for Life Detection Community Workshop” which took place in 2021 among the Network for Life Detection (NfoLD) and Nexus for Exoplanet System Science (NExSS) communities, and led to a publication by Meadows et al. (2022). One major distinction between this new framework and the CoLD scale is the rejection of any enforced “linearity”. Five questions are thus proposed, and interpreting confidence levels amounts to an iterative process which may or may not respect the particular order in which they are presented (Fig. 2):

The process presented by the NfoLD/NExSS communities would better reflect the scientific methodology, but at the same time somewhat leaves behind the objective of quantifying confidence on a numerical scale.

One approach we can use in quantifying probabilities is Bayes’ theorem: to determine the probabilities \(P(L, E)\) we have detected life given some evidence \(E\), we can write

$$ p(L, E)=\frac{p(E, L) p(L)}{p(E)}=\frac{p(E, L) p(L)}{p(E, L) p(L)+p(E, \neg L) p(\neg L)} $$

In this equation however we can see that the term \(P(E|\neg L)\) is included, which is the probability, discussed above, that the evidence \(E\) we’re considering is due to some abiotic process. We can thus see that Bayes’ theorem can’t really help here, as by definition it requires us to determine the probability of the same processes that we were previously having difficulty quantifying. Other options, Vickers and colleagues mention, include arbitrarily choosing \(P(E|\neg L)=0.5\) or some interval, but these aren’t really interesting options in the case at hand.

One other area where uncertainty definitely plays an important role is that of reports on global warming. Specifically, the Intergovernmental Panel on Climate Change (IPCC) uses in its reports statements such as these:

The ocean has absorbed about 30% of the anthropogenic carbon dioxide, resulting in ocean acidification and changes to carbonate chemistry that are unprecedented for at least the last 65 million years (high confidence).

These statements can include one of six levels of confidence:

One of the hallmarks of this methodology used by the IPCC is that it has stood the test of time: this scale has been considered appropriate for more than 20 years, and is the best solution that has been found to communicate on these important matters with laypersons and policymakers. Vickers and colleagues thus suggest that there’s no reason for it to work in the same way for astrobiology.

The authors however stress the fact that they’re not trying to offer some new framework or methodology. What could be done is trying to adapt the IPCC framework to the ones that were developed already—such as the CoLD scale or NfoLD/NExSS frameworks presented above—by making use of this vocabulary describing confidence about the various claims being made.

One possible way to make this work for astrobiology and put forth by Vickers et al. would consequently be to use both “first-order” and “second-order” assessments of evidence (Fig. 3): first-order evaluations would refer to robustness of evidence as traditionally considered in scientific publications, while second-order evaluations would indicate how confident is the community about these claims.

How does the confidence about current biosignatures, such as a CH\(_4\)—O\(_2\) disequilibrium, would be considered in this scheme? Vickers et al. point out that while some scientists would think this is robust evidence, others might be more cautious when considering the many “cautionary tales” discussed above—hence the consensus would implicitely reflect this diversity. And sometimes, the IPCC framework would inevitably lead to an “undefined” confidence level, which would be the case if it’s too early to reach any agreement.

To sum it up, three suggestions are being made by Vickers and colleagues as a way to improve assessments of confidence levels in extraterrestrial life: