A Better Credibility Spectrum

It’s been over a year since I wrote The Credibility Spectrum, my first post ever. Since then I’ve learned a lot, and have altered the credibility spectrum in my own mind – so I thought I’d alter it here, too.

This credibility spectrum is sort of split into two: the scientific community, and the non-scientific community. The scientific community starts with scientists, and I want to stress that this category only includes scientists with experience in the issue at hand. Just because someone has a PhD in one area of science doesn’t mean that they are an expert in all areas. For example, it’s very easy for a computer scientist to go through ten years of university without studying any biology at all.  Treating them as an expert in evolution, therefore, would be illogical.

These scientists write peer-reviewed papers, published in journals like Nature and Science, which are another step up the credibility spectrum. Instead of just having the name of an expert attached to them, their methods and conclusions have been evaluated for robustness and accuracy. This is the minimum level of credibility from which I recommend citing scientific claims.

However, as thousands of papers are published every month, and they’re generally studying the frontier of their field, it’s inevitable that some of them will be proven wrong later. As Gavin Schmidt and Michael Mann wisely said, peer review is a necessary but not sufficient condition.

That’s why there are scientific organizations and assessment reports, like NASA or the IPCC, which compile peer-reviewed knowledge which has stood the test of time into consensus statements. Even the top level of the credibility spectrum isn’t infallible, but it sure has a low error rate compared to other sources.

Everyone who isn’t a scientist, which is most of us, falls into the lower half of the credibility spectrum. The category I refer to as “communicators” includes the mainstream media, projects like Manpollo or 350.org, high school teachers, politicians…….They’re not part of the scientific community, so you should always always always check their citations, but they’re held more accountable for what they say than just any random person on the street. If they make glaring errors, people will be more upset than if the same errors were made by individuals – comments on YouTube, discussions with your neighbours – which make up the lowest rung of our credibility spectrum.

Something that I found really interesting  when I put this together was the general flow of information between different sources. In the scientific community, research starts with scientists, and the best research is published in journals, and the best journal articles are picked up by major organizations. As the scientific knowledge progresses through the different sources, the weaker assertions are weeded out along the way. The flow of information is going up the pyramid, towards the narrower part of the pyramid, so that only the best is retained.

However, in the non-scientific community, the flow of information goes the other way. Communicators present information to individuals, which is a much larger group. Information travelling down the pyramid, instead of up, allow rumours and misconceptions to flourish much more easily.

This isn’t to say that, when they come head-to-head, organizations are always right and individuals are always wrong. But given the history of such disagreements, and the levels of credibility involved, you’ll know where to place your bets.


4 thoughts on “A Better Credibility Spectrum

  1. Very astute!

    It’s possible for scientists to be misled — they, too, are also susceptible to confirmation bias, though perhaps not to such a large extent as those below them in the pyramid. The ‘communicators,’ however, will have more bias (both positive and negative) for a variety of reasons, many of which will be entirely unrelated to the science. This will have the effect of enhancing schisms at the bottom of the pyramid, fuelled by hyperbole, spin, and even outright lies.

    In effect, I see an arrow from the top to the bottom, labelled ‘increasing bias.’

    Better results would be obtained by improving connections between the top and the bottom of the pyramid. As the second tier up is the main conduit for information transmission to the bottom, would it be fair to claim that the ‘communicators’ have a duty to ensure that they are informed by the top of the pyramid?

  2. How much more nuance do you want to add?
    There are journals that have an exceptional reputation (and even they occasionally get it wrong) then there are journals that have a poor reputation.
    The journals you have pictured would be consised by many is the pick of the bunch.

    Some journals have a very restricted readership, just too specialised for the wider scientific community.

    Climate science is such a wide field covering a multitude of specialities and using even more. So a statistical can validly question the statistical methodology of a scientific paper. But some statisticians have a reputation for excessively questioning one side of the argument, totally ignoring the faults in the other. Then should not be surprised that their own methodology also gets questioned.

    A nice job.

  3. One correction. NASA does not do scientific assessments. Instead it is a producer of basic research and so more properly belongs with the scientists. The National Academy of Science however does produce assessment reports and make statements on what the science means and so should be in your top tier.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.