The Real Story of Climategate

A year ago today, an unidentified hacker published a zipped folder in several locations online. In this folder were approximately one thousand emails and three thousand files which had been stolen from the backup server of the Climatic Research Unit in the UK, a top centre for global temperature analysis and climate change studies. As links to the folder were passed around on blogs and online communities, a small group of people sorted through the emails, picking out a handful of phrases that could be seen as controversial, and developing a narrative which they pushed to the media with all their combined strength. “A lot is happening behind the scenes,” one blog administrator wrote. “It is not being ignored. Much is being coordinated among major players and the media. Thank you very much. You will notice the beginnings of activity on other sites now. Here soon to follow.”

This was not the work of a computer-savvy teenager that liked to hack security systems for fun. Whoever the thief was, they knew what they were looking for. They knew how valuable the emails could be in the hands of the climate change denial movement.

Skepticism is a worthy quality in science, but denial is not. A skeptic will only accept a claim given sufficient evidence, but a denier will cling to their beliefs regardless of evidence. They will relentlessly attack arguments that contradict their cause, using talking points that are full of misconceptions and well-known to be false, while blindly accepting any argument that seems to support their point of view. A skeptic is willing to change their mind. A denier is not.

There are many examples of denial in our society, but perhaps the most powerful and pervasive is climate change denial. We’ve been hearing the movement’s arguments for years, ranging from illogic (“climate changed naturally in the past, so it must be natural now“) to misrepresentation (“global warming stopped in 1998“) to flat-out lies (“volcanoes emit more carbon dioxide than humans“). Of course, climate scientists thought of these objections and ruled them out long before you and I even knew what global warming was, so in recent years, the arguments of deniers were beginning to reach a dead end. The Copenhagen climate summit was approaching, and the public was beginning to understand the basic science of human-caused climate change, even realize that the vast majority of the scientific community was concerned about it. A new strategy for denial and delay was needed – ideally, for the public to lose trust in researchers. Hence, the hack at CRU, and the beginning of a disturbing new campaign to smear the reputations of climate scientists.

The contents of the emails were spun in a brilliant exercise of selective quotation. Out of context, phrases can be twisted to mean any number of things – especially if they were written as private correspondence with colleagues, rather than with public communication in mind. Think about all the emails you have sent in the past decade. Chances are, if someone tried hard enough, they could make a few sentences you had written sound like evidence of malpractice, regardless of your real actions or intentions.

Consequently, a mathematical “trick” (clever calculation) to efficiently analyse data was reframed as a conspiracy to “trick” (deceive) the public into believing the world was warming. Researchers discussed how to statistically isolate and “hide the decline” in problematic tree ring data that was no longer measuring what it used to, but this quote was immediately twisted to claim that the decline was in global temperatures: the world is cooling and scientists are hiding it from us!

Other accusations were based not on selective misquotation but on a misunderstanding of the way science works. When the researchers discussed what they felt were substandard papers that should not be published, many champions of the stolen emails shouted accusations that scientists were censoring their critics, as if all studies, no matter how weak their arguments, had a fundamental right to be published. Another email, in which a researcher privately expressed a desire to punch a notorious climate change denier, was twisted into an accusation that the scientists threatened people who disagreed with them. How was it a threat if the action was never intended to materialize, and if the supposed target was never aware of it?

These serious and potentially damaging allegations, which, upon closer examination, are nothing more than grasping at straws, were not carefully examined and evaluated by journalists – they were repeated. Early media reports bordered on the hysterical. With headlines such as “The final nail in the coffin of anthropogenic global warming” and “The worst scientific scandal of our generation“, libelous claims and wild extrapolations were published mere days after the emails were distributed. How could journalists have possibly had time to carefully examine the contents of one thousand emails? It seems much more likely that they took the short-cut of repeating the narrative of the deniers without assessing its accuracy.

Even if, for the sake of argument, all science conducted by the CRU was fraudulent, our understanding of global warming would not change. The CRU runs a global temperature dataset, but so do at least six other universities and government agencies around the world, and their independent conclusions are virtually identical. The evidence for human-caused climate change is not a house of cards that will collapse as soon as one piece is taken away. It’s more like a mountain: scrape a couple of pebbles off the top, but the mountain is still there. For respected newspapers and media outlets to ignore the many independent lines of evidence for this phenomenon in favour of a more interesting and controversial story was blatantly irresponsible, and almost no retractions or apologies have been published since.

The worldwide media attention to this so-called scandal had a profound personal impact on the scientists involved. Many of them received death threats and hate mail for weeks on end. Dr. Phil Jones, the director of CRU, was nearly driven to suicide. Another scientist, who wishes to remain anonymous, had a dead animal dumped on his doorstep and now travels with bodyguards. Perhaps the most wide-reaching impact of the issue was the realization that private correspondence was no longer a safe environment. This fear only intensified when the top climate modelling centre in Canada was broken into, in an obvious attempt to find more material that could be used to smear the reputations of climate scientists. For an occupation that relies heavily on email for cross-national collaboration on datasets and studies, the pressure to write in a way that cannot be taken out of context – a near-impossible task – amounts to a stifling of science.

Before long, the investigations into the contents of the stolen emails were completed, and one by one, they came back clear. Six independent investigations reached basically the same conclusion: despite some reasonable concerns about data archival and sharing at CRU, the scientists had shown integrity and honesty. No science had been falsified, manipulated, exaggerated, or fudged. Despite all the media hullabaloo, “climategate” hadn’t actually changed anything.

Sadly, by the time the investigations were complete, the media hullabaloo had died down to a trickle. Climategate was old news, and although most newspapers published stories on the exonerations, they were generally brief, buried deep in the paper, and filled with quotes from PR spokespeople that insisted the investigations were “whitewashed”. In fact, Scott Mandia, a meteorology professor, found that media outlets devoted five to eleven times more stories to the accusations against the scientists than they devoted to the resulting exonerations of the scientists.

Six investigations weren’t enough, though, for some stubborn American politicians who couldn’t let go of the article of faith that Climategate was proof of a vast academic conspiracy. Senator James Inhofe planned a McCarthy-like criminal prosecution of seventeen researchers, most of whom had done nothing more than occasionally correspond with the CRU scientists. The Attorney General of Virginia, Ken Cuccinelli, repeatedly filed requests to investigate Dr. Michael Mann, a prominent paleoclimatic researcher, for fraud, simply because a twelve-year-old paper by Mann had some statistical weaknesses. Ironically, the Republican Party, which prides itself on fiscal responsibility and lower government spending, continues to advocate wasting massive sums of money conducting inquiries which have already been completed multiple times.

Where are the politicians condemning the limited resources spent on the as yet inconclusive investigations into who stole these emails, and why? Who outside the scientific community is demanding apologies from the hundreds of media outlets that spread libelous accusations without evidence? Why has the ongoing smear campaign against researchers studying what is arguably the most pressing issue of our time gone largely unnoticed, and been aided by complacent media coverage?

Fraud is a criminal charge, and should be treated as such. Climate scientists, just like anyone else, have the right to be presumed innocent until proven guilty. They shouldn’t have to endure this endless harassment of being publicly labelled as frauds without evidence. However, the injustice doesn’t end there. This hate campaign is a dangerous distraction from the consequences of global climate change, a problem that becomes more difficult to solve with every year we delay. The potential consequences are much more severe, and the time we have left to successfully address it is much shorter, than the vast majority of the public realizes. Unfortunately, powerful forces are at work to keep it that way. This little tussle about the integrity of a few researchers could have consequences millennia from now – if we let it.

Update: Many other climate bloggers are doing Climategate anniversary pieces. Two great ones I read today were Bart Verheggen’s article and the transcript of John Cook’s radio broadcast. Be sure to check them out!

We Have Slides!

After a marathon PowerPoint-session yesterday I finally got my 63 slides out of the way. Here is the presentation for anyone who is interested. The script is written in the notes beneath the slides.

I like to have things fading in and out of my slides, so sometimes the text boxes and images are stacked on top of each other and it won’t make sense until you view the animation.

Researching the median lethal dose of arsenic during my spare at school was really awkward. I had to do a lot of hasty explaining to my friends about how it was a metaphor for small concentrations having large effects, and no, I wasn’t planning to poison anyone.

Anyway, enjoy.

Mind the Gap (12 MB)

What The Press Should Cover, and Won’t

The first of three investigations into the CRU emails has been released. You can read the British House of Commons’ entire report here, but I found the summary on page 7 to be just as useful. In part, it reads:

We believe that the focus on CRU and Professor Phil Jones, Director of CRU, in particular, has largely been misplaced. Whilst we are concerned that the disclosed emails suggest a blunt refusal to share scientific data and methodologies with others, we can sympathise with Professor Jones, who must have found it frustrating to handle requests for data that he knew – or perceived – were motivated by a desire simply to undermine his work.

In the context of  the sharing of data and methodologies, we consider that Professor Jones’s actions were in line with common practice in the climate science community. It is not standard practice in climate science to publish the raw data and the computer code in academic papers. However, climate science is a matter of great importance and the quality of the science should be irreproachable. We therefore consider that climate scientists should take steps to make available all the data that support their work (including raw data) and full methodological workings (including the computer codes). Had both been available, many of the problems at UEA could have been avoided.

We are content that the phrases such as “trick” or “hiding the decline” were colloquial terms used in private e-mails and the balance of evidence is that they were not part of a systematic attempt to mislead. Likewise the evidence that we have seen does not suggest that Professor Jones was trying to subvert the peer review process. Academics should not be criticised for making informal comments on academic papers.

In the context of Freedom of Information (FOIA), much of the responsibility should lie with UEA. The disclosed e-mails appear to show a culture of non-disclosure at CRU and instances where information may have been deleted, to avoid disclosure. We found prima facie evidence to suggest that the UEA found ways to support the culture at CRU of resisting disclosure of information to climate change sceptics. The failure of UEA to grasp fully the potential damage to CRU and UEA by the non-disclosure of FOIA requests was regrettable. UEA needs to review its policy towards FOIA and re-assess how it can support academics whose expertise in this area is limited.

DeSmogBlog also has a great summary which you can read here.

We know that the system of climate science is not perfect, and that the folks at CRU did not handle things in the best of ways all the time, but who ever does, especially when you are the target of organized campaigns to discredit your field? The real problem, though, is that everyone who keeps up with North American or British news heard that climate scientists were accused of fudging and manipulating data. There is no evidence to support these allegations, and the House of Commons’ report confirms this. However, I’m not naive enough to believe that the media will cover the result of this “scandal” as intensely as they covered the allegations themselves.

Imagine that you read in the newspaper that a man has been charged with murder. It will be months before you find out the verdict of his trial, and unless it’s OJ Simpson, you probably won’t hear the verdict at all. Many, perhaps most, people would assume that the man is guilty.

We assume that allegations have merit, when – at least when it comes to climate science – they just as often do not.

Academic Culture From the Inside – a Guest Post by Steve Easterbrook

Steve Easterbrook is a comp-sci professor at the University of Toronto who has also worked at the University of Sussex and NASA. Recently, he decided to apply his software engineering expertise to the challenge of climate change, particularly relating to climate models.

This post began as a comment on a recent RealClimate post about media coverage of the CRU hack. I liked it so much that I requested his permission to reprint it here. Enjoy!

I’m afraid to say that a lot of the personal emails between academics in any field are probably not very nice. We tend to be very blunt about what appears to us as ignorance, and intolerant of anything that wastes our time or distracts us from our work. And when we think (rightly or wrongly) that the peer review process has let another crap paper through, we certainly don’t hold back in expressing our opinions to one another.

Of course, this is completely different to how we behave when we meet one another. Most scientists seem able to distinguish clearly between the intellectual cut and thrust (in which we’re very rude about one another’s ideas) and social interactions (in which we all get together over a beer and bitch about the downsides of academic life). Occasionally, there’s someone who is unable to separate the two, and takes the intellectual jabs personally, but such people are rare enough in most scientific fields that the rest of us know exactly who they are, and try to avoid them at conferences!

Part of this is due to the nature of the academic research. We care deeply about intellectual rigor, and preserving the integrity of the published body of knowledge. But we also know that many key career milestones are dependent on being respected (and preferably liked) by others in the field, such as the more senior people who write recommendation letters for tenure and promotion and honors, or the scientists with competing theories who will get asked to peer review our papers, etc.

Most career academics have large egos and very thick skins. I think the tenure process and the peer review process filter out those who don’t. So, expect to see rudeness in private, especially when we’re discussing other scientists behind their backs with likeminded colleagues, coupled with a more measured politeness in public (e.g. at conferences).

Now, in climate science, all our conventions are being broken. Private email exchanges are being made public. People who have no scientific training and/or no prior exposure to the scientific culture are attempting to engage in a discourse with scientists, and these people just don’t understand how science works. The climate scientists whom they attempt to engage are so used to interacting only with other scientists (we live rather sheltered lives- they don’t call it the ivory tower for nothing) that they don’t know how to engage with these outsiders. What in reality is a political streetfight, we mistake for an intellectual discussion over brandy in the senior commonroom. Scientists have no training for this type of interaction, and so our responses look (to the outsiders)  rude, dismissive, and perhaps unprofessional.

Journalists like Monbiot, despite all his brilliant work in keeping up with the science and trying to explain it to the masses, just haven’t ever experienced academic culture from the inside. Hence his call, which he keeps repeating, for Phil Jones to resign, on the basis that Phil reacted unprofessionally to FOI requests. You don’t get data from a scientist by using FOI requests, you do it by stroking their ego a little, or by engaging them with a compelling research idea you want to pursue with it. And in the rare cases where this doesn’t work, you do the extra work to reconstruct it from other sources, or modify your research approach (because it’s the research we care about, not any particular dataset itself). So to a scientist, anyone stupid enough to try to get scientific data through repeated FOI requests quite clearly deserves our utter contempt. Jones was merely expressing (in private) a sentiment that most scientists would share – and extreme frustration with people who clearly don’t get it.

The same misunderstandings occur when outsiders look at how we talk about the peer-review process. We’re used to having our own papers rejected from time to time, and we learn how to deal with it – quite clearly the reviewers were stupid, and we’ll show them by getting it published elsewhere (remember, big ego, thick skin). We’re also used to seeing the occasional crap paper get accepted (even into our most prized journals), and again we understand that the reviewers were stupid, and the journal editors incompetent, and we waste no time in expressing that. And if there’s a particularly egregious example, everyone in the community will know about it, everyone will agree it’s bad, and some will start complaining loudly about the editor who let it through.

Yet at the same time, we’re all reviewers, so it’s understood that the people we’re calling stupid and incompetent are our colleagues. And a big part of calling them stupid or incompetent is to get them to be more rigorous next time round, and it works because no honest scientist wants to be seen as lacking rigor. What looks to the outsider like a bunch of scientists trying to subvert some gold standard of scientific truth is really just scientists trying to goad one another into doing a better job in what we all know is a messy, noisy process.

The bottom line is that scientists will always tend to be rude to ignorant and lazy people, because we expect to see in one another a driving desire to master complex ideas and to work damn hard at it. Unfortunately the outside world (and many journalists) interpret that rudeness as unprofessional conduct. And because they don’t see it every day (like we do!) they’re horrified.

Freedom of Information

The only real issue that the hacked CRU emails brought up, the only allegation that didn’t fall apart if you were familiar with the literature (*cough cough hide the decline*), was the failure of Phil Jones to respond to some of the FOI (Freedom of Information) requests.

This looks bad on the surface, and it certainly has been spun that way – climate scientists hiding their data because they know it’s wrong and they don’t want anybody to find out. And ignoring FOI requests is a really stupid thing to do, no matter what the situation is. However, as with all the other allegations, some more context as to the nature and volume of these requests makes ignoring them understandable, if not excusable.

The Freedom of Information Act is important to a democratic society, but its major flaw is that it fails to distinguish its abuse. An article from the Sunday Times describes, in an interview with Phil Jones, what the FOI situation at CRU was.

In July 2009 alone, they received 60 FOI requests – most asking for data that was already freely available online. However, turning down a request takes 18 hours of work, and they only had 13 staff at CRU – all of which had better things to do than respond to needless FOI requests.

In another instance, over a matter of days, they received 40 FOI requests, which obviously all came from the same form letter – but each asked for data from a different 5 countries. So in total, temperature data for 200 different countries (again, most of which was already freely available) was requested, and all the forms came to CRU rather than the offices in the countries the data came from, or even the countries the authors of the FOI forms lived in. Phil Jones is sure that this coordinated attack originated at Climate Audit, which “just wanted to waste our time….they wanted to slow us down.”

Out of irritation, Phil Jones made some comments over email to his colleagues about how he wished that they could just get rid of the data rather than do all this work distributing it needlessly. This was purely a hypothetical proposition, though, as CRU doesn’t own any of the data. “We have no data to delete,” he says. “It comes to us from institutions around the world….it’s all available from other sources.”

When you are abused with FOI requests, ignoring them is not the right thing to do, and Phil Jones knows it – “I regret that I did not deal with them in the right way,” he says. His actions and words cannot be excused, but with more context, it’s obvious that his motives were not to cover up flaws in the data or hide it from critics. He just wanted to do his work.

It’s a great example of how the CRU hack compromises the professional reputations of some of the scientists involved, but it does not compromise one iota of the science. “I am obviously going to be much more careful about my emails in future, ” remarks Phil Jones. “I will write every email as if it is for publication. But I stand 100% behind the science. I did not manipulate or fabricate any data.”

CRU was not the only institution to be abused with FOI requests. The field of climate research has been grappling with this issue for the past few years. Take Benjamin Santer, for example. In a story he relays here, he describes how, following the publication of his 2008 paper, an FOI request by Stephen McIntyre asked for all the raw data used in his study so it could be replicated. Santer pointed him to the data, which was already freely available online. But then he was given two subsequent FOI requests, which asked for all of his intermediate calculations and two years of email correspondence related to the data. Obtaining this information is completely unnecessary to replicate a study, and it is certainly not normal scientific practice – the only reason you would want them would be to find material that could be framed as embarrassing and used to discredit the study and the researcher – as if Ben Santer hasn’t been through enough already. So he turned the FOI requests down, and was immediately flooded with hate mail from Climate Audit readers until he released the intermediate calculations, purely because he “wanted to continue with my scientific research…….I did not want to spend all of my available time and energy responding to harassment incited by Mr. McIntyre’s blog.”

Gavin Schmidt, a climate modeler at NASA, adds to the list of instances of FOI abuse in climate science. He remarked that “In my previous six years I dealt with one FoIA request. In the last three months, we have had to deal with I think eight…..These FoIAs are fishing expeditions for potentially embarrassing content but they are not FoIA requests for scientific information.”

James Hansen, the director of GISS at NASA, has similar opinions. Following the CRU hack, he writes, “I am now inundated with broad FOIA requests for my correspondence, with substantial impact on my time and on others in my office. I believe these to be fishing expeditions, aimed at finding some statement(s), likely to be taken out of context, which they would attempt to use to discredit climate science.”

The broad abuse of the Freedom of Information Act in the field of climate science is worrying, and it calls for some kind of caveat that will distinguish it from legitimate use of FOI. Research into climate change is vital at this point in human history, but if top researchers are forced to spend their time filling out needless paperwork instead, the field will suffer. The past few months have shown us that institutions of climate science are in need of representatives specialized in media relations. Perhaps they also need to employ dozens of students to fill out FOI forms, or lawyers to defend them from the constant attack they are under.

The Antithesis to Nitpicking

Sometimes we have to step back and look at the big picture. We have to remember that not everyone has heard or believed the one about global warming stopping in 1998. Denialists centre around nitpicking and ideas that global warming is a “house of cards”, so we respond the same way: countering all the “mistakes” they claim to have found.

In reality, climate change is an incredibly robust phenomenon that we’ve known about for decades – and the basic physics behind it, for over a century. It’s not some new, shaky discovery. It’s not going to be overturned because scientists at CRU do not always say nice things about their critics.

So I was very pleased when I opened up YouTube today to see that Peter Sinclair’s latest video was all about this big picture. If I had to choose just one of his videos to share with everyone I knew, this would certainly be it. This is the kind of message we need to get out there; this is the kind of angle we need to take.

A Good Batch of News

Today’s batch of news feeds was great. I have not one, but two, posts to comment on from elsewhere in the climate blogosphere.

Firstly, James from The Island of Doubt has written a fantastic article on the new line of denialist attack. This is the best bit:

“Here’s IPCC author Phil Duffy, whose thoughts on the subject inspired mine:

Things happen, but let’s react appropriately. Medical doctors make mistakes every day. (In fact, medical errors in the US alone kill hundreds of people daily–the equivalent of a jumbo-jet crash.) And no doubt many of these errors happen because established procedures are ignored, sometimes knowingly. Does this mean the entire edifice of western medicine is wrong, or prejudiced, or the product of a conspiracy, and should be rejected? Of course not. Furthermore, the medical profession as a whole is still held in high regard, as it should be.

No one worth listening to is calling for a massive inquiry into the science underpinning modern medicine, or the engineering foundations of the car industry. But pseudoskeptics argue that the IPCC is systematically fraudulent simply because a couple of statements among thousands of pages of heavily edited and re-editing (and re-re-edited) documents cite gray literature instead of the peer-reviewed literature that supplied the science in the first place.

Is it controversial among those study such things that 40% of the Amazon is susceptible to drought? No. Is it controversial that Himalayan glaciers are receding? No. Only the way in which that science was presented and attributed was found faulty. To thrown out anthropogenic global warming because of such missteps is the climatological analog of dismissing an entire faculty of medicine because someone correctly diagnosed a patient because of a story they read in New Scientist instead of the medical journal article on which the story was based. Bad judgment? Yes. Fatal error? No.”

And one more:

“This isn’t about censorship. Thanks to the Internet, everyone can find a way to spread their point of view. It’s about applying the same standards to coverage of climate change that “respectable” media apply to fields like sports, business and other fields. Sports bloggers and journalists for major news organizations couldn’t get away with making up baseball statistics for long. They’d be laughed out of the office. Business reporters can’t supply false stock market numbers because that would be a violation of very essence of what they’re supposed to be doing. And yet climate science is somehow different. If you work for theDaily Mail or Telegraph in the UK, or Fox News (or the Washington Post‘s op-ed section) in the U.S., you can say or print anything you want about climatology, without regard for the facts. That should not be tolerated.”

Read the rest of the article here. While you’re at it, see if you can get the New York Times to print it.

James’ last point about censorship leads nicely into the second post I want to discuss. DeSmogBlog is having some problems in the comment section, and plans to tightly moderate comments in the future.

I certainly sympathize with the DeSmogBlog writers. I find that the vast majority of Internet discussions regarding climate change turn into such a food fight that reasonable and insightful discussion falls through the cracks. Well-meaning and fact-checking people are so busy responding to the same old objections, or are so intimidated by trolling commenters (quick poll – who here has been called a Nazi for explaining basic atmospheric science?), that they do not post the wonderfully thought-provoking things that they have to say.

Keeping comments completely open and unmoderated, therefore, is catering to those who wish to waste others’ time, hold nothing back in their criticisms of individuals, and can’t be bothered to check citations before spreading something around. I don’t think it’s necessary to give that kind of discussion any more space. Instead, I wish to cater to those who have insightful (and accurate!) things to say, by providing a supportive community for them to do so. RealClimate sums up my feelings on this topic quite well:

“Comments that accuse as of bad faith, fraud and dishonesty are not ways to move forward any conversation – how can you have a dialog with people who don’t believe a word you say? We choose to try and create a space for genuine conversation, which means weeding out the trolls and the noise. This is an imperfect process, but the alternative is a free-for-all that quickly deteriorates into a food fight. There are plenty of places to indulge in that kind of crap. There are only a few places where it’s not and we are not embarrassed to try to make this site one of them.”

By the sounds of things, DeSmogBlog’s comment policy is going to be much more stringent than mine. I don’t “tightly” moderate comments – on the contrary, there are many remarks that, in retrospect, I realize should never have passed moderation. But I do have two basic rules that I try to adhere to: 1) provide legitimate peer-reviewed citations for your claims (unless they’re common knowledge – I won’t make you cite Fourier and Arrhenius), and 2) refrain from personal attacks, aggression, etc. I like to show when a person’s comment has been deleted, and give a brief explanation why, so readers can see what’s going on without having to go through the agony of actually reading it.

The first complaint when a policy like this is enacted is invariably “censorship!!!!”, and, almost six months later, I still get those complaints. But “censorship” is not an accurate term for such policies. You can still go and say whatever you want about climate change elsewhere – on your own blog, on YouTube, in a national news release. There are infinite other ways to make unsupported claims and spread rumours about people, so filtering them out of my own blog (or RealClimate, or DeSmogBlog) in effort to promote a decent discussion is by no means totalitarian.