When I was a brand-new PhD student, full of innocence and optimism, I loved solving bugs. I loved the challenge of it and the rush I felt when I succeeded. I knew that if I threw all of my energy at a bug, I could solve it in two days, three days tops. I was full of confidence and hope. I had absolutely no idea what I was in for.
Now I am in the final days of my PhD, slightly jaded and a bit cynical, and I still love solving bugs. I love slowly untangling the long chain of cause and effect that is making my model do something weird. I love methodically ruling out possible sources of the problem until I eventually have a breakthrough. I am still full of confidence and hope. But it’s been a long road for me to come around full circle like this.
As part of my PhD, I took a long journey into the world of model coupling. This basically consisted of taking an ocean model and a sea ice model and bashing them together until they got along. The coupling code had already been written by the Norwegian Meteorological Institute for Arctic domains, but it was my job to adapt the model for an Antarctic domain with ice shelf cavities, and to help the master development team find and fix any problems in their beta code. The goal was to develop a model configuration that was sufficiently realistic for published simulations, to help us understand processes on the Antarctic continental shelf and in ice shelf cavities. Spoiler alert, I succeeded. (Paper #1! Paper #2!) But this outcome was far from obvious for most of my PhD. I spent about two and a half years gripped by the fear that my model would never be good enough, that I would never have any publishable results, that my entire PhD would be a failure, etc., etc. My wonderful supervisor insisted that she had absolute confidence in my success at every step along the way. I was afraid to believe her.
Model coupling is a shitfight, and anyone who tells you otherwise has never tried it. There is a big difference between a model that compiles and runs with no errors, and a model that produces results in the same galaxy as reality. For quite a while my model output did seem to be from another galaxy. Transport through Drake Passage – how we measure the strongest ocean current in the world – was going backwards. In a few model cells near the Antarctic coast, sea ice grew and grew and grew until it was more than a kilometre thick. Full-depth convection, from the ocean surface to the seafloor, was active through most of the Southern Ocean. Sea ice refused to export from the continental shelf, where it got thicker and thicker and older and older, while completely disappearing offshore.
How did I fix these bugs? Slowly. Carefully. Methodically. And once in a while, frantically trying everything I could think of at the same time, flailing in all directions. (Sometimes this works! But not usually.) My colleagues (who seem to regard me as The Fixer of Bugs) sometimes ask what my strategy is, if there is a fixed framework they can follow to solve bugs of their own. But I don’t really have a strategy. It’s different every time.
It’s very hard to switch off from model development, as the bugs sit in the back of your brain and follow you around day and night. Sometimes this constant, low-level mulling-over is helpful – the solutions to several bugs have come to me while in the shower, or walking to the shops, or sitting in a lecture theatre waiting for a seminar to start. But usually bug-brain just gets in the way and prevents you from fully relaxing. I remember one night when I didn’t sleep a wink because every time I closed my eyes all I could see were contour plots of sea ice concentration. Another day, at the pub with my colleagues to celebrate a friend’s PhD submission, I stirred my mojito with a straw and thought about stratification of Southern Ocean water masses.
***
When you spend all your time working towards a goal, you start to glorify the way you will feel when that goal is reached. The Day When This Bug Is Fixed. Or even better, The Day When All The Bugs Are Fixed. The clouds will part, and the angels will sing, and the happiness you feel will far outweigh all the strife and struggle it took to get there.
I’m going to spoil it for you: that’s not how it feels. That is just a fiction we tell ourselves to get through the difficult days. When my model was finally “good enough”, I didn’t really feel anything. It’s like when your paper is finally accepted after many rounds of peer review and you’re so tired of the whole thing that you’re just happy to see the back of it. Another item checked off the list. Time to move on to the next project. And the nihilism descends.
But here’s the most important thing. I regret nothing. Model development has been painful and difficult and all-consuming, but it’s also one of the most worthwhile and strangely joyful experiences I’ve had in my life. It’s been fantastic for my career, despite the initial dry spell in publications, because it turns out that employers love to hire model developers. And I think I’ve come out of it tough as nails because the stress of assembling a PhD thesis has been downright relaxing in comparison. Most importantly, model development is fun. I can’t say that enough times. Model development is FUN.
***
A few months ago I visited one of our partner labs for the last time. I felt like a celebrity. Now that I had results, everyone wanted to talk to me. “If you would like to arrange a meeting with Kaitlin, please contact her directly,” the group email said, just like if I were a visiting professor.
I had a meeting with a PhD student who was in the second year of a model development project. “How are you doing?” I asked, with a knowing gaze like a war-weary soldier.
“I’m doing okay,” he said bravely. “I’ve started meditating.” So he had reached the meditation stage. That was a bad sign.
“Try not to worry,” I said. “It gets better, and it will all work out somehow in the end. Would you like to hear about the kinds of bugs I was dealing with when I was in my second year?”
I like to think I gave him hope.
ClimateSight 
Hi Kaitlin,
I have been following you since you are a childhood researcher, but unfortunately, I see you as a child still in your research methodology. The model development is not about finding out bugs and correcting them, but it is about finding bugs in the physical processes and correcting them through proper observational research. As I said a few years ago, you are under the influence of improper guidance. I hope you are wasting your brain on unimportant bugging of the models.
Regards,
Kishore
I can totally relate. Model development is hell!
This really applies to any kind of nontrivial software development. Finding some bugs is easy. Finding others can be excruciating. I’ve found some in minutes, others have taken weeks. I can completely relate to the odd-moments’ “solutions.” I’ve had some of them suddenly come to me in the middle of the night. Once in a while they’re actually right. Most of the time, they’re ridiculous, but it’s a sign the your mind is working on the problem all the time.
In any event, it’s good to see you back. You’ve been missed. I suggest that you ignore ungraceful comments. And what the hell are “bugs in physical processes”?
Excuse me chrisd, I am not here to belittle anybody. Kaitlin is more matured than you. If you can’t accept negative comments, you are not supposed to post online. If you have the problem with me, please suggest her off-line block my ID. I don’t have any problem!
You have to understand that the finding out of bugs and correcting them is mostly software kind of thing. If you fully focus on such things, you are not treated as a climate scientist/meteorologist, but you are called a just software testing programmer.
Coming to your “HELL”, If you don’t know, why the hell I am wasting my time to discuss this. At least, here I would like to let you know what I mean. If you look through the parameterizations in the models, you will come across plenty of them and they are bigger bugs, which is missed by so-called SCIENTISTS.
I suggest you to discuss in a better way in the future.
Thanks chrisd.
I think part of the problem here is a misunderstanding of what different people mean by “bugs”. Should this term be limited to software problems which are the result of human error – incorrect units, indexing errors, and other clear-cut mistakes? Or should we also consider biases in the model output which are the result of unsuitable parameterisations, poor tuning, etc. (what kishoreragi calls “bugs in physical processes”?) It’s often not clear which category a bug falls into until you solve it, and even the first category of bugs can require a great deal of specialised knowledge of the climate system to solve. The code base is simply so massive that you need to be able to properly interpret the model output in order to know where to look.
I like to use the definition of Pipitone and Easterbrook, who say that a software defect is “anything worth fixing”. This encompasses both categories of bugs that I mentioned above. During my PhD I dealt with some bugs which were the result of stupid mistakes, and some which were due to well-known biases common to many models which nobody fully understands yet (and I like to think I contributed something to the discussion). I learned something valuable about the climate system from all of them.
Finally, let’s not denigrate software engineers. Several of my colleagues are employed full-time in this capacity and they do an incredible job to advance climate model development. They deserve no less respect than scientists, and in some areas their knowledge far exceeds ours. A narrow definition of what does and does not constitute “science” is helpful to nobody.
Way to go Kaitlin, and a very important post which shows that the important bugs eradication for climate models is neither eliminating errors in the coding or insects, but finding better representations of physical processes
FWIW bugs in physical processes: Termite contributions to the carbon cycle
https://link.springer.com/chapter/10.1007/978-94-017-3223-9_19
Sorry it took me 2 days to approve this – clearly the WordPress notification system is fallible…
Actions not just models
Stop waste of all kinds – walk, bike, ebike, avoid petrol cars whenever possible, avoid single use plastics, use sweaters (lower heater settings only turn up when needed, off at night if possible) LED lights, of course.
Promote wind/solar whenever possible.
Fly as little as possible.
Stop wars. Cut military spending.
You get the idea.
You most certainly did Kaitlin!
Hi Kaitlin,
I never wanted to denigrate software engineers’ contribution to the climate model development. That is the career they chose and helping in the models to work properly, but I wanted to say is that the scientists’ work is not only that but also to look at the errors in the representation of physical processes if one really chooses to develop the models. Maybe, I misunderstood what you mean by the “BUGS”. Thank you for the clarification on that.
Yeah, I understand most of the modeling world does not care about what is happening in the model but looks at the output like having no brains. I really pity that!
“I learned something valuable about the climate system from all of them”. Could you please tell me what is the climate system? The climate system is an objective reality of the earth system. There is no such climate system exists, so calling the climate system is meaningless; though, the whole climate research community uses it often. Thanks to my advisor for enlightening me!
Regards,
Kishore
Then there is bat shit crazy being correct. You do get eddies in Drake’s Passage and one of the right scale could appear as a single cell going the wrong way. Here is an example of an Argo float going backwards. http://www.euroargo-edu.org/floatdata.php?float=1901227
Better you than me.
Well done. Seems like yesterday you were still in high school.
Sorting out fresh spaghetti that you’ve recently made can be a pleasure, but splicing together bowls of cold, stale spaghetti can be, well…, spaghetti hell. You have my sympathies. If your model can teach moron the difference between local weather and global climate events, I’ll consider it a success.
I’ve been developing and debugging theory, algorithms, and code for more than 40 years, and one thing I’ve learned over the years is that other people are often better at finding persistent bugs than I am. When really stuck on a problem, we might try something like this. Find one or more assistants who may be able to help you locate the problem(s). If the assistants are willing to help, the challenge will be to explain the problems to them in sufficient detail so they can help. Don’t just hand over the supporting theory, algorithms, and code, but carefully explain everything to the point where they can understand what the theory, algorithms, and code are supposed to do and not to do. Encourage them to ask questions and watch for glazed eyes. Be prepared to carefully explain and answer questions, because the goal is for them to give you clues that lead to the solution. More likely the assistants will not point to the bugs, but they will lead you to discover them.
Here is a personal example of how this may work. I supervised a number of engineers and computer scientists who were extending the theory, algorithms, and code that I and others had previously developed. One computer scientist, much smarter than me, had been struggling with a problem for days and finally asked if I could help him find a bug. He started to explain the theory, algorithms, and code, and within minutes my eyes were glazing over. I tried to ask intelligent questions, and he patiently explained. I probably understood about 10-20% of what he was saying, but after about two hours of explaining he said, “There’s the problem.”
I hadn’t provided a significant amount of useful input to this computer scientist, so how was he able to find the bug? Because I had been asking him a lot of dumb questions and he had to answer them in ways that I could understand. Forming answers to my stupid questions forced him to drill down into the obvious that he probably had been glossing over. Familiarity is the biggest culprit with debugging ones own theory, algorithms, and code. We have been working with this stuff for days, weeks, months, or even years and we know it like the back of our hands; we’ve even memorized some of it. A few months ago my 6-year-old granddaughter, who already reads better than I do, asked me to read her a story from one of her books. I put on my glasses and started reading. Somewhere on page 3 I accidentally substituted a word. “No, Grandpa,” she said, “that’s the wrong word.” When a person is too close to their theory, algorithms, and code they often gloss over the obvious, where the bugs are more likely to lurk.
Getting help debugging theory, algorithms, and code probably works best when the assistants really want the satisfaction of solving the problem. Since bugs may lie in theory, algorithms, or code, it may be useful to seek help from more than one assistant with expertise in different areas. If the debugging assistants don’t ask questions, and they should be encouraged to do so, we should be wary because we aren’t being forced to revisit the obvious. And we should take seriously all suggestions because some of them may lead us down paths we hadn’t considered.
Long ago I developed software for the U.S. Government where they have a lot of regulations. One regulation included the sentence, “A computer program shall have no more than one bug.” I liked that because it didn’t say “… shall have no more than zero bugs.” We always strived to send out computer programs with zero bugs, but if we just couldn’t find that last pesky bug we’d send it out anyway. (We didn’t develop life-threatening software, so we were safe there.) Then we would have to take seriously anyone who complained of two or more bugs, but not just one.