I do this kind of work. It's frustrating how awful most of the code is. You'd think people smart enough to pursue stem phds would understand basic programming abstractions, but that is often not the case. Another issue is that researchers don't normally use good software engineering practices. I have yet to be given any code that has even a single unit test. Source control is being used these days, but the repositories are usually unorganized messes with unhelpful commit histories. No one keeps track of system dependencies, and few understand build systems. I can spend a week just trying to get some software to build.
I could keep complaining, but I don't want to give people the impression that I don't like what I do. It beats the hell out of writing CRUD apps.
> You'd think people smart enough to pursue stem phds would understand basic programming abstractions, but that is often not the case.
Another way of looking at it: People doing STEM PhDs are extremely busy. They have to master the content in their field, keep up with new research, conduct their own research, write papers, go to conferences, give talks, teach, keep up with whatever department service responsibilities they have, etc.
They're smart people so they're going to be able to slap together some code that does what they need it to do. But to expect them to learn git and how to use it well, to learn about managing dependencies, about unit tests, about build systems, about best practices for documenting code, about programming abstractions or OOP whatever else... that's a lot to ask.
(And I say this as a STEM PhD who did learn all that stuff, more or less.)
I moved from CS into Earth science a decade ago and I was similarly appalled at the state of affairs in the beginning. First, this doesn't have anything to do with how smart people are, it's a question of time and other resources. Our job simply isn't to write pretty code, it's just a means to an end. If you know how to do something in FORTRAN, why would you invest into learning how to do it with <lastest trend>? It's a huge cost! Second, the evolution of such code occurs often over decades; as the article states, with students and post-docs hammering along until it finally does what they need to get their project to move forward. The code usually begins with "Do X" and then the rest of the alphabet is tacked on later. We all know it's bad. But in the end, it's usually only your own lab group using this stuff; maybe a handful of people around the world. The cost of polishing something that started out as hastily written code to get something done in time for {conference,paper,proposal} a decade ago is simply to high. It works. Done.
I wish the state of affairs was prettier, but I doubt it will change any time soon. And frankly, I don't think it has to. Software engineers will always find something to complain about others' code (coding style discussions come to mind). If there's value in commercializing scientific prototypes, experts should rush in and do it. There's no need for us to invest in perfect code just in case someone may want to reuse it. Having said that, NSF is starting to fund initiatives to improve the state of affairs. For my field, it's EarthCube: https://www.earthcube.org/ but this seems to be fumbling along; all of their goals would need to be adapted by scientists outside of that community (standards, etc). Without the right incentives (funding, publications), it's just not going to happen.
Can you identify (say) five things that academics can do with an existing code base that will make your job easier (and therefore cheaper to the downstream users)? Perhaps pitch the top five pick at the abilities of a computer science student looking for a project?
Can you identify (say) three key practices to adopt when starting a new project given the likely skill levels of postgrad students in your domain?
You'd think people smart enough to pursue stem phds would understand basic programming abstractions, but that is often not the case. Another issue is that researchers don't normally use good software engineering practices. I have yet to be given any code that has even a single unit test. Source control is being used these days, but the repositories are usually unorganized messes with unhelpful commit histories. No one keeps track of system dependencies, and few understand build systems.
Good developers get paid enough, that it indicates that maybe our job isn't quite that easy to learn.
Our basic programming abstractions and software engineering practices are things that we as a field developed over time, based on experience; it's not like they're something you just automatically know without having to sink time into studying if you're above some IQ level. They also depend to varying degrees on how big your program is, which parts are expected to change more or less rapidly, what kind of program it is, etc.
Unit tests assume you know what the code is supposed to do before you write it, that the code will change significantly more often than the required functionality, and that the tests are easier to read and check than the code being tested.
Agree with everything except for unit tests, they're something of a cargo-cult fad. Any of my developers caught wasting their time on these would get a stern talking to.
You're not alone in feeling this way. Greg Wilson organized 'Software Carpentry' [0] to teach researchers the basics of software engineering. They regularly do workshops (list/schedule at site).
> Another issue is that researchers don't normally use good software engineering practices. I have yet to be given any code that has even a single unit test. Source control is being used these days, but the repositories are usually unorganized messes with unhelpful commit histories.
Funny, that sounds exactly like my IT Dept.'s approach to coding.
I work in Operations and was exposed to their approach when helping them translate some business rules into SQL for the warehouse management system. During the course of interaction the IT Director lost the code we had worked on together twice, keeps the requirements docs in his email etc. etc.
Me: your engineering practices are terrible
He: We don't have time to do it properly
Me: but somehow we have time to do it three times
I told my manager but no-one really understands or cares enough. The Warehousing Director calling out the IT Director about his approach to IT - where would that all end ?
I remember carefully crafting a software framework for my Ph.D. I got it running, it worked perfectly, it didn't deliver any advantages vs state of the art. I think it took me 7 mths to write. I then slapped together another one, and that took me 3 mths, it worked (just about), it produced some decent results, and I could move onto the next thing. The software I was producing now was not good but this was in fact excellent experience for research software development as as my work progressed into more innovative areas I found that many, many ideas didn't work out at all, so the priority was to be able to develop code really really fast. I suspect that this is what is going on with most research code - write it really fast, not really good.
Give me a break, you are complaining. I don't expect academics to follow enterprise standards. They are more rooted in theory than practice. I expect shitty code, uncommented code, no unit tests. You know that thing SV calls the real MVP? Yup, that's what I expect from academia, the most basic Proof of concept to demonstrate that an idea is possible. Correct is cherished, beauty is not. Note, the correctness is enough for best case/data at hand. What's more important and useful to me and society at large is the papers that follow.
Then there's the group of folks who tend to read those papers and turn it into products. Yes, those are the ones I expect to have repos, unit tests, build systems and all that.
Something that goes against the grain of the contemporary "reproducibility" mindset but worth keeping in mind: that's always the option of reading the paper, extracting the core ideas (but not necessarily the exact algorithm) then sitting down and writing your own. To me, that seems like what "replication" should really mean -- not downloading a Docker image and re-running someone else's analysis verbatim.
This is far from an original thought:
In the good old days physicists repeated each other's experiments, just to be sure. Today they stick to FORTRAN, so that they can share each other's programs, bugs included.
One big problem with software developed in academia is that there is almost no incentive for continued development and maintenance.
Grants are time-limited, and at some point usually the money for developing the software runs out. The PhD students working on the software move on to something else, and you have yet another abandoned piece of scientific software.
There are of course exceptions, but in general it's much harder to get money for maintaining and improving software over a longer timeframe than for building something new.
Software gets maintained (somewhat) by those who make a career out of it. Build something in your PhD, get a postdoc where you can add some more of your own things to it, and get funding for follow on work in which you get your own PhD's to work on it. That's the only model I've seen work, but it's usually very implicit, because when you get to the tenure track stage (or even beyond), you have an incentive to make it look like you're not yet again re-using that thing you were doing 10 years ago. So it's not called 'maintenance' or even 'upgrading' - you call it something else that sounds new, yet has enough links to the old thing to give you validation because people have heard of it, and in this way drag this mutant bastard code through 5 or 10 iterations over 25 years until your career is 'done' and you can finish off the last years in standby mode.
This sounds negative but I don't mean it that way; it's just how it is, no judgement meant. But it's not something you really hear anyone teaching new PhD students as an option, and even if they would, it's highly uncertain and success depends on many factors out of your control. So I wouldn't call it a career path, or even viable career advice.
In my experience there is only little money to be made in scientific software. Perhaps it depends on the field (I worked in design-automation for electronic circuits). Scientists often have difficult budgets (depend on funding), and they often expect software to be free (it's not budgeted for). I'm curious about the experience of others.
PS: I couldn't read the article, presumably because of some anti-adblocker mechanism.
From the HN comment page, click on the "web" link just under the post title, then click on the first result of the google search. It might work better in a Private window.
I'm as guilty of this as anyone, but I'll tell you how to fix it:
Make code review part of peer review.
I think the next generation of scientists at least understand that things like unit tests are useful, but passing peer review is the only incentive to do anything, and reviewers don't read code.
Beyond impractical. Besides it taking 10x as long to peer review something, I can't even imagine my advisor willing to do this. Most professors don't even touch code, they leave it to the graduate students. Hell, I know professors (in EE) who teach code heavy classes that don't even know how to run a single line of the code that they teach; but that's another topic.
Some conferences require artifacts, but this is ludicrously impossible.
I review a paper in an hour or so. If somebody comes to me with some fancy new symbolic execution framework it is going to take me days to review the code. Unless you are planning on paying me for a months work to review 20 papers, this is impossible.
I did a project like this to get three body physics for an iOS/Android app Three Body. (http://nbodyphysics.com/blog/2015/12/). Converting Fortran from 1973 into C# for Unity. This code (only 1000 or so lines) is very dense.
I also cleaned up my own mess from 1996, when I recently re-released a General Relativity package for Maple, GRTensorIII (https://github.com/grtensor/grtensor) that was part of my PhD work. The 1996 code is not that tragic, but is far from my best work - although I did earn my PhD.
People usually respond to how they are measured and rewarded. This applies to software written by graduate students. (Most professors code in English using students.)
So, I'm a professor and, like a lot of things, think there's truth to both sides.
Computer science is something that seems underrecognized in the field I work. People clearly acknowledge its importance, but then turn around and basically ignore it when talking to potential grad students or mentoring undergrads in prep for grad school. We don't offer any kind of course like "programming for X" even though most of the students need it.
My experience closely parallels the "why bad scientific code beats code following best practices." I've had comp sci students come in and what happens is they clearly understood python and java, but had difficulty understanding the problems with inheritance, and wrapping their head around other more functional languages we were using. They also were unfamiliar with the statistical/content areas, so had difficulty implementing things. I had thought it would be great to have comp sci students involved (and still do) but it didn't solve my problems like I thought--so instead of having students who understood the concepts but not the programming, now I had students who understood the programming but not the concepts.
When you're dealing with really intense math and statistics, it's difficult to separate out the programming from the math. It's not like web development where you have an "insert text here" kind of approach that works often; the algorithms and the problems are really wrapped up in one another. This might all be changing with data science DL and AI and that kind of stuff infiltrating comp sci's assumed territory, but I'm not really seeing it much so far.
It seems the prototypical situation in software design is some software that's team-developed for mass consumption. In science, you have the reverse often, which is software designed by small units that might be a one-off thing. These constraints put different kinds of pressures on the process, such as intense pressure on getting something to work correctly at all costs, including elegant design.
Also, the unit testing thing is kind of confusing to me. Every time discussion about a new language comes up in the context of numerical/scientific computing, one of the big questions is "does it have a REPL"? It seems one of the big reasons for doing this is basically unit testing. It might not be unit testing in the formal sense that you might have at some software design companies, but anything someone complicated involves feeding each tiny separable part of the code something with known expected output, sometimes in strange, boundary-testing ways, so that seems pretty similar to me. There's also a plethora of test-case datasets out there for this very purpose.
To me the bigger problem is homogenization in software in science, that is, a domain being dominated by a single piece of software. I think it leads to unrecognized errors due to lack of replication across implementations, and problems typical of monopolies (even when something is open source). There's a kind of development benefit:cost supply:demand problem that leads to dominance of single works of code that is really unhealthy for science (replicating with standardized methods is good too, but to me that's a slightly different issue).
Over the course of my career, I've had the job of working with environmental models of lake ecosystems, aided environmental/civil engineers in debugging the then state-of-the-art flood water analysis programs, given (brief) guidance to Ph.D. graduate engineering students on the programing for their dissertations, and worked with mechanical engineers that insisted on doing the real-time machine control programming themselves because the software guys took too long.
In each case, the people being given responsibility for the programs where in way over their heads. They were smart and educated and generally viewed programming as a skill somewhat akin to typing, something anyone could learn to do adequately, if not quickly, in a few days of practice.
The ecosystem simulation made unnecessary oversimplifications (assuming that an exponential relationship could be modeled as a linear one because the engineers didn't know how to handle the integration of an exponential(!!!) and how that should be handled in a program).
The flood control modeling program, used by the Federal government, was some of the worst code I had ever seen. It was written in a fashion where variables were treated a bit like a small finite set of registers. Ten or twelve global variables in the program were reused over and over for different purposes, sometimes to return computed answers, sometimes as temporaries inside of some function, and sometimes as iteration counters. It was a complete mess.
A graduate student that had never learned C or C++ was being given a previous grad student's C++ simulation code to use as the basis for his dissertation work. That code was pretty useless and poorly documented. Software engineering principles played no part in the work these students were doing, they were from a different discipline.
In the case of the real time machine control, programs were written in thousands of lines of assembly code and would control perhaps a dozen asynchronous activities though a combination of interrupt handlers, polling loops, and time outs. They were frustrated that the machines would simply hang every day or two. What a mess.
I think there is a perspective that has been overlooked in these comments. It is that the "professors' unprofessional programs" are probably very understandable to someone who deeply understands the science/engineering concepts the programs represent. Especially, if you get some history about what was developed first and what order features were added, the spaghetti starts to become much more understandable. Of course, after passing through the hands of multiple grad students the code should probably be refactored, but how many of you recode a big framework just because it isn't perfect for your situation.
I'm not a professor, but I started my career as an industrial engineering grad (MS, not PhD) who was writing code to build and solve a linear program for a supply chain application.
It was definitely one of those programs that does this, than that, the sometimes the other thing, and it went on forever. I had trouble understanding it myself.
I didn't end up rewriting it because, well, startups. The whole thing went kaput, and nah, didn't have much to do with the code.
I would agree with you that these programs are more understandable to someone with expertise, and that can be why it's so hard to refactor them. It's difficult to become this expert in a branch of science and also take the time to learn good software design and programming practices.
But overall, I'd say that these programs could become vastly more manageable and easier to understand through better programming practices.
Finding the right pitch point for someone to learn in the right way is really hard. People come to things like git and build systems from all sorts of angles.
I have a lot of sympathy for this. My first job out of school was working in a lab, writing a lot of software for them that was for cortical mapping. It was the kind of job where they expected you to joint a PhD program. You got to co-author papers. It was the first time I programmed a lot. I'm sure everything I wrote was pretty ugly.
[+] [-] greydius|8 years ago|reply
I could keep complaining, but I don't want to give people the impression that I don't like what I do. It beats the hell out of writing CRUD apps.
[+] [-] mikebenfield|8 years ago|reply
Another way of looking at it: People doing STEM PhDs are extremely busy. They have to master the content in their field, keep up with new research, conduct their own research, write papers, go to conferences, give talks, teach, keep up with whatever department service responsibilities they have, etc.
They're smart people so they're going to be able to slap together some code that does what they need it to do. But to expect them to learn git and how to use it well, to learn about managing dependencies, about unit tests, about build systems, about best practices for documenting code, about programming abstractions or OOP whatever else... that's a lot to ask.
(And I say this as a STEM PhD who did learn all that stuff, more or less.)
[+] [-] torchous|8 years ago|reply
I wish the state of affairs was prettier, but I doubt it will change any time soon. And frankly, I don't think it has to. Software engineers will always find something to complain about others' code (coding style discussions come to mind). If there's value in commercializing scientific prototypes, experts should rush in and do it. There's no need for us to invest in perfect code just in case someone may want to reuse it. Having said that, NSF is starting to fund initiatives to improve the state of affairs. For my field, it's EarthCube: https://www.earthcube.org/ but this seems to be fumbling along; all of their goals would need to be adapted by scientists outside of that community (standards, etc). Without the right incentives (funding, publications), it's just not going to happen.
[+] [-] keithpeter|8 years ago|reply
Can you identify (say) five things that academics can do with an existing code base that will make your job easier (and therefore cheaper to the downstream users)? Perhaps pitch the top five pick at the abilities of a computer science student looking for a project?
Can you identify (say) three key practices to adopt when starting a new project given the likely skill levels of postgrad students in your domain?
You could get an ebook out of those I imagine.
[+] [-] tbrownaw|8 years ago|reply
Good developers get paid enough, that it indicates that maybe our job isn't quite that easy to learn.
Our basic programming abstractions and software engineering practices are things that we as a field developed over time, based on experience; it's not like they're something you just automatically know without having to sink time into studying if you're above some IQ level. They also depend to varying degrees on how big your program is, which parts are expected to change more or less rapidly, what kind of program it is, etc.
Unit tests assume you know what the code is supposed to do before you write it, that the code will change significantly more often than the required functionality, and that the tests are easier to read and check than the code being tested.
[+] [-] britworst|8 years ago|reply
[+] [-] pjmorris|8 years ago|reply
[0] https://software-carpentry.org/
[+] [-] PeachPlum|8 years ago|reply
Funny, that sounds exactly like my IT Dept.'s approach to coding.
I work in Operations and was exposed to their approach when helping them translate some business rules into SQL for the warehouse management system. During the course of interaction the IT Director lost the code we had worked on together twice, keeps the requirements docs in his email etc. etc.
Me: your engineering practices are terrible
He: We don't have time to do it properly
Me: but somehow we have time to do it three times
I told my manager but no-one really understands or cares enough. The Warehousing Director calling out the IT Director about his approach to IT - where would that all end ?
[+] [-] sgt101|8 years ago|reply
[+] [-] segmondy|8 years ago|reply
Then there's the group of folks who tend to read those papers and turn it into products. Yes, those are the ones I expect to have repos, unit tests, build systems and all that.
[+] [-] Malice|8 years ago|reply
[+] [-] dasmoth|8 years ago|reply
This is far from an original thought:
In the good old days physicists repeated each other's experiments, just to be sure. Today they stick to FORTRAN, so that they can share each other's programs, bugs included.
[+] [-] pps43|8 years ago|reply
If you have original dataset and .Rmd file, you can then bisect both analyses to find the bug.
[+] [-] fabian2k|8 years ago|reply
Grants are time-limited, and at some point usually the money for developing the software runs out. The PhD students working on the software move on to something else, and you have yet another abandoned piece of scientific software.
There are of course exceptions, but in general it's much harder to get money for maintaining and improving software over a longer timeframe than for building something new.
[+] [-] roel_v|8 years ago|reply
This sounds negative but I don't mean it that way; it's just how it is, no judgement meant. But it's not something you really hear anyone teaching new PhD students as an option, and even if they would, it's highly uncertain and success depends on many factors out of your control. So I wouldn't call it a career path, or even viable career advice.
[+] [-] amelius|8 years ago|reply
PS: I couldn't read the article, presumably because of some anti-adblocker mechanism.
[+] [-] akuji1993|8 years ago|reply
> I couldn't read the article, presumably because of some anti-adblocker mechanism.
That is so stupid of news sites to do. They only thing that makes me do is leave your site immediately.
[+] [-] toyg|8 years ago|reply
[+] [-] plaidfuji|8 years ago|reply
Make code review part of peer review.
I think the next generation of scientists at least understand that things like unit tests are useful, but passing peer review is the only incentive to do anything, and reviewers don't read code.
[+] [-] newen|8 years ago|reply
[+] [-] UncleMeat|8 years ago|reply
I review a paper in an hour or so. If somebody comes to me with some fancy new symbolic execution framework it is going to take me days to review the code. Unless you are planning on paying me for a months work to review 20 papers, this is impossible.
[+] [-] musgravepeter|8 years ago|reply
I also cleaned up my own mess from 1996, when I recently re-released a General Relativity package for Maple, GRTensorIII (https://github.com/grtensor/grtensor) that was part of my PhD work. The 1996 code is not that tragic, but is far from my best work - although I did earn my PhD.
[+] [-] MurrayHill1980|8 years ago|reply
[+] [-] dasmoth|8 years ago|reply
http://yosefk.com/blog/why-bad-scientific-code-beats-code-fo...
[+] [-] belusidaty|8 years ago|reply
Computer science is something that seems underrecognized in the field I work. People clearly acknowledge its importance, but then turn around and basically ignore it when talking to potential grad students or mentoring undergrads in prep for grad school. We don't offer any kind of course like "programming for X" even though most of the students need it.
My experience closely parallels the "why bad scientific code beats code following best practices." I've had comp sci students come in and what happens is they clearly understood python and java, but had difficulty understanding the problems with inheritance, and wrapping their head around other more functional languages we were using. They also were unfamiliar with the statistical/content areas, so had difficulty implementing things. I had thought it would be great to have comp sci students involved (and still do) but it didn't solve my problems like I thought--so instead of having students who understood the concepts but not the programming, now I had students who understood the programming but not the concepts.
When you're dealing with really intense math and statistics, it's difficult to separate out the programming from the math. It's not like web development where you have an "insert text here" kind of approach that works often; the algorithms and the problems are really wrapped up in one another. This might all be changing with data science DL and AI and that kind of stuff infiltrating comp sci's assumed territory, but I'm not really seeing it much so far.
It seems the prototypical situation in software design is some software that's team-developed for mass consumption. In science, you have the reverse often, which is software designed by small units that might be a one-off thing. These constraints put different kinds of pressures on the process, such as intense pressure on getting something to work correctly at all costs, including elegant design.
Also, the unit testing thing is kind of confusing to me. Every time discussion about a new language comes up in the context of numerical/scientific computing, one of the big questions is "does it have a REPL"? It seems one of the big reasons for doing this is basically unit testing. It might not be unit testing in the formal sense that you might have at some software design companies, but anything someone complicated involves feeding each tiny separable part of the code something with known expected output, sometimes in strange, boundary-testing ways, so that seems pretty similar to me. There's also a plethora of test-case datasets out there for this very purpose.
To me the bigger problem is homogenization in software in science, that is, a domain being dominated by a single piece of software. I think it leads to unrecognized errors due to lack of replication across implementations, and problems typical of monopolies (even when something is open source). There's a kind of development benefit:cost supply:demand problem that leads to dominance of single works of code that is really unhealthy for science (replicating with standardized methods is good too, but to me that's a slightly different issue).
[+] [-] todd8|8 years ago|reply
In each case, the people being given responsibility for the programs where in way over their heads. They were smart and educated and generally viewed programming as a skill somewhat akin to typing, something anyone could learn to do adequately, if not quickly, in a few days of practice.
The ecosystem simulation made unnecessary oversimplifications (assuming that an exponential relationship could be modeled as a linear one because the engineers didn't know how to handle the integration of an exponential(!!!) and how that should be handled in a program).
The flood control modeling program, used by the Federal government, was some of the worst code I had ever seen. It was written in a fashion where variables were treated a bit like a small finite set of registers. Ten or twelve global variables in the program were reused over and over for different purposes, sometimes to return computed answers, sometimes as temporaries inside of some function, and sometimes as iteration counters. It was a complete mess.
A graduate student that had never learned C or C++ was being given a previous grad student's C++ simulation code to use as the basis for his dissertation work. That code was pretty useless and poorly documented. Software engineering principles played no part in the work these students were doing, they were from a different discipline.
In the case of the real time machine control, programs were written in thousands of lines of assembly code and would control perhaps a dozen asynchronous activities though a combination of interrupt handlers, polling loops, and time outs. They were frustrated that the machines would simply hang every day or two. What a mess.
[+] [-] c517402|8 years ago|reply
[+] [-] geebee|8 years ago|reply
It was definitely one of those programs that does this, than that, the sometimes the other thing, and it went on forever. I had trouble understanding it myself.
I didn't end up rewriting it because, well, startups. The whole thing went kaput, and nah, didn't have much to do with the code.
I would agree with you that these programs are more understandable to someone with expertise, and that can be why it's so hard to refactor them. It's difficult to become this expert in a branch of science and also take the time to learn good software design and programming practices.
But overall, I'd say that these programs could become vastly more manageable and easier to understand through better programming practices.
[+] [-] SeanDav|8 years ago|reply
> "Those who do put effort into producing good code risk being seen by their colleagues as time-wasters."
Producing good code takes time and effort. It would seem a complete culture shift is required before any significant changes will happen.
[+] [-] zwischenzug|8 years ago|reply
I wrote this course as a basis:
http://learngitthehardway.tk/learngitthehardway.pdf
Finding the right pitch point for someone to learn in the right way is really hard. People come to things like git and build systems from all sorts of angles.
[+] [-] georgeecollins|8 years ago|reply
[+] [-] chris_wot|8 years ago|reply
[+] [-] gaius|8 years ago|reply
[+] [-] klez|8 years ago|reply
But maybe a (2015) in the title would be a good idea.
[+] [-] j45|8 years ago|reply
Both appear to be hyper focused on solving the problem that little else matters.
[+] [-] roel_v|8 years ago|reply
[+] [-] eric_bullington|8 years ago|reply
Thanks! Do you do this kind of work as a contractor, presumably?
Do you also typically write documentation for the projects you're working on (inside from inline comments)?
How did you get started in this line of work?
[+] [-] imperssonator|8 years ago|reply
https://gtfiber.github.io
Follow-up: say I have a lot of free time to patch it up, write better documentation, unit tests, etc. What's the first, most value-added thing to do?
[+] [-] chubot|8 years ago|reply
[+] [-] honestoHeminway|8 years ago|reply
[deleted]