This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude.
Most of most jobs is a bunch of mundane stuff. I've seen it in software development, and I've seen it in management consulting. The best people, typically, are those that will happily do both, understanding that the fun stuff comes with a lot of baggage.
The "someone else is better at the stuff I don't want to do than me" argument rarely holds up either. The friction that comes from dividing the work along lines like modeling and production and trying to hand off is rarely worth it when one person can do both.
Anyway, I've been where the author is, but personally I think it's wishful thinking, unless maybe you want to start your own shop and structure it around yourself that way.
I dunno... I am a software/data engineer who partners with data scientists. I think that comparative advantage here is a real and important. Don't get me wrong, I'm happy when my data scientist partners write good code or show interest in getting better, but I'm more than happy to take their janky code and make it production ready. It often needs to be optimized for scale or refactored for reusability, and a lot of that often falls very solidly in the engineering discipline.
I'll often tell my DS partners, "Don't worry about the code; you get the math right then give it to me. You can go about doing more math, I'll do the engineering." In my experience, this is often a really pragmatic division of labor.
This is also important because often, the data scientists are never on-call, so if something breaks in production, engineering needs to know what is going on.
That’s what makes technical interviews even more frustrating. Most jobs day to day are like 5-10% solving deep technical puzzles and 90-95% fiddling with tooling and automating things that are incredibly frustrating or time consuming.
So ya I’m not sure how to solve your silly brain teaser, but I have written custom test frameworks to automate the tedium away to save the team hundreds of man hours. It took a deep understanding of VERY specific tools (shout out to ASIC EDA tools). But that doesn’t matter on a technical interview. So you hire somebody who can Leetcode, but can’t figure out how to fit all the pieces of the actual job together.
I totally agree. I work on the private sector, coming from a research position too. I was also focused on the "interesting" side of the problem: the modeling, integrating domain knowledge into the analysis, drawing all sorts of plots... But there were other unavoidable and "uninteresting" needs for the research project, like building a data gathering system with its API and everything. This required my best software engineering abilities. Needless to say my best weren't precisely THE best, so as the project got bigger, the not-so temporary fixes increased, as well as poor design choices (if any). This finally led to a complete reestructure and almost fresh start.
I feel some of it could be avoided, so I learned the hard way that the whole modelling + software engineering process is a subtle craft. It is important to take care on the implications of your code and, specially, on how its done, since it may fall back onto you eventually. This reconciled me with the more technical stuff (my tools) to eventually put up a good work in a more satisfying way.
I think there is value in both and it sort of depends on the organization.
With the commoditization of models, we are seeing the rise of MLEs over data scientists. Engineers that understand enough DS to make things work are wildly proficient in this space.
However, not all models have been commoditized, and there is still a need for new math in many places and that’s where the division of labor makes sense. You can’t be an all star engineer and an all star data scientist it’s just too much for one human
I have a name for the sort of mundane-yet-employable programming tasks. "Plumbing work". You're not doing the clever problem solving that once sucked you into programming, you're welding pipes together that other people made.
>The "someone else is better at the stuff I don't want to do than me" argument rarely holds up either. The friction that comes from dividing the work along lines like modeling and production and trying to hand off is rarely worth it when one person can do both.
This was always my attitude. Every time you split something you add coordination overhead. This overhead gets worse the more times you split.
Of course there is specialization that can make someone else sufficiently more effective that you shouldn't do everything.
But every gain in specialization has to be weighed against increased communication costs.
Add to that a lot of problems don't need a lot of specialization but touch on many different disciplines.
This is why I think, contrary to the trend of specialization, we need generalists that can cover most bases at once and decrease the communication overhead considerably.
It's often still good to have specialists, but you should mostly employ generalists and only a few specialists in key technologies that set your company apart from others.
Well, in purely software shops, there's often people dreaming up the 'what to do' and a different group of people actually writing the code. Same with systems design, we have 'architects.'
This should be no different for statistical modelers or other disciplines. Employers are just cheap.
It is not that surprising because software engineers are usually more expensive than scientists. That leads business people to ask serious questions about which staff they really need, how much of that work can be done by lower-paid specialists, and whether they really need the professional code quality when the stuck-together python that scientists tend to write usually also works.
To all the specialists out there, if you value job security, become a specialist that can produce. When the company I'm in did layoffs, the area that had the biggest layoffs were in groups that did not actively produce the product we were selling.
I love boring stuff because often it allows me to do something really well. Hard problems are fun but hard to solve well by their nature. Programming is fun because I get to do both of these things.
Having done both sides of the work, I totally understand the author’s perspective. Modelling and coding production are certainly two different skillsets, and many people will prefer one or the other, but not both. Now, there is one thing I guess the author doesn’t get (or he gets but doesn’t like) and a thing that most companies don’t get.
The former is that companies don’t care enough about what the individuals they hire prefer - especially not if it doesn’t address their need. They don’t need a beautifully crafted model that can’t be run - they want actionable results that hit the bottom line, and in the energy forecasting model that means generating new forecasts at every cycle (months, days, hours, minutes - whatever). A great model that can’t be put in production and run efficiently has about the same value as no model, and an average model in production will have much more impact.
The latter is that companies don’t usually understand that ML is not software development. Putting ML in production is, but finding the right model is research work, and code is mostly a discardable tool for research. Its goal is not to go live, it is to validate a hypothesis (in this case, that algorithm X, when presented with data Y, generates a model with enough predicting power to be useful to the business). This validation requires code to gather Y, clean/join/analyze/reshape/featurize it into a more informative and clean representation (clean from the algorithm’s perspective, not necessarily a human’s), run X, run inference with the generated model and run some test of the results against additional data.
If this test is negative, some or all the code written above is useless, and we go back to the drawing board. Given the very coupled nature of this process (a new data source has to be joined to the rest - coupling; a new data transformation changing a feature changes the data schema downstream - coupling; a new algorithm needs a different data input - coupling, and so it goes). If you have an experience like mine, you may actually be able to write in a way where you can reuse some of it, but I have 28 years of experience with data, there are simply not enough people in the market with that level of background or the interest in learning all this. Companies must accept that they will not always get this perfect candidate with all the skills they want, and start thinking of pairing the right people in teams.
Some who have been around for a while may remember the Venn diagram of the perfect Data Scientists - it usually was an intersection of business, math and programming skills (also often communication skills and a few others). My thoughts since I first saw this diagram were: “Even if there are people out there with all these skills, why would they want to work for others?”
This, more than anything else, is my guess at the core reason why so few companies are successful in putting ML in production.
Everybody wants to work on the fun stuff. Turns out in software, getting something to actually deliver value (meaning convert it from prototype or proof of concept to something running in production) is 90% not fun stuff. Most people learn that in their first job.
As a software engineer the only way I’d become a “data scientist/trader’s servant” is if I’m getting paid exorbitantly. Otherwise it’s the worst kind of work because someone else is going to get credit for everything that goes right while I take the flak for all the hard stuff.
The converse is that to get a software engineer to be your servant you better also be really good, or else you’re probably going to end up paired with someone who doesn’t have the luxury of taking other jobs and maybe won’t be particularly good at getting your stuff to actually run.
One of the first lessons that a junior developer learns is:
>My job is not to write code. My job is to solve business problems efficiently, using computers as a medium.
Sounds like data scientists have the same problem. Sometimes, to solve business problems efficiently, you have to step out of your comfort zone and learn to do things that seem ridiculous.
Why just the other day, I had to write a report for humans to read. Using words and pictures! What am I, a technical writer?
I strongly disagree, most software engineers will not have excellent knowledge of modelling, and someone who is mainly a data scientist will not be able to compete with a dedicated engineer that studied distributed systems, kubernetes, and knows undefined C behaviour by heart.
When building a house, the guy that lay concrete are welders are different people, Anastesiologists and surgeons are not interchangeable, etc.
Many Industries will produce code, just like many people often need to nail two wooden bits together, but that does not make them a qualified structural engineer.
everyone differs on what the "fun stuff" is. i would find doing data modelling and statistics unbearably tedious, but quite enjoy taking working code and refactoring it to be performant, maintainable and well tested.
The modeler who can write code unlike you will get hired first.
Okay, now that I said the provocative thing to kind of drive home how real and serious this point is, I will say that I am not a statistician, but I run more computational physics simulations, so less statistical modelling but modelling of experiments and systems based on the PDEs. The one thing I observe is that there really is only this patience for this lack of understanding your own tools for theorists. Experimentalists can fix their own tools, they can open up the casing and resolder the boards if they need to, heck most of them can fix their own cars. But you have no idea how many computational scientists just load up Lumerical or Ansys and just click around but really have no concept or idea how it works under the hood beyond just things they show on intro slides to talks. Some know how to script say Meep or something if they're good but they've never implemented a DE solver themselves unless it was in a class in college or first year grad school then they forgot it all.
You really only have this disconnect from your own tools for theorists. Programming is your breadboard, your substrate. Code is the material you use to do your work. I don't understand why it is okay for theorists of all stripes to slide on never really understanding how their on research actually works on a computer whereas every experimentalist I've ever known could recreate their entire experimental apparatus from scratch if they were paid to do so. But that's okay, because that means as long as too many theorists can only write equations and then have to have someone to hand hold them so they actually do the things they've written down, I will be valuable and have job opportunities for myself. It would however make life easier for myself and lessen the many headaches I've been subject to, and heck, may be science could move forward a little better, yadda yadda.
I probably shouldn't encourage my competition like that especially when they're injuring themselves but that "move science forward and lessen my headaches" vibe does make me want to share the sentiment so that theorists at least understood on some level how the libraries they import work sometimes.
Computation is not theory. Theorists use paper and pencil and sometimes mathematica, not simulation. As for not being able to rebuild a computer, that’s truly ridiculous.
I loved Python. It was easy to learn, very powerful, has libraries for everything...
Then I started supporting researchers and scientists who wrote "python code" to run simulations etc.
Most of it's pretty basic, install some scientific code published by some research group. They chuck their data in and run it.
But then they started abusing virtual environments, writing their own code, cutting and pasting, commenting out random lines because they saw someone else "fix" something that way... and they all want Jupyter notebooks.
Now it's like an eternal September plus I get to deal with annoyingly slow package managers like Conda and rough academic projects with poor documentation and little testing.
Having been in the same situation as you, you have to bring some technical leadership if you want to change things. And some empathy, these people are stuck in a local maximum, they try their best probably with limited tools and knowledge.
Conda is a shitshow for sure: teach them how things could be better.
Notebooks are a reproducibility nightmare: show them how nice eg. Pycharm is
People are unqualified to write O(n) code in less than O(n^3) time: identify skill gaps and get their management to sign off on some software development / python training.
If you can make a convincing case why your approach is better, you can make everyone's life better.
Conda may suck in some ways but it solved a big problem; installing software without root accesss. It effectively removed os dependencies allowing for operating system changes. HPC operating systems were effectively ossified prior to that.
Notebooks on the other hand are a god damned abomination. Obligatory past discussion;
That sounds similar to my experience with the Python ecosystem (of which I have a profound dislike at the moment). If you haven't tried Mamba yet I encourage you to give it a shot; for me it has had a pretty significant impact on the time I have to spend sitting around waiting for Conda packages to install.
The situations such as improving package management in python is a problem but fundamentally in any software stack you will eventually find bad code. Especially as the code base gets larger and people move on.
Instead you can try to fight it by teaching the rest of the group how to write more idiomatic code but that only goes so far. Also training and documentation can go out the window when the objective is to ship something that works once.
> every one of them wants Python. I haven’t seen a single one where they’re looking for R or even C++; Python rules this roost.
Tried putting R into production recently? It’s a frustrating and brittle experience. Don’t get me wrong, R is fantastic at what it does - analysis, research, statistics, and arguably the API’s on the R data frame packages are a lot saner than Pandas.
C++ is out for different reasons I suspect. As this touches on, “modellers” (and data scientists/engineers) ought to be decent developers, but a lot of them are not, and in my experience actively refuse to learn any of these skills, the comfort zone is “jupyter notebooks” and that’s it. Getting them to write C++ (disregarding language debates), a language that is unequivocally more difficult and fraught with complexity than Python is basically a non-starter.
Do I wish it was different? Yep. Do I wish there was some more variety in the “language ecosystem” so it’s more than just the “lowest common denominator Python” dominance? Absolutely.
>Tried putting R into production recently? It’s a frustrating and brittle experience.
Oh man, can I sign onto this rant. While Python has spent a decade+ trying-and-failing to standardize on one of a dozen tools to properly manage dependencies, at least it is trying. R is still global-namespace, no-pinning by default. Sorta-kinda you can squint where renv is going, but still needs a lot of development.
I had some hopes of Julia stealing the R mindshare and righting some of the more egregious wrongs in that ecosystem, but (as an outsider) it feels like Julia has lost a lot of steam.
> I’d much rather get something working and then hand it off to someone else who can refactor it for speed and clarity, and have it conform to the desired style conventions, etc. etc.
I've been on the coding end of this - when everyone actually has those fixed roles and goes into it eyes wide open, it goes pretty well! When instead the PhD is assigned to go do the feature, and then the programmer is called in later when it's not implemented well, it tends to go quite poorly and nobody is happy, as everyone's time is wasted.
I'm a modeler and best described as a lifelong scientific programmer. I'm much, much better at doing the specialty science I was trained to do and have done than write unit tests (for Jupyter notebooks? why?) or struggle through big-O questions (again, why? I write ~100 line programs that are never production code). Like the author of the post, I am not a professional developer and don't pretend to be or want to be. There are people out there way better than me for doing those jobs.
Recently, I interviewed for a role as a computational chemist - this is an ideal fit for a person like me with the domain knowledge, advanced degree in the subject area, passion, and a proven (if dated) track record of publishing in the domain. The idea is to use software, amend what's there if/when needed, and apply my knowledge to the process and what comes out of it.
What did the interview start with? A surprise interactive coding challenge that I wasn't prepared for, and thankfully the interviewer was kind and professional enough to understand that my value-add is in, well, computational chemistry and not in the details covered by a CS major.
I thought Jesus, I bet not a single software engineer at this company got asked to do a simple organic synthesis or even a redox problem during their interview process.
> I bet not a single software engineer at this company got asked to do a simple organic synthesis or even a redox problem during their interview process.
Well they should have! One of the most important traits for a good software engineer is to be able to pick up a working understanding of the domain they're programming in. If they can't do that, they're more likely to make negative contributions than any positive ones.
I might be reading too much into your story, but if your CV indicates that you're "much better at doing the specialty science I was trained to do and have done than" CS stuff, a good interviewer might be concerned whether you had the necessary CS knowledge / programming skills.
To me at least, an interview is not to flatter the candidate by asking questions they're most comfortable with, but to make sure they have the requisite skills expected for the job. If the job expects X, Y and Z, and they seem like they're good at X and Y, you spend the time checking whether they're reasonably competent at Z as well.
If OP wants to make models but never worry about them after doing the fun bits, it sounds like they might enjoy academia. The industry premium salary is in part from doing all the work around the “fun part”. As others have mentioned that’s where a lot of the value is, and nobody wants to be your servant.
Though even in academics, you have to write the paper yourself after doing the fun bits.
> If a company says that they need excellent Python skills, and they mean it, then I’m not the right person for that job.
Eh, let the company decide that. That's one of the biggest things I tell my buddy who thinks he's bad at programming and really doesn't want to have to look for a job. He's convinced he's horrible but I don't think he is. He just doesn't want to get rejected, so he just avoids interviews as much as possible and tries to stay at the company he's at.
I would imagine that most companies are okay with python code that is rough and ready as long as it can be integrated by the rest of the team into the production code. Most data programmers are probably expected to be better at the data than the programming.
He should find a SWE and partner up. As more clients enter the ML space there are more less sophisticated ones who has less of a concept of cross functional teams. Rather than educate themselves these clients have unrealistic demands.
While I am a huge advocate of DSea writing better code going fullstack is unrealistic for them. What does it even mean to integrate? Fastapi? Docker? Helm charts? Monitoring and observability? SRE? The list is endless.
This is a classic case of "the client doesn't care the want business value".
If as a DS you want to get better at writing code join our Code Quality for Data Science (CQ4DS) discord:
As someone who was naively hired as a "data-scientist" without my employer checking if I actually knew anything about modeling I ended up falling into the "person who fixes up the code, maintains tests it, etc." role, while my seniors, who were much less enthused about software engineering, were the ones pushing out models that were admittedly very impressive but complete disasters in terms of code.
We worked very well off of each other, it was interesting to pick at a model from a software engineering perspective, how the code could be structured and improved, where some tradeoffs would need to be made and how we would test and verify if it actually worked for our users. I eventually left because the company was more concerned with getting new models out as soon as possible regardless of their actual performance, but it did ignite my passion for software engineering and devops.
> Python doesn’t yet have anything remotely close to ggplot for rapidly making exploratory graphics, for example.
Plug for plotnine (https://plotnine.readthedocs.io/en/stable/). I don't know R but use ggplot indirectly through this library for exploratory data analysis, and comparing the experience to any other python plotting library, I understand why R folks are usually so sad to be using Python.
I think nobody here really pointed out a very relevant issue that's completely widespread at least with tech job market: companies don't want to pay people.
You don't see this kind of problem in other established professions, you don't expect an accountant to be able to perform the job of a lawyer neither you expect a nurse to be able wear the hat of a nutritionist.
Now with the technological professions, let's use the term knowledge professions as an umbrella term, companies take advantage of the fact that these professions have not been around for that long and are not that established to keep expanding their rol of responsibilities.
We see that all the time with tech companies. It's not rare that you're supposed to know the frontend, backend, testing, devops, some of them even domain knowledge and the list keeps expanding even though sometimes they entail different sets of skills. The salary, not surprisingly, doesn't grow proportionally to the list of requirements. Companies don't want specialized people anymore, they want someone who will quickly pick the job of other people when/if they finish theirs.
That's what I believe the author's rant was about. He has been looking for a job in his field, he is not a software engineer. Yet people are expecting him to be a professional developer on the top of being a professional data modeler.
> you don't expect an accountant to be able to perform the job of a lawyer neither you expect a nurse to be able wear the hat of a nutritionist.
You'd be surprised. I mean, they're not competent to do so professionally, so they can't formally give professional advice outside of their field of competence, but you can be sure people do ask them such questions, and get disappointed if the answer is "that's not my job".
Not sure that I would recommend actually hiring that guy. He does not seem to understand that modelling is only one part of the equation. If you model something non-trivial, usually you can not just hand over your R scripts to someone else and say: Please implement that in python/C. You have implementation constraints which feed back into the modelling itself, like latency or scalability. Furthermore, good luck in letting someone translate your non-trivial math into another language hoping that he won't break it in some subtle or non-subtle way. Its just far more efficient if you have someone who can actually do the prototype directly in python or C, and let then a pro-developer optimize specific parts of the code.
People who are experts at their domain are not always good at explaining things to others. This means that if a data scientist does not know how to code, they need to partner with a programmer and be able to explain things to them. Same goes for a bunch of other professions. For some people, it is easier to learn to code than it is to learn to communicate their ideas with people.
If you are not going to be able to implement something, whether in code, or with a saw and hammer, you must be able to explain things really well. If you can not do either, you will have limited ability to apply your craft.
modeling is 1-2% of the overall work involved in having a model serve customers something useful. There's just so much other work to do that it rarely makes sense to have people who can only model. At some level of scale, building out infrastructure to support people to model full time and constantly run a/b tests make sense but that is not the vast majority of use cases for ML in the real world and building out that support infrastructure is a huge investment
My dad got me into computers, but he's not a programmer by trade. He's a mechanical engineer, and he used BASIC on a TRS-80 to do engine simulations to characterize the mechanical forces on the crankshaft for different crank types. Back then, BASIC was basically what MATLAB is today for engineers. They just slammed some code in there to do the math they needed to do and ran it. They didn't care about things we care about: testability, maintainability, observability.
Similarly, Judge Alsup of Oracle v. Google fame writes astronomy software in QBasic. He doesn't give a shit about best practices, if it helps him aim his telescope correctly it's all good.
Welcome to a world with citizen programmers. A world of terrible code that does the job. I frickin' love it.
Yes, that’s expected of every scientist I am working with. The reason is quite simple and have little to do with Engineering work being more expensive than Scientific one (it’s actually the opposite): get a nice problem to solve or a great model to build, and very soon, you have an ivory tower completely disconnected from the original objective.
To be fair, I don’t ask scientists to understand concurrency issues in programming. Only the basic stuff that is required for delivering a functional program. Yes, Pandas and Scikit-learn belong to the basic stuff.
To be honest I don't believe this to be isolated to just modelling and production code, I feel as though this issue presents itself in most of the job ads I see today, Backend roles with requirements for React, Python roles with requirements for Javascript, Backend roles with requirements for DevOps/SRE.
I don't necessarily have an issue with widely skilled engineers, but I would prefer it's for the right reasons, and I largely believe that it's an exercise in laziness on most companies behalf. They just want to hire less people and have more of their workers do tasks that are outside of their remit.
I have zero interest in writing Javascript, absolutely none, I don't want to do it, and I have pushed my career in directions that mean that for the most part, I don't have too. I'm happy with this decision and have made it willingly.
It's the same with a lot of "DevOps" tasks, having previously been a DevOps Engineer, I now just want to write code, real code, but it feels as though most places now are just not hiring DevOps/Infra people, and just telling their other engineers to do it, which I understand, but it results in a far worse experience for both sides of it. I have to regularly force my hand down from volunteering for things that I have the technical experience to do, and do properly, versus colleagues that don't have the experience, because I'm tired of being shoehorned back into a role that I intentionally left. All of this is because the idea of "cross functional teams" no longer means hiring specialised engineers to do specialised roles, and just getting everyone to do everything, and then being surprised when the context switch penalty actually exists, and it's done to a worse standard than someone who is skilled in that role.
I know exactly what you feel like and I used to be exactly like you. Even down to preferring R to Python.
All I can say is: Bite the bullet. Learn Python, forget about R. R is nice and all but there is nothing that couldn't also be done in Python.
What helped me is: Learn to appreciate the beauty in actual coding, in deployments in environments in well structured, maintainable code. In scaling issues, databases etc.. There is an endless world out there which is extremely fascinating as soon as you get over the "all I want to do is modeling" mindset.
Good luck. You can definitely do it, because I did it as well.
[+] [-] version_five|3 years ago|reply
Most of most jobs is a bunch of mundane stuff. I've seen it in software development, and I've seen it in management consulting. The best people, typically, are those that will happily do both, understanding that the fun stuff comes with a lot of baggage.
The "someone else is better at the stuff I don't want to do than me" argument rarely holds up either. The friction that comes from dividing the work along lines like modeling and production and trying to hand off is rarely worth it when one person can do both.
Anyway, I've been where the author is, but personally I think it's wishful thinking, unless maybe you want to start your own shop and structure it around yourself that way.
[+] [-] mynameisash|3 years ago|reply
I'll often tell my DS partners, "Don't worry about the code; you get the math right then give it to me. You can go about doing more math, I'll do the engineering." In my experience, this is often a really pragmatic division of labor.
This is also important because often, the data scientists are never on-call, so if something breaks in production, engineering needs to know what is going on.
[+] [-] polalavik|3 years ago|reply
So ya I’m not sure how to solve your silly brain teaser, but I have written custom test frameworks to automate the tedium away to save the team hundreds of man hours. It took a deep understanding of VERY specific tools (shout out to ASIC EDA tools). But that doesn’t matter on a technical interview. So you hire somebody who can Leetcode, but can’t figure out how to fit all the pieces of the actual job together.
[+] [-] dollo_7|3 years ago|reply
I feel some of it could be avoided, so I learned the hard way that the whole modelling + software engineering process is a subtle craft. It is important to take care on the implications of your code and, specially, on how its done, since it may fall back onto you eventually. This reconciled me with the more technical stuff (my tools) to eventually put up a good work in a more satisfying way.
[+] [-] mountainriver|3 years ago|reply
With the commoditization of models, we are seeing the rise of MLEs over data scientists. Engineers that understand enough DS to make things work are wildly proficient in this space.
However, not all models have been commoditized, and there is still a need for new math in many places and that’s where the division of labor makes sense. You can’t be an all star engineer and an all star data scientist it’s just too much for one human
[+] [-] IIAOPSW|3 years ago|reply
[+] [-] carlmr|3 years ago|reply
This was always my attitude. Every time you split something you add coordination overhead. This overhead gets worse the more times you split.
Of course there is specialization that can make someone else sufficiently more effective that you shouldn't do everything.
But every gain in specialization has to be weighed against increased communication costs.
Add to that a lot of problems don't need a lot of specialization but touch on many different disciplines.
This is why I think, contrary to the trend of specialization, we need generalists that can cover most bases at once and decrease the communication overhead considerably.
It's often still good to have specialists, but you should mostly employ generalists and only a few specialists in key technologies that set your company apart from others.
[+] [-] linuxftw|3 years ago|reply
This should be no different for statistical modelers or other disciplines. Employers are just cheap.
[+] [-] User23|3 years ago|reply
That’s why we call it “work.”
[+] [-] whatshisface|3 years ago|reply
[+] [-] friedman23|3 years ago|reply
[+] [-] unknown|3 years ago|reply
[deleted]
[+] [-] roflyear|3 years ago|reply
[+] [-] bobbruno|3 years ago|reply
The former is that companies don’t care enough about what the individuals they hire prefer - especially not if it doesn’t address their need. They don’t need a beautifully crafted model that can’t be run - they want actionable results that hit the bottom line, and in the energy forecasting model that means generating new forecasts at every cycle (months, days, hours, minutes - whatever). A great model that can’t be put in production and run efficiently has about the same value as no model, and an average model in production will have much more impact.
The latter is that companies don’t usually understand that ML is not software development. Putting ML in production is, but finding the right model is research work, and code is mostly a discardable tool for research. Its goal is not to go live, it is to validate a hypothesis (in this case, that algorithm X, when presented with data Y, generates a model with enough predicting power to be useful to the business). This validation requires code to gather Y, clean/join/analyze/reshape/featurize it into a more informative and clean representation (clean from the algorithm’s perspective, not necessarily a human’s), run X, run inference with the generated model and run some test of the results against additional data.
If this test is negative, some or all the code written above is useless, and we go back to the drawing board. Given the very coupled nature of this process (a new data source has to be joined to the rest - coupling; a new data transformation changing a feature changes the data schema downstream - coupling; a new algorithm needs a different data input - coupling, and so it goes). If you have an experience like mine, you may actually be able to write in a way where you can reuse some of it, but I have 28 years of experience with data, there are simply not enough people in the market with that level of background or the interest in learning all this. Companies must accept that they will not always get this perfect candidate with all the skills they want, and start thinking of pairing the right people in teams.
Some who have been around for a while may remember the Venn diagram of the perfect Data Scientists - it usually was an intersection of business, math and programming skills (also often communication skills and a few others). My thoughts since I first saw this diagram were: “Even if there are people out there with all these skills, why would they want to work for others?”
This, more than anything else, is my guess at the core reason why so few companies are successful in putting ML in production.
[+] [-] opportune|3 years ago|reply
As a software engineer the only way I’d become a “data scientist/trader’s servant” is if I’m getting paid exorbitantly. Otherwise it’s the worst kind of work because someone else is going to get credit for everything that goes right while I take the flak for all the hard stuff.
The converse is that to get a software engineer to be your servant you better also be really good, or else you’re probably going to end up paired with someone who doesn’t have the luxury of taking other jobs and maybe won’t be particularly good at getting your stuff to actually run.
[+] [-] plastiquebeech|3 years ago|reply
>My job is not to write code. My job is to solve business problems efficiently, using computers as a medium.
Sounds like data scientists have the same problem. Sometimes, to solve business problems efficiently, you have to step out of your comfort zone and learn to do things that seem ridiculous.
Why just the other day, I had to write a report for humans to read. Using words and pictures! What am I, a technical writer?
[+] [-] ClumsyPilot|3 years ago|reply
When building a house, the guy that lay concrete are welders are different people, Anastesiologists and surgeons are not interchangeable, etc.
Many Industries will produce code, just like many people often need to nail two wooden bits together, but that does not make them a qualified structural engineer.
[+] [-] zem|3 years ago|reply
[+] [-] noobermin|3 years ago|reply
Okay, now that I said the provocative thing to kind of drive home how real and serious this point is, I will say that I am not a statistician, but I run more computational physics simulations, so less statistical modelling but modelling of experiments and systems based on the PDEs. The one thing I observe is that there really is only this patience for this lack of understanding your own tools for theorists. Experimentalists can fix their own tools, they can open up the casing and resolder the boards if they need to, heck most of them can fix their own cars. But you have no idea how many computational scientists just load up Lumerical or Ansys and just click around but really have no concept or idea how it works under the hood beyond just things they show on intro slides to talks. Some know how to script say Meep or something if they're good but they've never implemented a DE solver themselves unless it was in a class in college or first year grad school then they forgot it all.
You really only have this disconnect from your own tools for theorists. Programming is your breadboard, your substrate. Code is the material you use to do your work. I don't understand why it is okay for theorists of all stripes to slide on never really understanding how their on research actually works on a computer whereas every experimentalist I've ever known could recreate their entire experimental apparatus from scratch if they were paid to do so. But that's okay, because that means as long as too many theorists can only write equations and then have to have someone to hand hold them so they actually do the things they've written down, I will be valuable and have job opportunities for myself. It would however make life easier for myself and lessen the many headaches I've been subject to, and heck, may be science could move forward a little better, yadda yadda.
I probably shouldn't encourage my competition like that especially when they're injuring themselves but that "move science forward and lessen my headaches" vibe does make me want to share the sentiment so that theorists at least understood on some level how the libraries they import work sometimes.
[+] [-] onos|3 years ago|reply
[+] [-] bluedino|3 years ago|reply
Then I started supporting researchers and scientists who wrote "python code" to run simulations etc.
Most of it's pretty basic, install some scientific code published by some research group. They chuck their data in and run it.
But then they started abusing virtual environments, writing their own code, cutting and pasting, commenting out random lines because they saw someone else "fix" something that way... and they all want Jupyter notebooks.
Now it's like an eternal September plus I get to deal with annoyingly slow package managers like Conda and rough academic projects with poor documentation and little testing.
[+] [-] isoprophlex|3 years ago|reply
Conda is a shitshow for sure: teach them how things could be better.
Notebooks are a reproducibility nightmare: show them how nice eg. Pycharm is
People are unqualified to write O(n) code in less than O(n^3) time: identify skill gaps and get their management to sign off on some software development / python training.
If you can make a convincing case why your approach is better, you can make everyone's life better.
[+] [-] mattpallissard|3 years ago|reply
Notebooks on the other hand are a god damned abomination. Obligatory past discussion;
https://news.ycombinator.com/item?id=17856700
[+] [-] Tcepsa|3 years ago|reply
[+] [-] zitterbewegung|3 years ago|reply
Instead you can try to fight it by teaching the rest of the group how to write more idiomatic code but that only goes so far. Also training and documentation can go out the window when the objective is to ship something that works once.
[+] [-] unknown|3 years ago|reply
[deleted]
[+] [-] FridgeSeal|3 years ago|reply
Tried putting R into production recently? It’s a frustrating and brittle experience. Don’t get me wrong, R is fantastic at what it does - analysis, research, statistics, and arguably the API’s on the R data frame packages are a lot saner than Pandas.
C++ is out for different reasons I suspect. As this touches on, “modellers” (and data scientists/engineers) ought to be decent developers, but a lot of them are not, and in my experience actively refuse to learn any of these skills, the comfort zone is “jupyter notebooks” and that’s it. Getting them to write C++ (disregarding language debates), a language that is unequivocally more difficult and fraught with complexity than Python is basically a non-starter.
Do I wish it was different? Yep. Do I wish there was some more variety in the “language ecosystem” so it’s more than just the “lowest common denominator Python” dominance? Absolutely.
[+] [-] fbdab103|3 years ago|reply
Oh man, can I sign onto this rant. While Python has spent a decade+ trying-and-failing to standardize on one of a dozen tools to properly manage dependencies, at least it is trying. R is still global-namespace, no-pinning by default. Sorta-kinda you can squint where renv is going, but still needs a lot of development.
I had some hopes of Julia stealing the R mindshare and righting some of the more egregious wrongs in that ecosystem, but (as an outsider) it feels like Julia has lost a lot of steam.
[+] [-] a_t48|3 years ago|reply
I've been on the coding end of this - when everyone actually has those fixed roles and goes into it eyes wide open, it goes pretty well! When instead the PhD is assigned to go do the feature, and then the programmer is called in later when it's not implemented well, it tends to go quite poorly and nobody is happy, as everyone's time is wasted.
[+] [-] toddm|3 years ago|reply
I'm a modeler and best described as a lifelong scientific programmer. I'm much, much better at doing the specialty science I was trained to do and have done than write unit tests (for Jupyter notebooks? why?) or struggle through big-O questions (again, why? I write ~100 line programs that are never production code). Like the author of the post, I am not a professional developer and don't pretend to be or want to be. There are people out there way better than me for doing those jobs.
Recently, I interviewed for a role as a computational chemist - this is an ideal fit for a person like me with the domain knowledge, advanced degree in the subject area, passion, and a proven (if dated) track record of publishing in the domain. The idea is to use software, amend what's there if/when needed, and apply my knowledge to the process and what comes out of it.
What did the interview start with? A surprise interactive coding challenge that I wasn't prepared for, and thankfully the interviewer was kind and professional enough to understand that my value-add is in, well, computational chemistry and not in the details covered by a CS major.
I thought Jesus, I bet not a single software engineer at this company got asked to do a simple organic synthesis or even a redox problem during their interview process.
[+] [-] feoren|3 years ago|reply
Well they should have! One of the most important traits for a good software engineer is to be able to pick up a working understanding of the domain they're programming in. If they can't do that, they're more likely to make negative contributions than any positive ones.
[+] [-] hnfong|3 years ago|reply
To me at least, an interview is not to flatter the candidate by asking questions they're most comfortable with, but to make sure they have the requisite skills expected for the job. If the job expects X, Y and Z, and they seem like they're good at X and Y, you spend the time checking whether they're reasonably competent at Z as well.
[+] [-] c54|3 years ago|reply
Though even in academics, you have to write the paper yourself after doing the fun bits.
[+] [-] LAC-Tech|3 years ago|reply
If you're doing frontend, you're a servant for the user. Backend, you're a servant for the frontend.
I feel like a specialist who can communicate well and respects what other people do - is incredibly valuable, if a bit rare.
[+] [-] kgwgk|3 years ago|reply
Most people in IT industry are servants of others.
It’s not just an industry/academia divide. The issue here is independent contractor / large company.
[+] [-] djha-skin|3 years ago|reply
Eh, let the company decide that. That's one of the biggest things I tell my buddy who thinks he's bad at programming and really doesn't want to have to look for a job. He's convinced he's horrible but I don't think he is. He just doesn't want to get rejected, so he just avoids interviews as much as possible and tries to stay at the company he's at.
I would imagine that most companies are okay with python code that is rough and ready as long as it can be integrated by the rest of the team into the production code. Most data programmers are probably expected to be better at the data than the programming.
[+] [-] xLaszlo|3 years ago|reply
While I am a huge advocate of DSea writing better code going fullstack is unrealistic for them. What does it even mean to integrate? Fastapi? Docker? Helm charts? Monitoring and observability? SRE? The list is endless.
This is a classic case of "the client doesn't care the want business value".
If as a DS you want to get better at writing code join our Code Quality for Data Science (CQ4DS) discord:
https://discord.com/invite/8uUZNMCad2
[+] [-] gundamdoubleO|3 years ago|reply
We worked very well off of each other, it was interesting to pick at a model from a software engineering perspective, how the code could be structured and improved, where some tradeoffs would need to be made and how we would test and verify if it actually worked for our users. I eventually left because the company was more concerned with getting new models out as soon as possible regardless of their actual performance, but it did ignite my passion for software engineering and devops.
[+] [-] nevermore|3 years ago|reply
Plug for plotnine (https://plotnine.readthedocs.io/en/stable/). I don't know R but use ggplot indirectly through this library for exploratory data analysis, and comparing the experience to any other python plotting library, I understand why R folks are usually so sad to be using Python.
[+] [-] disgruntledphd2|3 years ago|reply
[+] [-] lp4vn|3 years ago|reply
You don't see this kind of problem in other established professions, you don't expect an accountant to be able to perform the job of a lawyer neither you expect a nurse to be able wear the hat of a nutritionist.
Now with the technological professions, let's use the term knowledge professions as an umbrella term, companies take advantage of the fact that these professions have not been around for that long and are not that established to keep expanding their rol of responsibilities.
We see that all the time with tech companies. It's not rare that you're supposed to know the frontend, backend, testing, devops, some of them even domain knowledge and the list keeps expanding even though sometimes they entail different sets of skills. The salary, not surprisingly, doesn't grow proportionally to the list of requirements. Companies don't want specialized people anymore, they want someone who will quickly pick the job of other people when/if they finish theirs.
That's what I believe the author's rant was about. He has been looking for a job in his field, he is not a software engineer. Yet people are expecting him to be a professional developer on the top of being a professional data modeler.
[+] [-] hnfong|3 years ago|reply
You'd be surprised. I mean, they're not competent to do so professionally, so they can't formally give professional advice outside of their field of competence, but you can be sure people do ask them such questions, and get disappointed if the answer is "that's not my job".
[+] [-] zmachinaz|3 years ago|reply
[+] [-] koliber|3 years ago|reply
If you are not going to be able to implement something, whether in code, or with a saw and hammer, you must be able to explain things really well. If you can not do either, you will have limited ability to apply your craft.
[+] [-] jvans|3 years ago|reply
[+] [-] bitwize|3 years ago|reply
Similarly, Judge Alsup of Oracle v. Google fame writes astronomy software in QBasic. He doesn't give a shit about best practices, if it helps him aim his telescope correctly it's all good.
Welcome to a world with citizen programmers. A world of terrible code that does the job. I frickin' love it.
[+] [-] Fiahil|3 years ago|reply
To be fair, I don’t ask scientists to understand concurrency issues in programming. Only the basic stuff that is required for delivering a functional program. Yes, Pandas and Scikit-learn belong to the basic stuff.
[+] [-] selimthegrim|3 years ago|reply
[+] [-] notacop31337|3 years ago|reply
I don't necessarily have an issue with widely skilled engineers, but I would prefer it's for the right reasons, and I largely believe that it's an exercise in laziness on most companies behalf. They just want to hire less people and have more of their workers do tasks that are outside of their remit.
I have zero interest in writing Javascript, absolutely none, I don't want to do it, and I have pushed my career in directions that mean that for the most part, I don't have too. I'm happy with this decision and have made it willingly.
It's the same with a lot of "DevOps" tasks, having previously been a DevOps Engineer, I now just want to write code, real code, but it feels as though most places now are just not hiring DevOps/Infra people, and just telling their other engineers to do it, which I understand, but it results in a far worse experience for both sides of it. I have to regularly force my hand down from volunteering for things that I have the technical experience to do, and do properly, versus colleagues that don't have the experience, because I'm tired of being shoehorned back into a role that I intentionally left. All of this is because the idea of "cross functional teams" no longer means hiring specialised engineers to do specialised roles, and just getting everyone to do everything, and then being surprised when the context switch penalty actually exists, and it's done to a worse standard than someone who is skilled in that role.
[+] [-] lysecret|3 years ago|reply
What helped me is: Learn to appreciate the beauty in actual coding, in deployments in environments in well structured, maintainable code. In scaling issues, databases etc.. There is an endless world out there which is extremely fascinating as soon as you get over the "all I want to do is modeling" mindset.
Good luck. You can definitely do it, because I did it as well.