I have always been an engineer, but when I started as an intern in 2015, most of my work centered around helping analyses: cleaning up CSVs, data mining, minor processing and transformations, etc. None of it was connected to production but these were ad-hoc requests. I think this start was invaluable to my general philosophy today (devops or die), and believe there are a lot of missed opportunities identifying the right analysts that should take the next step into complexity and help craft better devops processes for analysts. They're developers too.
One thing I don't think gets talked about enough: How much time analysts spend doing analysis vs. time spent dealing with data quality/architecture/process. So much analyst time is lost in the latter, which I think contributes greatly to burnout. But the growing intersection with engineering has and will continue to address this, albeit slowly.
I run a small data consulting company and whenever I find an open minded data scientist or analyst I tell them to consider data and software engineering or business analytics (learning about real business problems). DA, BA and SE roles are harder to find, manage more complexity, but most importantly are closer to having an impact as they are the ones close to prod. It’s a hard pitch, especially for juniors, AI and DS are very hyped but it’s hard to guess how much they will grow (or not). My current hunch is that AI and DS will be productized in libraries and services, so the majority of companies will not need people fully dedicated to build models.
My educational background is much closer to data science (Math with CompSci Minor B.Sc., Data Science M.Sc.) but I've never worked in a pure data science role.
Training and tweaking models looks like the easy part of developing data driven products. Hiring compenent enough people also seems easier than for software engineers.
Many ML libraries produce good enough results without having to design elobare models myself. I see data science more as yet another tool in the belt of a a software engineer - like server admin, CI/CD, IaC, databases,...
bobleeswagger|3 years ago
One thing I don't think gets talked about enough: How much time analysts spend doing analysis vs. time spent dealing with data quality/architecture/process. So much analyst time is lost in the latter, which I think contributes greatly to burnout. But the growing intersection with engineering has and will continue to address this, albeit slowly.
kfk|3 years ago
thomzi12|3 years ago
lytefm|3 years ago
Training and tweaking models looks like the easy part of developing data driven products. Hiring compenent enough people also seems easier than for software engineers.
Many ML libraries produce good enough results without having to design elobare models myself. I see data science more as yet another tool in the belt of a a software engineer - like server admin, CI/CD, IaC, databases,...