Working in the ML/data space and have been fortunate to have largely steered clear of this problem so far. My heuristic when evaluating potential jobs is only to consider positions where the output of statistical analysis and machine learning models has a clear and immediate impact within the organization, while avoiding adtech for ethical reasons and because of concerns toxic work could enable a toxic environment.
Jobs like this exist, though you may have to take a pay cut compared to the bullshit. My first job after finishing my PhD was as a staff scientist in an academic lab using ML engineering + data science to support scientific research. There seems to be a fair number of grant supported jobs like this and pay isn’t terrible. Just under or just at 6 figures. You can make more in industry, but scientific work feels very meaningful.
Now I’m working in credit risk modeling, something I never expected to be doing, but so far it’s been a good fit. The models are applied directly in decision making for the business, and there’s a real incentive to get everything right because mistakes could harm real people’s lives. The team I’m on is strong and ethically sound and I feel good about what I’m doing.
For anyone in the data space who’s despairing about the state of the industry, non-bullshit jobs do exist, you just have to look for them, and use your judgment when scoping out new roles.
I really like your heuristic, and I'll add to your list of examples where one could work
- Any tech company where the Stats/ML model is one of THE products and differentiators. You'll need to cut through a lot of buzz word and sales-speak to find the good ones.
- Banks and other financial institutions where making uneducated guess is a big no-no when it comes risk, pricing and anything related to financial products. Your example of Credit Risk Modeling is a classic example and very interesting problem.
- As much as consultancy gets a bad rep due to some shady practice from big players, there exists a solid demand for professionals that know Statistics in the Large Construction Projects space. Let's say modeling demand and return financial for a project, proving environmental impact, preparing/implementing/analyzing unbiased surveys in the area, and so on.
- Government agencies where data is one of the Key Outputs. Such as the Census, Bureau of Labor Statistics, CDC, and so many others.
As you said, sometimes it will mean a pay cut, especially if you want to remain in the technical work and not deal with the business and managerial side of things. But there's solid demand.
I’ve been doing “big data” in the retail space for almost a decade now and it’s pretty clear to me how it affects the bottom line. It really can be as simple as “we need more of this milk” and “the spicy queso is very popular”
> My heuristic when evaluating potential jobs is only to consider positions where the output of statistical analysis and machine learning models has a clear and immediate impact within the organization, while avoiding adtech for ethical reasons and because of concerns toxic work could enable a toxic environment.
Currently starting to look for a new role and couldn’t have articulated my goal more clearly. The problem is that this narrows the field considerably, to finance/insurance/fraud (“pure money work”), healthcare/EMR (which as far as I can tell is also actually just profit optimization work for hospitals), and then you have manufacturing (closer to where I currently am), but the actual applications of data science are more limited and data Eng / analytics are more valuable, bioinformatics (which is really interesting, but also a large learning curve), and then just traditional BI (kind of generic and boring).
Also, if you’re thinking about long term career development, many of these functions fall under the IT/CIO umbrella as orgs grow, which means that career advancement would require learning cloud architecture, cybersecurity, IAM, networking, etc. which I’m not saying is a bad thing, just an observation. Just applying statistics and modeling can only rise so high in an org, unless like you said, their core business and value prop is being the best at modeling some phenomena.
When I was starting out my career, my father gave me a piece of advice that has turned out to be incredibly applicable and valuable - "Stay close to the money". What he meant by that is work in environments where the work you perform is very closely tied to the revenue of the company - or another way of saying it - "Do work that is directly part of the organization's mission". It sounds like you've applied this to your career with good results.
Yeah, I was basically filtering out jobs applications by whether or not I could see what effect ML was supposed to have on the functioning of the business, and importantly, whether or not they actually had access to the data to make it work.
Ended up at a place where ML doesn't just make things better, but is necessary to make things work at scale.
There were a lot of job ads I looked at that just seemed dreadful to me though. So many recruiting companies wanted ML Engineers/Data scientists. I could kinda see how it would work, but didn't think I would enjoy it.
As a non-ad tech example, a significant component to what we're trying to do at https://www.auxon.io is provide tech to companies they can use for testing & analysis of robots and cyber-physical systems. From our perspective internally what we're getting is the ability to perform something akin to "materials science" for the increasingly critical software parts of these systems and construct models of their behavior to use for predictive & comparative analysis purposes.
We have a research partnership with the University of Ottawa & University of Luxembourg specifically on the statistical analysis end of things to go deep in some areas to later incorporate the findings into our products. In fact the first go around of that research cycle has already happened (https://arxiv.org/abs/2301.13807v1) and the insights are being integrated into our Deviant product (https://auxon.io/products/deviant).
It's definitely not ad-tech. It definitely has a specific applied use case. Most of our marketplace traction is in aerospace, energy, automotive, and defense. We're not immediately hiring for roles on the data analysis end of things (we're in much more immediate need of visualization & frontend help), but we will be this year.
Focusing on jobs where your output makes an immediate impact is an incredibly smart move to make sure you matter. Of course, most roles matter but there are too few managers who know how to vouch and articulate a team’s business value properly.
I actually found myself working on a credit risk modelling project on the capital markets side and it’s been great as well.
- energy management (shifting loads to times when energy is cheap) for consumer/commercial/industrial use cases
- energy markets, especially power trading: often highly algorithmic, and driven by models that turn fundamentals data (weather, calendar, …) into supply/demand predictions, and from there into price predictions
I agree they do exist, and this heuristic sounds sensible to me. It's the good old patio11 "go work in a profit center" situation, and I wish I had done so. I'd probably be grappling with a new existential concern, but that's life (I wish I had a job that didn't matter so I wouldn't have to worry about performance all day!).
I find it strange that AI/ML people would avoid "adtech for ethical reasons".
I really wish the websites I visited made better use of my actual history on those sites to tailor relevant ads to my interests. It seems like an ideal application of AI to me. They could do a far better job than serving me the lowest common denominator stuff they keep throwing at me.
My Twitter news feed these days:
* 10% - posts of interest from people I follow, i.e. the stuff I actually go to Twitter for
* 90% - Ads and recommendations of topics to follow that I have zero interest in
If it's going to insist on showing ads and recommending topics of interest, you'd think those could be better personalized, given that Twitter has years of my tweet, reply, and like history to train its AI on.
But no... what I get is crypto ads, Hollywood events, celebrity news, sports news, etc.
Long-time non-profit guy here. Totally get where the author is coming from and I'm in a very similar position in terms of where my role relates to my organisation, but I still love my job despite having exactly the same issues. Why?
Non-profits are a funny old space. In theory, they're not out there to make money, so people gravitate towards them seeking "purpose". Everything is fine once you realise that you're not going to get that, or at least in the way people come to it for.
Non-profits have ridiculous, completely unachievable goals, particularly when you put their budgets in context. Some provide truly excellent services and products (e.g., reports), most don't.
So what's so great about a non-profit tech job? I've got an interesting problem space, I love trying to work out how to measure things that are inherently difficult to measure. I have a hell of a lot of autonomy, freedom of tooling, people listen to me on tech issues -- even when they probably shouldn't.
I bust my arse because I'm interested in what I'm doing, but I could easily coast if I wanted to. The pay is enough for me, and my colleagues are for the most part super nice and interesting. If I want to learn something, I can make an excuse to play with it at work and run with it.
Seeking to have "impact" through a data job at an average non-profit is naive, but a lot of these jobs have stuff to recommend them beyond that.
----
Edit: slight addition to my list of joys of non-profit nerding
> The absolutely fucked up thing is that everyone I've met in this space seems to have totally given up on doing anything meaningful at work. The goal is to get paid, not stress out, have a happy office where everyone can collect their strange handout, and not think too deeply about how unfulfilling is it to produce nothing for forty hours a week.
this is why i think its kind of funny that people object to the idea of Universal Basic Income, when it's already being experienced in so many aimless offices. I do think this is a gross misallocation of resources, but insofar as moral judgment goes it's not obviously less moral than many other inequities in society.
UBI would just socialize the cost which at least now, the private sector partly pays.
UBI is a yearly recurring expense in the trillions.
Lots of people would work less, (remember COVID?) lowering tax revenue.
This means higher demand for workers, who will require even higher pay because of supply constraints and high taxes.
Unlocked demand, decreased labor supply and higher taxes means inflation. Lots of inflation, specially in services.
Businesses will offshore or automate to try and reduce cost, but that will push even more workers onto UBI.
What will the new equilibrium be? Nobody knows. It could stabilize with extreme income inequality creating a permanent underclass, It could drive the United States into hyperinflation, creating civil unrest and destabilizing world security. Nobody knows. It would be a society wide experiment that would be politically impossible to undo democratically.
If you think BS jobs are bad, how about no jobs in a bankrupt United States, sliding into Venezuelan hyper inflation?
Imagine China, Russia, and North Korea undeterred and the death and human suffering that would follow.
The US may not be perfect, but the world is a harsh harsh place and people have no idea how good they have it right here, right now.
So good that professionals not only demand high pay from their employers, but think it's totally reasonable to require "meaning" as well.
Were he still alive, I would pay good money to see the author try and read his essay out loud to Victor Frankel.
> people object to the idea of Universal Basic Income, when it's already being experienced in so many aimless offices
Private money being wasted due to inefficiencies at the office (as you say, a misallocation) is surely entirely different than a government-granted right to not produce anything yet still be supported.
People don't object to UBI because they don't like the idea of people getting money for no work.
They dislike UBI because either
a) It's financially impossible. Assuming the basic income is "enough to live a basic life", say £1500/month in the UK, that is totally impossible.
b) They don't like the idea of people getting free money and not having to work when they do work. I mean you see that already with benefits/welfare, but UBI makes it more extreme.
Although obviously with UBI the people paying lots of taxes have the option of stopping work and moving somewhere cheap. Which is what a huge number of people would do, and the whole system would collapse.
But yeah, nobody is against a magical utopia where nobody has to work. They just know it can't exist.
Everyone I know who works in these kind of jobs are financially secure but extremely depressed.
One of the mistaken beliefs underlying the UBI argument is that the fulfillment part will come automatically for most people. Who wouldn’t want to spend every day painting, writing, or playing music? The problem is that most people who are not ideologically self-motivated are unable to find sources of fulfillment without some sort of economic influence.
The late David Grabber had a great book about this - Bullshit Jobs.
And when you draw the line, we do keep a lot of work and a lot of jobs around purely because if we don't a lot of people are going to have a lot of free time to ask a lot of questions about the way that the world works. And the very foundation of our system is that you have to go work, sell your labour to make someone else rich, then buy things to make someone else rich and eventually die.
> this is why i think its kind of funny that people object to the idea of Universal Basic Income, when it's already being experienced in so many aimless offices. I do think this is a gross misallocation of resources, but insofar as moral judgment goes it's not obviously less moral than many other inequities in society.
Absolutely. I've worked in multiple different companies, of different sizes, since about 2011 with breaks in between. One common theme was the amount of pointless work going on, stuff that could've been streamlined or automated, or manual reports that no-one reads or acts upon.
If you’re a data professional in a company where data quality is low or non-existent, then improving the quality of data in the organization is absolutely part of your job, most likely the most important one
It’s less cool and flashy than the real time online machine learning model you want to build, or the multi-level bayesian model to determine causality between revenue and an obscure event, but that data dictionary and table/db catalog really needs to be built before everything else
Then document the essential reports to the extent that a high schooler should be able to produce/maintain them
Then adjust the reports so that the metrics indicate what management actually needs to learn from them (if you don’t know what’s important for them and for the business, then finding out is part of the job, too)
All these things are extremely important, and it’s silly to suggest your work is meaningless just because it’s not fun or interesting. There are no tv series about civil engineers building sewage networks but public sanitation saves more lives than medicine. Data work works the same
> then improving the quality of data in the organization is absolutely part of your job, most likely the most important one
You're absolutely right, but it can be hard to get management to bat for this kind of stuff. The biggest hurdle I've seen with getting "good" data is fixing the issues involves other teams prioritizing the work on their backlogs. Depending on the company and where the teams lie on the org chart, fixing things might require a multi-quarter directive from the CTO. Which isn't happening.
This is already what I and the others I've spoken to do (who haven't given up). But, at the types of organisation I'm describing, it's near impossible.
You know that Airflow instance I mentioned? Three years later, that team still doesn't have anything to schedule Python workloads, despite extremely smart people trying to angle the politics of the damn thing to finally get traction for months on end. When you've got no leverage, you've got no leverage, and you don't want to spend years of your precious life arguing so that at some point, in a few years, you can do your job properly.
I absolutely agree with what you're saying, but I suppose I'm saying that, at many jobs, you're better off leaving and somehow locating the places where you have a chance at accomplishing what you've laid out. There's a sliding scale of incompetence, and some organisations are close enough to competence that you can actually move them the right way with some diligence and effort.
That isn't possible most of the time working for a larger company. Too often the problem with bad data starts not at the collection stage but the data is bunk to begin with.
Since the data is not under your control -- it belongs to the "real" engineering team, which has vastly greater status -- you will not be able to. You will complain and they will ignore you. You will offer to help and they will refuse. You will open a PR and they will reject it.
You know it's a badly run team when they keep pulling you off your self-built, self-motivated ETL packages to breathe down your neck about incremental, barely-there releases.
The author seems to extrapolate from their experience of having a “six figure data job straight out of college clocking out at 5pm everyday producing no value” (paraphrasing) that all “data work” must be producing no societal value.
I don’t like throwing around out the phrase “privileged/out of touch” too often but this post doesn’t seem fit for the top position of HN.
I disagree, the author has touched on something widespread and hard to articulate that I suspect a large part of the HN readers have had experience with.
The problem is that lots of organisations say they want to be "data driven" but then leave all the management decisions to the prejudices of individual managers as well as management as a whole within the organisation. It's a problem that's been with us since Taylorism and the dawn of "scientific management".
You end up with "we spent years and $ to get the data which says do X, but we don't feel like doing X, so we're just going to ignore it because data in and of itself has no power within the organization".
It's like buying a gym membership and not going to the gym. Having a data science department satisfies the organisation's need to believe it's self-improving.
I found it resonated (although agree the title is click-baity, they just talk about their own experiences in multiple orgs).
It is hard to do good data work. Offhand it takes some combination of:
- business understanding and goals (that don't themselves come directly from data)
- using data to effectively orient those goals to the best opportunities
- using data to measure whether you are successful or not
Part of that can involve meta goals -- our data is not sufficient now to meet bullet 2, so we need to start doing something different to measure it.
I find many people in these roles act like they are just human machines producing reports. You really need to take agency in many situations IMO, and direct higher ups to look at data in the right way. If you just wait to be told what report to produce, it will not go well.
The CTO of my previous company was fond of saying something like:
> According to [some book he read once], the average company stands to increase their profits by [50%?] by applying data science. Even if [author] is only 10% correct, that's still tens of millions of dollars for us.
(some variation on this at every all hands meeting)
When I left the company, they were still desperately searching for that first 1% of benefit after nearly a decade of effort. Like most management fads, there is some solid foundational truth there, but it quickly becomes a solution in search of a problem when it gets tossed at every single problem faced.
Very early in my programming career I still couldn't decide whether I want to do programming or graphic design. I worked for an online game written in ActionScript (Flash) and was designing armors and some other in-game objects. I also participated in the game's forum on behalf of game's admins where they collected and discussed player's requests and suggestions for in-game new features.
My ICQ number was thus available to the wide audience as I was sometimes the focal point of player's complaints or requests. One day a player I didn't know personally contacted me on ICQ, and his avatar featured a piece of headgear I designed. I had tears standing in my eyes knowing that someone valued my work so much they actually decided to use it to represent their online identity.
Never since have I been so close to the situation that whatever I did mattered. My wife works for the government (also in IT sector), but she has better job satisfaction than I have in the sense of knowing that whatever she does has any impact at all.
In my case, I'm automating testing and deployment of a piece of software that's essentially exclusive in its market niche. So much so that customers have no idea how bad it actually is, since they have nothing to compare it to. The testing of the software I work on is outright worthless, so whether it's more automated or not -- it will have no impact on the outcome. Not to mention that the biggest problems with the software are in its design, in the decisions that went towards the core elements of it, that, at this point, nobody dears even to question, let alone to reverse them. So, it sucks. It's going to suck. And it's going to suck for a very long time in the very same way because no replacement is coming.
But, hey, I get a visa! And a permanent contract! Yee-haw!
I'm 53 and have been in the IT Security space since before there WAS an IT security space. I have installed 6 SIEMs, just put a bullet in SIEM number 5 (oddly the first time I'd ben in an organization long enough to both install and uninstall one.) and the Current generation has an on-prem and off-prem installation due to data sensitivity...we're re-doing the on-prem installation for reasons...so I'm kinda at SIEM 6.5
of the first 5...NONE of them are still in existance. I recognize the circle of life nature of my job and have gradually changed my raison de etre from 'Protect the company!' to 'Advance my career!' to 'Pay off my mortgage.'
It was a bitter pill when I realized my job was 'look at magnetic patterns on a spinning disk on a computer half a world away and determine if they were GOOD patterns (company data) or BAD Patterns (bad guys)'.
I'm going to take an educated guess that you've just turned 30 or are about to turn 30?
I had this exact same problem where I became entirely disillusioned with the output of my work after having become reasonably senior and realising that the majority of my time and effort ammounted to projects which either failed or treaded water. The unfortunate reality in business is that the projects that truly change the world are 1 in 100. They don't look or feel much different from any other at the begining but when traction really does start to take off you know you have something right. I think from your description, you need to work at a startup where you have genuine impact. This is the only way you truly "feel" meaning. You might still build something that's reasonably worthless to humanity, but you will do it your way and take your own risks.
I am surprised there are no private equity like entities that are driven by a single thesis which is to buy non-tech public corporations (or as far away from tech as possible: like commodities and real estate) then cancelling all 'digital transformation' or 'data engineering' contracts with consultancies like McKinsey, replicate the important 10% part of the contract in-house by recruiting a solid small tech team.
Then take the company public later at 10x the valuation, having slashed the cost by 10x without impacting operations or growth.
Unfortunately this post is basically the author just showing they're young and out of touch.
Here's the thing-- some data work is fundamentally worthless. Yes, some companies are a total mess when it comes to their data. Some roles are tasked with projects that are... to put it bluntly, a waste of time.
But it's your job to be improving the status quo rather than wringing your hands and declaring it all pointless. This is frankly where the difference lies between someone more junior and someone that can be an effective engineering leader-- they're looking to constantly improve.
Yeah, that means making a case to leadership sometimes. Being an effective engineer (software/data/ML) ultimately requires strong people skills.
This is somewhat the reason I switched careers from data science into data engineering. My experience was that most companies weren't mature enough to really make use of any sort of "real" data science. But there was always some data sources sitting around somewhere that could be copied around, cleaned up, then thrown into Looker. After a while, this collection of data sources will turn into a data warehouse, and a network effect will kick in, making looker (or tableau/etc) broadly useful for other people in the company.
Most of the teams I work with need simple reports to do their job. There's no need for predictions or self-learning classifications. Mostly things like, is the information in these three systems consistent? If not, can an alert be sent to notify someone?
That being said, even my current company has 10x as many DSes as DEs and I'm pretty confident that all of them are like the author of this article. I can't blame them for taking a better paying, far more prestigious job with several times as many openings in the industry.
> Piles of money + unclear outcomes = every grifter under the sun begins to migrate to your organisation. It is very hard to keep them all out, and they naturally begin to let other grifters in because they all run interference for each other. Sure, they might betray each other constantly, but they won't challenge the social fiction that some sort of meaningful work is happening.
One of the best summaries I’ve ever read about this phenomenon.
This resonates. I've moved from data engineering into sales engineering. The variety of problems is fun, albeit the context switching with Account Execs that don't respect my calendar is... something.
A lot of my previous data work was someone saying "show me what's interesting"... "no, not that- something else"... "huh, this doesn't jive with my gut, start over".
Bringing _value_ out of data sciene / engineering is incredibly hard. You have to have the engineering skills as baseline, but I firmly believe everyone undervalues the amount of work to "sell" your analysis to less technical folks in a way that's inline with the needs of the org. It's incredibly difficult.
For part of my career over 10 years ago, I worked in data science for big tech, and I can definitely see the hazards ...
It is extremely easy to get hoodwinked into joining a useless data science team, one that is mere signaling for executives, ineffective, ignored, or otherwise
OMG, pretty much a lot of what I have personally been through. But the problem is deeper and goes beyond a company not knowing what to do with someone that does ML/Data Science ... Most of software that comes out either does not contribute to society at all or is in some form harmful to it, usually due to the need for such things to be somehow profitable. And pretty much all the senior devs I know are burned out on this in one form or another and want to quit and have a farm/bakery or whatever that actually does something
Startups are a good place to look for data jobs where you can make an impact. It's pretty easy to use data the "right way" if you start early, but it's incredibly hard to change the way data is used at 30+ year old companies or ones that are already hundreds of people large.
If you're the first data hire at a company of 10 to 20 (mostly engineers) and the founders seem sharp, there's a good chance you'll be doing things that actually help the business.
There's obviously a risk that the company will go under, but if the answer to "would I use this?" is "yes", that's been enough for me to be happy at work.
> The institution produced a fair amount of academic research, but the vast majority of it was essentially fraudulent. We had some contracts with industry to produce technology in the space, but these mostly seemed to consist of the organisation fumbling the projects and massaging the truth (read: lying) until they could collect more funding.
> Piles of money + unclear outcomes = every grifter under the sun begins to migrate to your organisation. It is very hard to keep them all out, and they naturally begin to let other grifters in because they all run interference for each other.
Wow, this is very honest, which is refreshing in today's world. Thanks for the post OP.
[+] [-] steppi|3 years ago|reply
Jobs like this exist, though you may have to take a pay cut compared to the bullshit. My first job after finishing my PhD was as a staff scientist in an academic lab using ML engineering + data science to support scientific research. There seems to be a fair number of grant supported jobs like this and pay isn’t terrible. Just under or just at 6 figures. You can make more in industry, but scientific work feels very meaningful.
Now I’m working in credit risk modeling, something I never expected to be doing, but so far it’s been a good fit. The models are applied directly in decision making for the business, and there’s a real incentive to get everything right because mistakes could harm real people’s lives. The team I’m on is strong and ethically sound and I feel good about what I’m doing.
For anyone in the data space who’s despairing about the state of the industry, non-bullshit jobs do exist, you just have to look for them, and use your judgment when scoping out new roles.
[+] [-] thegginthesky|3 years ago|reply
- Any tech company where the Stats/ML model is one of THE products and differentiators. You'll need to cut through a lot of buzz word and sales-speak to find the good ones.
- Banks and other financial institutions where making uneducated guess is a big no-no when it comes risk, pricing and anything related to financial products. Your example of Credit Risk Modeling is a classic example and very interesting problem.
- As much as consultancy gets a bad rep due to some shady practice from big players, there exists a solid demand for professionals that know Statistics in the Large Construction Projects space. Let's say modeling demand and return financial for a project, proving environmental impact, preparing/implementing/analyzing unbiased surveys in the area, and so on.
- Government agencies where data is one of the Key Outputs. Such as the Census, Bureau of Labor Statistics, CDC, and so many others.
As you said, sometimes it will mean a pay cut, especially if you want to remain in the technical work and not deal with the business and managerial side of things. But there's solid demand.
[+] [-] pram|3 years ago|reply
[+] [-] plaidfuji|3 years ago|reply
Currently starting to look for a new role and couldn’t have articulated my goal more clearly. The problem is that this narrows the field considerably, to finance/insurance/fraud (“pure money work”), healthcare/EMR (which as far as I can tell is also actually just profit optimization work for hospitals), and then you have manufacturing (closer to where I currently am), but the actual applications of data science are more limited and data Eng / analytics are more valuable, bioinformatics (which is really interesting, but also a large learning curve), and then just traditional BI (kind of generic and boring).
Also, if you’re thinking about long term career development, many of these functions fall under the IT/CIO umbrella as orgs grow, which means that career advancement would require learning cloud architecture, cybersecurity, IAM, networking, etc. which I’m not saying is a bad thing, just an observation. Just applying statistics and modeling can only rise so high in an org, unless like you said, their core business and value prop is being the best at modeling some phenomena.
[+] [-] santoshalper|3 years ago|reply
[+] [-] fnands|3 years ago|reply
Ended up at a place where ML doesn't just make things better, but is necessary to make things work at scale.
There were a lot of job ads I looked at that just seemed dreadful to me though. So many recruiting companies wanted ML Engineers/Data scientists. I could kinda see how it would work, but didn't think I would enjoy it.
[+] [-] im_down_w_otp|3 years ago|reply
We have a research partnership with the University of Ottawa & University of Luxembourg specifically on the statistical analysis end of things to go deep in some areas to later incorporate the findings into our products. In fact the first go around of that research cycle has already happened (https://arxiv.org/abs/2301.13807v1) and the insights are being integrated into our Deviant product (https://auxon.io/products/deviant).
It's definitely not ad-tech. It definitely has a specific applied use case. Most of our marketplace traction is in aerospace, energy, automotive, and defense. We're not immediately hiring for roles on the data analysis end of things (we're in much more immediate need of visualization & frontend help), but we will be this year.
[+] [-] ensemblehq|3 years ago|reply
I actually found myself working on a credit risk modelling project on the capital markets side and it’s been great as well.
[+] [-] pfalke|3 years ago|reply
- energy management (shifting loads to times when energy is cheap) for consumer/commercial/industrial use cases
- energy markets, especially power trading: often highly algorithmic, and driven by models that turn fundamentals data (weather, calendar, …) into supply/demand predictions, and from there into price predictions
- retail pricing, both offline and e-commerce
[+] [-] nuclearnice1|3 years ago|reply
Is it trying to take a huge dataset of consumer features and join it to a dataset of loan outcomes and then predict loan outcomes?
[+] [-] ludicity|3 years ago|reply
[+] [-] magicalhippo|3 years ago|reply
This reminded me of those Microsoft Viva emails I keep getting. Anyone finding those useful for anything at all?
To me it seems like a solution in search of a problem.
[+] [-] arbuge|3 years ago|reply
I really wish the websites I visited made better use of my actual history on those sites to tailor relevant ads to my interests. It seems like an ideal application of AI to me. They could do a far better job than serving me the lowest common denominator stuff they keep throwing at me.
My Twitter news feed these days:
* 10% - posts of interest from people I follow, i.e. the stuff I actually go to Twitter for
* 90% - Ads and recommendations of topics to follow that I have zero interest in
If it's going to insist on showing ads and recommending topics of interest, you'd think those could be better personalized, given that Twitter has years of my tweet, reply, and like history to train its AI on.
But no... what I get is crypto ads, Hollywood events, celebrity news, sports news, etc.
[+] [-] sizzle|3 years ago|reply
[+] [-] specproc|3 years ago|reply
Non-profits are a funny old space. In theory, they're not out there to make money, so people gravitate towards them seeking "purpose". Everything is fine once you realise that you're not going to get that, or at least in the way people come to it for.
Non-profits have ridiculous, completely unachievable goals, particularly when you put their budgets in context. Some provide truly excellent services and products (e.g., reports), most don't.
So what's so great about a non-profit tech job? I've got an interesting problem space, I love trying to work out how to measure things that are inherently difficult to measure. I have a hell of a lot of autonomy, freedom of tooling, people listen to me on tech issues -- even when they probably shouldn't.
I bust my arse because I'm interested in what I'm doing, but I could easily coast if I wanted to. The pay is enough for me, and my colleagues are for the most part super nice and interesting. If I want to learn something, I can make an excuse to play with it at work and run with it.
Seeking to have "impact" through a data job at an average non-profit is naive, but a lot of these jobs have stuff to recommend them beyond that.
----
Edit: slight addition to my list of joys of non-profit nerding
[+] [-] swyx|3 years ago|reply
this is why i think its kind of funny that people object to the idea of Universal Basic Income, when it's already being experienced in so many aimless offices. I do think this is a gross misallocation of resources, but insofar as moral judgment goes it's not obviously less moral than many other inequities in society.
[+] [-] francob411|3 years ago|reply
UBI is a yearly recurring expense in the trillions.
Lots of people would work less, (remember COVID?) lowering tax revenue.
This means higher demand for workers, who will require even higher pay because of supply constraints and high taxes.
Unlocked demand, decreased labor supply and higher taxes means inflation. Lots of inflation, specially in services.
Businesses will offshore or automate to try and reduce cost, but that will push even more workers onto UBI.
What will the new equilibrium be? Nobody knows. It could stabilize with extreme income inequality creating a permanent underclass, It could drive the United States into hyperinflation, creating civil unrest and destabilizing world security. Nobody knows. It would be a society wide experiment that would be politically impossible to undo democratically.
If you think BS jobs are bad, how about no jobs in a bankrupt United States, sliding into Venezuelan hyper inflation?
Imagine China, Russia, and North Korea undeterred and the death and human suffering that would follow.
The US may not be perfect, but the world is a harsh harsh place and people have no idea how good they have it right here, right now.
So good that professionals not only demand high pay from their employers, but think it's totally reasonable to require "meaning" as well.
Were he still alive, I would pay good money to see the author try and read his essay out loud to Victor Frankel.
[+] [-] prottog|3 years ago|reply
Private money being wasted due to inefficiencies at the office (as you say, a misallocation) is surely entirely different than a government-granted right to not produce anything yet still be supported.
[+] [-] IshKebab|3 years ago|reply
They dislike UBI because either
a) It's financially impossible. Assuming the basic income is "enough to live a basic life", say £1500/month in the UK, that is totally impossible.
b) They don't like the idea of people getting free money and not having to work when they do work. I mean you see that already with benefits/welfare, but UBI makes it more extreme.
Although obviously with UBI the people paying lots of taxes have the option of stopping work and moving somewhere cheap. Which is what a huge number of people would do, and the whole system would collapse.
But yeah, nobody is against a magical utopia where nobody has to work. They just know it can't exist.
[+] [-] Zanneth|3 years ago|reply
One of the mistaken beliefs underlying the UBI argument is that the fulfillment part will come automatically for most people. Who wouldn’t want to spend every day painting, writing, or playing music? The problem is that most people who are not ideologically self-motivated are unable to find sources of fulfillment without some sort of economic influence.
[+] [-] penguinvondoom|3 years ago|reply
[+] [-] patrickk|3 years ago|reply
Absolutely. I've worked in multiple different companies, of different sizes, since about 2011 with breaks in between. One common theme was the amount of pointless work going on, stuff that could've been streamlined or automated, or manual reports that no-one reads or acts upon.
[+] [-] jayd16|3 years ago|reply
[+] [-] unknown|3 years ago|reply
[deleted]
[+] [-] vslira|3 years ago|reply
It’s less cool and flashy than the real time online machine learning model you want to build, or the multi-level bayesian model to determine causality between revenue and an obscure event, but that data dictionary and table/db catalog really needs to be built before everything else
Then document the essential reports to the extent that a high schooler should be able to produce/maintain them
Then adjust the reports so that the metrics indicate what management actually needs to learn from them (if you don’t know what’s important for them and for the business, then finding out is part of the job, too)
All these things are extremely important, and it’s silly to suggest your work is meaningless just because it’s not fun or interesting. There are no tv series about civil engineers building sewage networks but public sanitation saves more lives than medicine. Data work works the same
[+] [-] mywittyname|3 years ago|reply
You're absolutely right, but it can be hard to get management to bat for this kind of stuff. The biggest hurdle I've seen with getting "good" data is fixing the issues involves other teams prioritizing the work on their backlogs. Depending on the company and where the teams lie on the org chart, fixing things might require a multi-quarter directive from the CTO. Which isn't happening.
[+] [-] ludicity|3 years ago|reply
You know that Airflow instance I mentioned? Three years later, that team still doesn't have anything to schedule Python workloads, despite extremely smart people trying to angle the politics of the damn thing to finally get traction for months on end. When you've got no leverage, you've got no leverage, and you don't want to spend years of your precious life arguing so that at some point, in a few years, you can do your job properly.
I absolutely agree with what you're saying, but I suppose I'm saying that, at many jobs, you're better off leaving and somehow locating the places where you have a chance at accomplishing what you've laid out. There's a sliding scale of incompetence, and some organisations are close enough to competence that you can actually move them the right way with some diligence and effort.
[+] [-] trompetenaccoun|3 years ago|reply
[+] [-] slotrans|3 years ago|reply
[+] [-] uoaei|3 years ago|reply
Thanks for the space to vent a sec...
[+] [-] berkle4455|3 years ago|reply
[+] [-] rrjjww|3 years ago|reply
I don’t like throwing around out the phrase “privileged/out of touch” too often but this post doesn’t seem fit for the top position of HN.
[+] [-] throw_pm23|3 years ago|reply
[+] [-] pjc50|3 years ago|reply
You end up with "we spent years and $ to get the data which says do X, but we don't feel like doing X, so we're just going to ignore it because data in and of itself has no power within the organization".
It's like buying a gym membership and not going to the gym. Having a data science department satisfies the organisation's need to believe it's self-improving.
[+] [-] apwheele|3 years ago|reply
It is hard to do good data work. Offhand it takes some combination of:
- business understanding and goals (that don't themselves come directly from data)
- using data to effectively orient those goals to the best opportunities
- using data to measure whether you are successful or not
Part of that can involve meta goals -- our data is not sufficient now to meet bullet 2, so we need to start doing something different to measure it.
I find many people in these roles act like they are just human machines producing reports. You really need to take agency in many situations IMO, and direct higher ups to look at data in the right way. If you just wait to be told what report to produce, it will not go well.
[+] [-] angry_moose|3 years ago|reply
> According to [some book he read once], the average company stands to increase their profits by [50%?] by applying data science. Even if [author] is only 10% correct, that's still tens of millions of dollars for us.
(some variation on this at every all hands meeting)
When I left the company, they were still desperately searching for that first 1% of benefit after nearly a decade of effort. Like most management fads, there is some solid foundational truth there, but it quickly becomes a solution in search of a problem when it gets tossed at every single problem faced.
[+] [-] crabbone|3 years ago|reply
My ICQ number was thus available to the wide audience as I was sometimes the focal point of player's complaints or requests. One day a player I didn't know personally contacted me on ICQ, and his avatar featured a piece of headgear I designed. I had tears standing in my eyes knowing that someone valued my work so much they actually decided to use it to represent their online identity.
Never since have I been so close to the situation that whatever I did mattered. My wife works for the government (also in IT sector), but she has better job satisfaction than I have in the sense of knowing that whatever she does has any impact at all.
In my case, I'm automating testing and deployment of a piece of software that's essentially exclusive in its market niche. So much so that customers have no idea how bad it actually is, since they have nothing to compare it to. The testing of the software I work on is outright worthless, so whether it's more automated or not -- it will have no impact on the outcome. Not to mention that the biggest problems with the software are in its design, in the decisions that went towards the core elements of it, that, at this point, nobody dears even to question, let alone to reverse them. So, it sucks. It's going to suck. And it's going to suck for a very long time in the very same way because no replacement is coming.
But, hey, I get a visa! And a permanent contract! Yee-haw!
[+] [-] Damogran6|3 years ago|reply
of the first 5...NONE of them are still in existance. I recognize the circle of life nature of my job and have gradually changed my raison de etre from 'Protect the company!' to 'Advance my career!' to 'Pay off my mortgage.'
It was a bitter pill when I realized my job was 'look at magnetic patterns on a spinning disk on a computer half a world away and determine if they were GOOD patterns (company data) or BAD Patterns (bad guys)'.
[+] [-] kinnth|3 years ago|reply
I had this exact same problem where I became entirely disillusioned with the output of my work after having become reasonably senior and realising that the majority of my time and effort ammounted to projects which either failed or treaded water. The unfortunate reality in business is that the projects that truly change the world are 1 in 100. They don't look or feel much different from any other at the begining but when traction really does start to take off you know you have something right. I think from your description, you need to work at a startup where you have genuine impact. This is the only way you truly "feel" meaning. You might still build something that's reasonably worthless to humanity, but you will do it your way and take your own risks.
[+] [-] ak_111|3 years ago|reply
Then take the company public later at 10x the valuation, having slashed the cost by 10x without impacting operations or growth.
[+] [-] alfalfasprout|3 years ago|reply
Here's the thing-- some data work is fundamentally worthless. Yes, some companies are a total mess when it comes to their data. Some roles are tasked with projects that are... to put it bluntly, a waste of time.
But it's your job to be improving the status quo rather than wringing your hands and declaring it all pointless. This is frankly where the difference lies between someone more junior and someone that can be an effective engineering leader-- they're looking to constantly improve.
Yeah, that means making a case to leadership sometimes. Being an effective engineer (software/data/ML) ultimately requires strong people skills.
[+] [-] mywittyname|3 years ago|reply
Most of the teams I work with need simple reports to do their job. There's no need for predictions or self-learning classifications. Mostly things like, is the information in these three systems consistent? If not, can an alert be sent to notify someone?
That being said, even my current company has 10x as many DSes as DEs and I'm pretty confident that all of them are like the author of this article. I can't blame them for taking a better paying, far more prestigious job with several times as many openings in the industry.
[+] [-] d23|3 years ago|reply
One of the best summaries I’ve ever read about this phenomenon.
[+] [-] Wonnk13|3 years ago|reply
A lot of my previous data work was someone saying "show me what's interesting"... "no, not that- something else"... "huh, this doesn't jive with my gut, start over".
Bringing _value_ out of data sciene / engineering is incredibly hard. You have to have the engineering skills as baseline, but I firmly believe everyone undervalues the amount of work to "sell" your analysis to less technical folks in a way that's inline with the needs of the org. It's incredibly difficult.
[+] [-] chubot|3 years ago|reply
https://news.ycombinator.com/item?id=33787270
The main reason I soured on data science is that the work felt like it didn’t matter, in multiple senses of the words “didn’t matter”:
https://ryxcommar.com/2022/11/27/goodbye-data-science/
For part of my career over 10 years ago, I worked in data science for big tech, and I can definitely see the hazards ...
It is extremely easy to get hoodwinked into joining a useless data science team, one that is mere signaling for executives, ineffective, ignored, or otherwise
[+] [-] penguinvondoom|3 years ago|reply
[+] [-] taylorius|3 years ago|reply
[+] [-] mritchie712|3 years ago|reply
If you're the first data hire at a company of 10 to 20 (mostly engineers) and the founders seem sharp, there's a good chance you'll be doing things that actually help the business.
There's obviously a risk that the company will go under, but if the answer to "would I use this?" is "yes", that's been enough for me to be happy at work.
[+] [-] vitehozonage|3 years ago|reply
[+] [-] icapybara|3 years ago|reply
> Piles of money + unclear outcomes = every grifter under the sun begins to migrate to your organisation. It is very hard to keep them all out, and they naturally begin to let other grifters in because they all run interference for each other.
Wow, this is very honest, which is refreshing in today's world. Thanks for the post OP.