The most amusing variant of the XY problem is when you want a seemingly ridiculous Y, and then, when people inevitably ask why, you tell them about your entirely-novel X, which they still don’t understand the use for, so you have to explain the entirely-novel W that motivates X, and so on... and by the time you’re done, you’ve explained your whole business model, written a ten-page journal paper, and completely killed the conversation—because they don’t feel they have the prerequisite knowledge to talk about A-through-X, so you just get a “good luck with that, buddy.”
Even though Y was a concrete problem that they do have domain experience in solving!
I call this “going up the rabbit hole.”
——
Example (deliberately shortened from its real depth):
• I’m trying to invent a novel algorithm for inter-procedural data-flow analysis, because none of the existing ones work for me.
• Why? Because I’m trying to reverse a stack machine bytecode with only indirect jumps and no reified CALL/RET opcodes, into a structured inter-procedural control-flow-graph.
• Why? Because I want to do program-slicing to recover HLL representations of key-value-store writes that the program does.
• Why? Because I want to use them as hints for my runtime execution tracer for said bytecode, so it can emit better traces.
• Why? Because I want to put those traces into an OLAP database and I want the data in the DB to be typed.
• Why? Because these traces are the only canonical representation of the state of the system, and our business model is allowing auditors to figure out what’s going on with the system.
This is why on the rare occasion that I find myself answering questions on IRC or Stack Overflow I always start with at least a simple answer to the question asked. Only once I've taken a stab at answering the question do I consider moving on to "but, what are you really doing..."
Another related type of question is a question born out of curiosity rather than need. Sometimes you want to do things just for fun to see how they work, but you get shut down every time you ask a question because "you should never do that in production".
The asker knows that providing the full A-X context would be an utter waste of time, and really does want an answer to Y only.
Certain IRC regs will flat out refuse to answer Y. Often this is because they don't know the answer to Y, but instead of saying so would prefer to wear the mantle of wise interrogator of "things you did not think of"™.
Ofc, sometimes they're talking to a newb and this is appropriate, but jumping to "this is an XY problem" seems to be a knee-jerk reaction for some people.
I hate that, I ask for help with an odd Y and say "I've researched all other solutions and they didn't work, so can someone please help with Y?" but people still make me show them my work and convince them that everything else actually won't work before they maybe help.
I get this sometimes when I'm trying to learn about a suboptimal solution. I can even preface the question with, "I know this isn't the best solution but can you explain why it works..." Inevitably I get, "you shouldn't do it that way..."
Sure, there are cases where seemingly-ridiculous things are actually needed.
Where a lot of people commenting on this article are failing, though is thinking only about their needs and what they want. The people responding to your IRC message/ML post/etc are (usually) giving up their valuable time to help others for free, and making some accommodation for this is essential if you want to get anywhere useful.
On top of this, when you join a room/list where you aren't well known, the responders have no context on who you are, your level of experience, etc[0] - but what they DO have is long, long experience with people asking XY questions where Y is both ridiculous and not going to solve X anyway.
I know it's frustrating to have to go through your reasons when you are certain that damn_silly_thing_Y is actually the only way of solving your problem, but it is needed. After all, you're asking these people because they're likely more experienced than you are (or else why ask them?) and thus there's a chance you could be wrong.
0: And you can't often shortcut this - due to astronomically high rates of Dunning-Kruger cognitive bias in many of these venues, starting out with "I'm an expert, I know what I'm doing" without very convincing evidence is usually extremely counter-productive.
I've just started immediately telling them my high level project if they ask why I am doing something. Even though I really doubt it will aid understanding and fixing my problem, since my high level projects are highly obscure research projects, it at least saves time. And then they'll either go away, or give a solution to my original problem; if they start questioning my research project I can just ignore them, though that has not actually happened yet.
It's pretty lame that the examples on the page show the answerer not just answering but also talking down to the questioner. XY is a problem that happens a lot, but disparaging people who are trying to learn is not the right response.
Here's how that conversation should have gone:
---
Q: How can I echo the last three characters in a filename?
A: If they're in a variable: echo ${foo: -3}
A: However, if what you really want is to get the file extension, it's probably better to handle extensions of any length. Try this instead: ${foo##*.}
A: That expression says to find the longest prefix of `foo` that matches the given glob pattern, and remove it.
Q: Oh wow! That is actually what I wanted, but it hadn't occurred to me that such powerful pattern-matching would be built into the shell. I already had a solution involving piping through sed, but it was pretty slow and ugly. I noticed that all the files I'm operating on have three-letter extensions and I figured there'd be an easier way to take a slice of a string, since that's a common thing to have built-in. But your answer is the best of both worlds. TIL!
---
There's really no need to admonish the questioner for starting with the wrong question. That won't help. It will only make them afraid to ask questions at all.
I agree with the rest of the commenters here that the XY problem problem is a way bigger one. Example:
"How do you implement a new system call in the Linux kernel?"
"Ah, well, a system call is overkill and leads to compatibility issues, you should try to see whether you cannot solve your problem some other way..."
"My problem is I want to understand how system calls are implemented thus I want to go through all the steps of making a new one so that I don't miss something"
Or, a more recent one from Reddit:
"When will we have desktop computers capable of replicating the feat of Google with AlphaGo?"
"Ah, well, we already have distributed computing trained NNs, also there is some research that it may not require such power..."
"I'm not interested in Go, I'm interested in the power of future computers!"
Seriously, people second-guessing you are annoying.
> "I'm not interested in Go, I'm interested in the power of future computers!"
Reminded me of this story. I told a co-worker, while we worked on our DB:
- I wish we had the money to run Oracle.
- Are you crazy? Postgres is awesome and perfect for us!
- I know. I'm not saying "I wish we were running Oracle". I just wanted to have the money to do that. :)
How is the second answer second-guessing? Replicating the feat is a question of both software and hardware. To give an answer that's just about hardware would require second-guessing the intent of the question.
try asking "how can I automate ssh authentication by password?" sometime. I've still never found an answer other than "never do that, here's how to use ssh-agent."
> "How do you implement a new system call in the Linux Kernel?"
> My problem is I want to understand how system calls are implemented thus I want to go through all the steps of making a new one so that I don't miss something"
These are very different statements, and require different answers. It's like asking "How do I sort a List" vs. "I would like to understand sorting algorithms, what are some resources to start with?".
People second guessing led to what you actually should have asked in the first place.
That feeling when you spend hours reducing a problem to its simplest form and then the experts pile in to your IRC / mailing list / ticket system / stack overflow post to explain that you must be suffering from the XY problem because your ultimate goal couldn't possibly be to do what the minimized program does.
Naturally, they didn't read the part at the top where you explain your ultimate goal.
Two days later you make a new post favoring contextual detail over brevity and the same experts pile in to complain that you didn't minimize your example.
I just had to left a job after 2 weeks, because as someone new to the clan, everything I said was noise to them. I guess there's a time-to-respect period to go through.
Unfortunately, the assumption that "someone who's asking for help with something doesn't really know what they want or need " breeds horrible culture.
This is evident by the examples in the article and the ones in the cited sources. If someone is asking for help, don't try to shame them into explaining what "they really want". What they want is help with a morsel of a problem they ask, nothing more. If it's not the right path to a solution, they will decide, not you.
Most comments here are about how annoying people asking about X are.
We get it, sometimes you are asking about X simply because you need to know about X and you know what you are doing.
But often, especially when dealing with more junior engineers, I've found this simple to keep in mind and helpful. It's also a good proof of experience to know when to ask about X and try to make sure you understand the context of the question before answering it.
OTOH, wanting to do y for other reasons, finding an xy SO post on Google, and not being able to get the answer because it went off topic about rather or not y is the solution to X is frustrating.
Bonus points, ask y again because the other question answered z only and get it closed as a dupe of the other question.
I'm one of the people who tend to ask why. Sorry to everyone on this thread who are frustrated by it.
From my point of view, I'm really not trying to undermine what you're doing, it's a genuine attempt at getting you the best solution.
Here are some examples:
>How do I use apt on Fedora?
>How do I unroll this bash loop?
>How do I multiply this number by 1024 with sed?
I could have a lot of fun answering each question exactly as stated and if you say "I know it's the wrong tool for the job but I'm doing it for fun" that's exactly what I'll do.
However, it turned out that:
>They just wanted to install stuff thinking apt was universal. The better solution was using yum
>Their loop was slow and they had read that unrolling was an optimization. The better solution was reducing the number of subshells spawned
>They just wanted a result, not to implement an integer multiplication state machine in sed. The better solution was using perl/awk
I know some people are jerks about it, and that sometimes you want answers to weird questions. When people do appear to be going down the wrong path though, it would be a disservice to help them do that without letting them know that there's probably a better way.
One thing I hate is when the XY problem gets encoded into a knowledge base:
1. It's common for people trying to do Y to ask about X, which is a bad way to accomplish Y.
2. I'm trying to accomplish Z, for which X is a perfectly good option.
3. Every answer on stack overflow for "How to X" is people assuming the XY problem and answering "here's how you do Y instead".
The question, "what are you trying to do?" is super useful to remember to ask. I work at a company that's trying to modernize and move to devops for dev environments. Because the infrastructure is not nailed down, this means that every single team needs to have at least one person who understands the devops.
The process of learning k8s and grokking the security concerns in the course of a few sprints is not pleasant. This "XY Problem" crops up all the time, but the devops team is absolutely slammed all the time so many times basic queries go unanswered. Everybody has to help everybody else.
I don't think anyone wants to go back to the days of rigid development infrastructure so we're kind of stuck with devops until the ecosystem can mature.
My great hope is for serverless to take off and so programmers can further specialize. I'm getting tired of learning new languages and stacks every few months or years. Just hand me a biz requirement and let me implement it any way I like.
Conversely, there is the XY fallacy: when one [frequently condescendingly] assumes they know better than the person asking, errantly classifies it as an XY problem, and fails to answer the original question.
Depends. This is a matter of prudential judgement. The proper attitude is contextually determined and the linked page lists some qualifications, though still takes too much of a narrow, absolutist attitude.
For example, if someone random poses a problem on IRC, the solution to the "real" problem to be solved is NOT the concern or responsibility of the people in the channel. Unless there's a reason to ask for more context, just answer the damn question. Blindly asking the user what they're REALLY trying to solve is taking an intrusive and patronizing attitude. However, if asking will provide helpful context, then that's a valid reason to ask. If the problem as posed seems onerous or contrived, you COULD courteously ask the other person what they're ultimately trying to solve in the spirit of offering to be helpful if the other person so desires, but no one is hiring you do solve the "real" problem for them, so mind your own business.
If a coworker approaches you with a question, similar principles apply, but if you're working on the same project and in the same context, it may be natural to ask about what the larger problem is in a spirit of collaboration, esp. when the questions posed by your coworker smell funny. After all, you both share concerns about a project, though again, this does not give you license to automatically wrest the problem from your co-worker. His ticket, his problem. Your role is advisory.
So, common sense. Don't be one of those doltish know-it-all jerks who like to run the show. Mind the proper boundaries of concern.
Is it just me or is the XY Problem really poorly named? It's unlikely that anyone could hear "XY Problem" for the first time and have any idea what it means.
Here's the thing, if you ask an unusual question, one of two scenarios is often playing out:
In case 1, you are a beginner. You have an XY question. But you've never heard of an XY question. You also probably couldn't identify your question as an XY question if you had heard of it. Trying to educate people about XY problem is pointless, because beginners don't see it or understand it or have the ability to apply it.
In case 2, you are advanced enough to avoid asking XY questions. You are instead stuck on some esoteric question which you can't resolve by exhausting all your skills. So you post a question on the internet. But you are only asking a question because it is a really hard question. Chances are that most other people cannot answer the question either. So, the only responses you get are people suggesting that you have an XY problem.
If a programmer whom I know and respect in general asks me how she can do X, I tend to start with an answer about X.
When a name I've never seen before on a -users mailing list asks me how to do X -- and X is bizarre -- I assume we have the XY problem.
We have the XY problem because the Internet is full of newbies at any given point, and we've wasted enough attention-span having sixteen-part conversations about how to extract the IP address from ifconfig output* that we'd really like to know if we can skip all that and answer with "Use a dynamic DNS client, don't write your own."
*Don't. Use ip a
...
Use awk.
...
There are an infinite number of guides to awk, but mostly you need to know how it splits strings into fields and prints the ones you specify. How are you planning on handling machines with multiple IPs, by the way?
I just want to point out that the example is really poor behavior on the part of the teacher: Shouting in all-caps to ASK FOR WHAT YOU REALLY WANT is more likely to result in the newbie not asking at all next time.
If this is your work environment you need to make it okay to ask questions, and to dialogue about a problem without shaming the learner. (Really that should be civil behavior in any environment.)
While it's an important phenomenon to be aware of, its name fails to carry any semantic meaning. Should be named something like "falsely known solution" or "context stripping" or something
[+] [-] derefr|6 years ago|reply
Even though Y was a concrete problem that they do have domain experience in solving!
I call this “going up the rabbit hole.”
——
Example (deliberately shortened from its real depth):
• I’m trying to invent a novel algorithm for inter-procedural data-flow analysis, because none of the existing ones work for me.
• Why? Because I’m trying to reverse a stack machine bytecode with only indirect jumps and no reified CALL/RET opcodes, into a structured inter-procedural control-flow-graph.
• Why? Because I want to do program-slicing to recover HLL representations of key-value-store writes that the program does.
• Why? Because I want to use them as hints for my runtime execution tracer for said bytecode, so it can emit better traces.
• Why? Because I want to put those traces into an OLAP database and I want the data in the DB to be typed.
• Why? Because these traces are the only canonical representation of the state of the system, and our business model is allowing auditors to figure out what’s going on with the system.
[+] [-] 0xffff2|6 years ago|reply
Another related type of question is a question born out of curiosity rather than need. Sometimes you want to do things just for fun to see how they work, but you get shut down every time you ask a question because "you should never do that in production".
[+] [-] jonahx|6 years ago|reply
The asker knows that providing the full A-X context would be an utter waste of time, and really does want an answer to Y only.
Certain IRC regs will flat out refuse to answer Y. Often this is because they don't know the answer to Y, but instead of saying so would prefer to wear the mantle of wise interrogator of "things you did not think of"™.
Ofc, sometimes they're talking to a newb and this is appropriate, but jumping to "this is an XY problem" seems to be a knee-jerk reaction for some people.
[+] [-] StavrosK|6 years ago|reply
[+] [-] Waterluvian|6 years ago|reply
[+] [-] dahart|6 years ago|reply
Love it, that’s great! Adding it to my lexicon.
[+] [-] jfindley|6 years ago|reply
Where a lot of people commenting on this article are failing, though is thinking only about their needs and what they want. The people responding to your IRC message/ML post/etc are (usually) giving up their valuable time to help others for free, and making some accommodation for this is essential if you want to get anywhere useful.
On top of this, when you join a room/list where you aren't well known, the responders have no context on who you are, your level of experience, etc[0] - but what they DO have is long, long experience with people asking XY questions where Y is both ridiculous and not going to solve X anyway.
I know it's frustrating to have to go through your reasons when you are certain that damn_silly_thing_Y is actually the only way of solving your problem, but it is needed. After all, you're asking these people because they're likely more experienced than you are (or else why ask them?) and thus there's a chance you could be wrong.
0: And you can't often shortcut this - due to astronomically high rates of Dunning-Kruger cognitive bias in many of these venues, starting out with "I'm an expert, I know what I'm doing" without very convincing evidence is usually extremely counter-productive.
[+] [-] catern|6 years ago|reply
[+] [-] floren|6 years ago|reply
I just call it "posting on Stack Overflow"
[+] [-] unknown|6 years ago|reply
[deleted]
[+] [-] iovrthoughtthis|6 years ago|reply
Hi, i need $10.
Why? Because I need to buy a beer.
Why? Because the man at the bike shop is super thirsty.
Why is that important? Because If I sate his thirst he said he’d introduce me to his manager.
Why is that important? Because I want to work at the bike store and it’s the only way to get an intro.
“The traces are the only canonical representation of the state of the system”
Why?
[+] [-] kentonv|6 years ago|reply
Here's how that conversation should have gone:
---
Q: How can I echo the last three characters in a filename?
A: If they're in a variable: echo ${foo: -3}
A: However, if what you really want is to get the file extension, it's probably better to handle extensions of any length. Try this instead: ${foo##*.}
A: That expression says to find the longest prefix of `foo` that matches the given glob pattern, and remove it.
Q: Oh wow! That is actually what I wanted, but it hadn't occurred to me that such powerful pattern-matching would be built into the shell. I already had a solution involving piping through sed, but it was pretty slow and ugly. I noticed that all the files I'm operating on have three-letter extensions and I figured there'd be an easier way to take a slice of a string, since that's a common thing to have built-in. But your answer is the best of both worlds. TIL!
---
There's really no need to admonish the questioner for starting with the wrong question. That won't help. It will only make them afraid to ask questions at all.
[+] [-] Rerarom|6 years ago|reply
"How do you implement a new system call in the Linux kernel?"
"Ah, well, a system call is overkill and leads to compatibility issues, you should try to see whether you cannot solve your problem some other way..."
"My problem is I want to understand how system calls are implemented thus I want to go through all the steps of making a new one so that I don't miss something"
Or, a more recent one from Reddit:
"When will we have desktop computers capable of replicating the feat of Google with AlphaGo?"
"Ah, well, we already have distributed computing trained NNs, also there is some research that it may not require such power..."
"I'm not interested in Go, I'm interested in the power of future computers!"
Seriously, people second-guessing you are annoying.
[+] [-] _uvvk|6 years ago|reply
Reminded me of this story. I told a co-worker, while we worked on our DB:
[+] [-] daveFNbuck|6 years ago|reply
[+] [-] leetcrew|6 years ago|reply
[+] [-] rbavocadotree|6 years ago|reply
> My problem is I want to understand how system calls are implemented thus I want to go through all the steps of making a new one so that I don't miss something"
These are very different statements, and require different answers. It's like asking "How do I sort a List" vs. "I would like to understand sorting algorithms, what are some resources to start with?".
People second guessing led to what you actually should have asked in the first place.
[+] [-] threatofrain|6 years ago|reply
[+] [-] koala_man|6 years ago|reply
[+] [-] mlevental|6 years ago|reply
[+] [-] emilfihlman|6 years ago|reply
And? You are the extremely small minority. You need to learn to live with it.
[+] [-] Can_Not|6 years ago|reply
"Does Django's query builder use prepared statements by default?"
"What are you really trying to do?"
"I'm trying to find out if Django's query builder uses prepared statements by default..."
Not everything is an XY problem. If somebody found that question on SO via Google, they would be pretty annoyed to see it XY'd instead of answered.
[+] [-] jacknews|6 years ago|reply
Eg,
Me: "What's the best trackball in 2019?"
Crowd: "Do you have RSI? You probably need a vertical mouse."
etc
[+] [-] jjoonathan|6 years ago|reply
Naturally, they didn't read the part at the top where you explain your ultimate goal.
Two days later you make a new post favoring contextual detail over brevity and the same experts pile in to complain that you didn't minimize your example.
[+] [-] zamadatix|6 years ago|reply
[+] [-] pezo1919|6 years ago|reply
Why do I want anything? Because I want to be happy and successful, lol.
Is it broad enough now?
[+] [-] agumonkey|6 years ago|reply
[+] [-] unknown|6 years ago|reply
[deleted]
[+] [-] kps|6 years ago|reply
Kensington Slimblade, still.
[+] [-] Doubl|6 years ago|reply
[+] [-] lxe|6 years ago|reply
This is evident by the examples in the article and the ones in the cited sources. If someone is asking for help, don't try to shame them into explaining what "they really want". What they want is help with a morsel of a problem they ask, nothing more. If it's not the right path to a solution, they will decide, not you.
[+] [-] jypepin|6 years ago|reply
We get it, sometimes you are asking about X simply because you need to know about X and you know what you are doing.
But often, especially when dealing with more junior engineers, I've found this simple to keep in mind and helpful. It's also a good proof of experience to know when to ask about X and try to make sure you understand the context of the question before answering it.
[+] [-] MrStonedOne|6 years ago|reply
Bonus points, ask y again because the other question answered z only and get it closed as a dupe of the other question.
[+] [-] koala_man|6 years ago|reply
From my point of view, I'm really not trying to undermine what you're doing, it's a genuine attempt at getting you the best solution.
Here are some examples:
>How do I use apt on Fedora?
>How do I unroll this bash loop?
>How do I multiply this number by 1024 with sed?
I could have a lot of fun answering each question exactly as stated and if you say "I know it's the wrong tool for the job but I'm doing it for fun" that's exactly what I'll do.
However, it turned out that:
>They just wanted to install stuff thinking apt was universal. The better solution was using yum
>Their loop was slow and they had read that unrolling was an optimization. The better solution was reducing the number of subshells spawned
>They just wanted a result, not to implement an integer multiplication state machine in sed. The better solution was using perl/awk
I know some people are jerks about it, and that sometimes you want answers to weird questions. When people do appear to be going down the wrong path though, it would be a disservice to help them do that without letting them know that there's probably a better way.
Sorry for any false positives.
[+] [-] fishtoaster|6 years ago|reply
1. It's common for people trying to do Y to ask about X, which is a bad way to accomplish Y. 2. I'm trying to accomplish Z, for which X is a perfectly good option. 3. Every answer on stack overflow for "How to X" is people assuming the XY problem and answering "here's how you do Y instead".
[+] [-] vinceguidry|6 years ago|reply
The process of learning k8s and grokking the security concerns in the course of a few sprints is not pleasant. This "XY Problem" crops up all the time, but the devops team is absolutely slammed all the time so many times basic queries go unanswered. Everybody has to help everybody else.
I don't think anyone wants to go back to the days of rigid development infrastructure so we're kind of stuck with devops until the ecosystem can mature.
My great hope is for serverless to take off and so programmers can further specialize. I'm getting tired of learning new languages and stacks every few months or years. Just hand me a biz requirement and let me implement it any way I like.
[+] [-] nphase|6 years ago|reply
[+] [-] bobthechef|6 years ago|reply
For example, if someone random poses a problem on IRC, the solution to the "real" problem to be solved is NOT the concern or responsibility of the people in the channel. Unless there's a reason to ask for more context, just answer the damn question. Blindly asking the user what they're REALLY trying to solve is taking an intrusive and patronizing attitude. However, if asking will provide helpful context, then that's a valid reason to ask. If the problem as posed seems onerous or contrived, you COULD courteously ask the other person what they're ultimately trying to solve in the spirit of offering to be helpful if the other person so desires, but no one is hiring you do solve the "real" problem for them, so mind your own business.
If a coworker approaches you with a question, similar principles apply, but if you're working on the same project and in the same context, it may be natural to ask about what the larger problem is in a spirit of collaboration, esp. when the questions posed by your coworker smell funny. After all, you both share concerns about a project, though again, this does not give you license to automatically wrest the problem from your co-worker. His ticket, his problem. Your role is advisory.
So, common sense. Don't be one of those doltish know-it-all jerks who like to run the show. Mind the proper boundaries of concern.
[+] [-] drngdds|6 years ago|reply
[+] [-] an_d_rew|6 years ago|reply
Sometimes they are right. Often they are wrong.
Stop making assumptions, people!
[+] [-] winstonewert|6 years ago|reply
In case 1, you are a beginner. You have an XY question. But you've never heard of an XY question. You also probably couldn't identify your question as an XY question if you had heard of it. Trying to educate people about XY problem is pointless, because beginners don't see it or understand it or have the ability to apply it.
In case 2, you are advanced enough to avoid asking XY questions. You are instead stuck on some esoteric question which you can't resolve by exhausting all your skills. So you post a question on the internet. But you are only asking a question because it is a really hard question. Chances are that most other people cannot answer the question either. So, the only responses you get are people suggesting that you have an XY problem.
[+] [-] enriquto|6 years ago|reply
Just give me X or shut up!
[+] [-] dsr_|6 years ago|reply
When a name I've never seen before on a -users mailing list asks me how to do X -- and X is bizarre -- I assume we have the XY problem.
We have the XY problem because the Internet is full of newbies at any given point, and we've wasted enough attention-span having sixteen-part conversations about how to extract the IP address from ifconfig output* that we'd really like to know if we can skip all that and answer with "Use a dynamic DNS client, don't write your own."
*Don't. Use ip a
...
Use awk.
...
There are an infinite number of guides to awk, but mostly you need to know how it splits strings into fields and prints the ones you specify. How are you planning on handling machines with multiple IPs, by the way?
...
Stop right there. Get a dynamic DNS client.
[+] [-] burlesona|6 years ago|reply
If this is your work environment you need to make it okay to ask questions, and to dialogue about a problem without shaming the learner. (Really that should be civil behavior in any environment.)
[+] [-] micimize|6 years ago|reply