Ask HN: How do you deal with Rabbit Hole Syndrome?

[photon_off]

I find myself striving for ideal and perfect solutions in parts of my work that might not matter much. Sometimes it's probably worth the time and detail, but admittedly, a lot of the time it isn't. It's just more fun and interesting to be "thorough."

This happens when working on something new and unfamiliar, often side projects. In the quest for a "perfect" solution, entire branches of mathematics or programming start to beckon me, and away I go... down a rabbit hole of Wikipedia articles, Stackoverflow questions, and Github projects. Even if I do find an adequate solution, it's not good enough until I can understand the class of problem and derive it for myself.

This also happens based on principle. When I encounter a problem that is actually solvable -- though it may not matter much, then I must conquer it entirely. As an example, for me it is unacceptable that something rated 5.0 with 1 review is ranked higher than a 4.9 with 10 reviews -- and I need to find the solution to this problem, even though it might only be a minor feature and won't matter if there are no reviews to begin with.

I call this "conditional" Rabbit Hole Syndrome. It also sounds an awful lot like the classic interview answer: "My biggest flaw is that I work too hard." Well it's true, damn it. These types of trivial things can easily become my "top idea" for days at time.

I am now able to admit it: I spend time working on things that probably won't matter that much, and I enjoy it. But how do I fix this? How do I learn to leave well enough alone? How do I learn the distinction between frivolous research and actually getting shit done?

A couple of edits: After more thought, this is really only an issue with work on my side projects. I also think it's due to lack of accountability -- I report to myself. As such, I tend to work on "most interesting first" and avoid schlep. So I'm wondering: how do you motivate yourself to schlep?

[EvanMiller]

If I had one piece of advice it would be: Descend the rabbit holes!

Rabbit Hole Syndrome is a symptom of having a curious, persistent, independent mind. If there were a drug that cured Rabbit Hole Syndrome, and everyone just worked on Shipping Their Products and Not Asking Questions, there'd be no one left to make the interesting discoveries that make the world better in the long run.

There are very famous quotes from Richard Hamming (see "You and Your Research") and Richard Feynman (see "Surely You're Joking!") about the importance of working on problems that seem trivial at first. Not only do they help you enjoy problem-solving for its own sake, but if you work on enough silly problems, then eventually the odds are good that you'll stumble upon something that other people will later think is really important. Incidentally if you read Thomas Kuhn you'll find that this tends to be how scientific revolutions happen: someone is bothered by some tiny little thing that doesn't quite make sense according to the prevailing theory, starts digging at the little cracks, and finally the whole system comes crashing down.

The major benefit of going down rabbit holes is that is opens yourself to serendipity: sometimes you'll turn out to be right about something for the wrong reason, or discover something that you later realize contains a solution to a seemingly unrelated problem. The more rabbit holes you've gone down, the more they start to connect.

In addition, frivolous research helps you develop a very "bottom-up" view of the world. If you know the details cold, then you are better able to see through high-level BS ("conventional wisdom") and evaluate things on their own terms. You'll find that regardless of what's optimal, most things are done a certain way only because they've always been done that way, and that regardless of what's true, most people believe things only because other people believe them.

Of course there are costs to all this -- in particular, "schleppers" will resent you for being irresponsible, and most people will think you're crazy if you ever explain what stupid little you've been working on lately. Which is why Rabbit Holers should try to should avoid actual responsibilities to the greatest possible extent and absolutely not care about what most people think about their work.

Anyway, to sum up, if you're lucky enough to be in a position to descend rabbit holes without impoverishing your family or bankrupting your company, I say go for it. It strengthens your most valuable asset (your mind), and who knows, maybe one day you'll discover something you can teach the rest of us.

[chmike]

I second this. The gift you have is the drive and passion to learn by your self and explore. Let it go free and you'll become an expert. Soon or later you'll be able to turn this knowledge to profit for you and the collectivity.

Our education system is broken in that we don't recognize and develop the skill you have. People are somehow forced to do things they don't feel like being doing.

My impression is that it's ADD. But It's sad ADD is called a disorder, because these people have the remarkable ability to be extremely efficient and concentrated in what they like and are interested into, well above the average. These people could be experts and genius in their field, provided they could develop their skill.

[DutchGrinner]

I agree with your advice to "Descend the rabbit holes!" but feel it needs a little more context. The real question here is not 'should I descend rabbit holes?' but rather how often or when.

If you are building something, you inevitably spend some part of your time writing code that is familiar and boring (let's call this WORK), and part of your time exploring unfamiliar code and concepts (RABBIT HOLES).

Going down rabbit holes is not work, it is a form of learning - in fact, in the world of software development it is the best form of learning because you have a highly relevant problem and supreme motivation to solve it.

At the end of the day, however, going down a rabbit hole is no different than reading a book, or articles on HN. It is not actually work, and fooling yourself otherwise leads to the blurring of what is work and what is learning.

You should always be learning, as much as you can. But you must be honest with yourself, there is probably more work to be done, and you can't possibly get on with it until you pull yourself out of the rabbit hole.

[opminion]

Yet the conflict between "It's the trip, not the destination" and the fact that your salary arrives at the end of the month can cause much grief when not balanced in the mid term.

[adrianbye]

big round of applause. you nailed it completely

[aiurtourist]

You're driving along a road and you notice a pothole. You pull over to the shoulder, put your hazards on, open up the trunk, take out a reflective vest and tape measure, then you begin to analyze the pothole. You spend an hour analyzing the depth and formation of the pothole and determine that the cause is due mostly to poor mixing of asphalt. You call the town hall and learn that the road construction crew uses a Caterpillar BG500E wheeled asphalt paver. After some extensive research you determine that poor mixing occurs from the inferior design of the BG500E's auger and that upgrading to the BG600D with its improved auger would cause better asphalt mixing and produce paved road conditions less conducive to potholes.

By now it's 9:30pm. It's dark and cold. You realize what your original purpose was: Dinner with friends. That was two hours ago. You missed dinner, but hey, you got some satisfaction.

The above was more of an analogy about Yak Shaving than Rabbit Hole Syndrome, but there are parallels. Like you, I used to be obsessive about details and solving subproblems. I used to come home from work and work on my own side projects for similar reasons as your own.

But then an advisor said something like, "You need to focus. If you want this thing of yours to succeed, you have to focus on making it succeed. Nothing else should matter." So I stopped my side projects and I became so effective at building our product that my employees wondered if I ever slept.

Stay on target. Make it to dinner.

[photon_off]

Even within the context your product, surely there are cases where you're presented with a problem that can be solved with several different depths of thoroughness. How do you, generally, choose a solution which optimizes for, success, time, and personal gratification?

[npsimons]

The sad thing is, this example, while ludicrous, is perfectly justifiable, at certain scales. If that town is, say a big one on the East coast, your two hours (or even half a day, or whole day!) are little loss compared to the savings that will occur from the analysis. Hell, your hypothetical "curious person" would be well worth the investment of a six figure salary for the city!

That being said, I guess it depends on what results these random maunderings produce, and it's probably a crapshoot as to whether they are more often than not to result in favorable outcomes. Also, I like your example (obviously :)

[adamgravitis]

Heh. Are you Neal Stephenson? :-)

[antirez]

To be honest I think that "it's not good enough until I can understand the class of problem and derive it for myself" kind of attitude is actually very positive. The problem with this attitude is if you do a "depth-first" kind of journey instead of refining the solution with successive passes.

So I suggest you to try to build the minimum viable working program ASAP. Along the way, write all the things you would like to improve in something like an Evernote note, just a few lines for every thing you want to address and make better.

Then if you have something working ship it ASAP, don't care about what other people will say about the sub-optimal parts of your work: many programmers trying to achieve perfection actually end with a mess of complexity that does not serve very well the purpose of the software, so there is little to be embarrassed for a programmer for shipping simple software.

In the second pass, refine every part with the same approach: find a solution that within the timeframe you have is better compared to the previous one, but will make you able to ship a new version.

Also when you face a problem, other than reading the existing literature, papers, and the proper way to do it, check if there is an intuitive solution that is comparable as a result (even if maybe not provable or not perfectly optimal) but much simpler to implement.

But IMHO the golden rule is: don't freaking care, ever, about what other people think about your work. Often perfectionism is just a form of insecurity.

[alok-g]

Depth-first vs. breadth first, brilliant!

I get it now: If either one is taken to the finish line, they are the same (all nodes visited). But when left mid-way (time constraint), breadth first still is something workable while depth-first has a few components perfectly done while the others not at all.

>> Often perfectionism is just a form of insecurity.

Possibly brilliant too. How do I tell whether it is coming from insecurity or from my drive to not avoid shlep?

[bsimpson]

Just be careful about scope when you say "refine every part with the same approach" or you'll end up with TextMate 2.

[jewel]

Here are a couple of random suggestions:

1) Try the pomodoro technique, or some other form of time tracking. When you're about to take a side path which may or may not be a distraction, you can decide how long it'd be worth investing in it. Once the timer dings, you can stop and evaluate if it was worth it.

2) A few years ago I started repeating in my head the phrase "real artists ship". Embrace imperfect or partially finished solutions that are viable.

3) Keep a list of things that you'd like to investigate more. I've found that the act of writing down the idea lets me stop obsessing over it. Later when you revisit the list you'll be able to cross off the things that you thought were important, but turned out not to be. By delaying work on these items, you're able to better explore the most important parts of the problem you're solving, and so your future self will be in a better position to evaluate which areas need deep research.

[invalidOrTaken]

I think you're not being ambitious enough in your side projects.

I'm like you. I write fairly vanilla production code, then I go home and work on alien technology. And I used to have the same problem as you: I'd get sidetracked and sidetracked, and somehow I went from writing a fart app to reading about type theory.

So I started skipping the fart app part, and started learning some more abstract theory. The nice thing abstract theory is that it's abstract---it's not incidentally connected to anything, so there are fewer places for you to get sidetracked.

So go sign up for a math course on Coursera, or learn the lambda calculus. They are so alien from your everyday programming experience that you won't have anything to connect them to---until you do. But then you'll be coming at the new topic in the direction of abstract to concrete ("a trivial application of x") rather than concrete to abstract ("there's a greater truth here and I MUST understand it!").

[pcrh]

That was my thought. Anything that can be grasped solely by reading wikipedia or blogs isn't actually very cutting-edge. Not to say that stuff on wiki or blogs can't be very difficult to grasp, if previously unfamiliar.

[engtech]

This is an experience problem. As you become more experienced, and see how you've wasted your time coming up with a "perfect" solution that ended up not mattering, then you'll be fine not having a perfect solution.

At some point you realize your best days are the days when you delete more lines of code than you write.

But that being said, chasing rabbits is what will eventually make you more skilled than your peers. Which is awesome, except now you'll have put yourself in a perpetual category of being paid less than your worth because you don't fit the same performance evaluation criteria as everyone else.

All anyone cares about is a) are you easy to work with, and do you b) get things done to contribute to profitability.

Continutally stressing yourself out by spending more time on problems than they deserve and eating away at your work/life balance does not contribute to a) or b).

[dasil003]

> All anyone cares about is a) are you easy to work with, and do you b) get things done to contribute to profitability.

This gets to the crux of the matter. I think we all here have an intuitive sense that being a bit of a perfectionist and regularly chasing a few rabbits down holes can lead to better long-term results. But it's very hard to quantify, certainly it's impossible to quantify objectively; all you can do is be a good salesman about it.

There is another path though—put your money where your mouth is. Start your own business based on your chosen principles and standards of software engineering. It will certainly bring you joy and if your instincts are right then it could even become a competitive advantage as your competitors drown under mountains of technical debt while you nip it in the bud at the first sign of trouble.

[Detrus]

Yea experience of wasting your life in rabbit holes, combined with getting older while getting little accomplished may help you focus. Unless you really find something in the rabbit holes, then down you go.

If you never find anything and still can't help yourself it's some lethargy/depression ADD brain dysfunction. Make sure to exercise enough to be in shape, eat, etc.

[davidw]

> I find myself striving for ideal and perfect solutions in parts of my work that might not matter much. Sometimes it's probably worth the time and detail, but admittedly, a lot of the time it isn't. It's just more fun and interesting to be "thorough."

This actually has a label: Maximizers vs Satisficers.

http://en.wikipedia.org/wiki/The_Paradox_of_Choice:_Why_More...

And a little something I wrote about it regarding programming languages:

http://journal.dedasys.com/2006/02/18/maximizers-satisficers...

I think the key is to "choose your battles", and be a maximizer where it really counts, and try and be more of a satisficer for the things that aren't so important.

[georgefox]

I was just going to post something very similar. Glad to see it's already been done for me. :)

I'm in the middle of The Paradox of Choice right now, and while the maximizing/satisficing thing applies to many aspects of life—the book was certainly not written about programming—programming is yet another aspect of life where we're confronted with a huge array of choices.

This is especially true in side projects. In the office, there may be a standard set of technologies; you're likely working on an existing product; etc. Additionally, there are probably a number of constraints, including time and product requirements. In side projects, there are hardly any constraints, and every single aspect of your work involves choice. Just deciding what you want to create is one massive choice. Then there are the big questions of "what programming language/framework should I use?" and "what database(-ish) platform do I want to use?" There's a good chance since this is a side project, you'll want to try out something new and cool to build your knowledge and gain experience. Those alone can be huge decisions. Then there are questions of libraries (which library? or build it from scratch?), etc. Heaven forbid you want to start thinking about your choice of IDE (time to learn Vim?) and tabs vs. spaces and every other little decision you can get caught up on.

I'm at least as guilty as anyone else of these things. I spend too much time focusing on the best possible outcome and the best possible technologies among lists of hundreds and thousands—because I want the best and because I don't want to regret any failure—rather than simply settling for something that's good enough and moving on to actually get something done. The result is generally painful and unproductive.

You should definitely set standards for your work—don't produce crap—but try bringing the bar down from "ideal and perfect" to something more attainable like "excellent" or "very good" or even just "adequate." This will help you focus on your real goals and appreciate your own success.

Another useful term/link: http://en.wikipedia.org/wiki/Analysis_paralysis

[samstokes]

As with others on this thread, I question the rightness of "curing" this tendency. But you can certainly improve the way you harness it.

Something that's helped me: make sure you have multiple rabbit holes available at any given time. i.e. several problems, any of which is interesting enough to tempt you. Then you can make reasoned decisions about which would be the most rewarding to work on, while still giving in to the temptation of rabbit holing.

Another benefit: maybe by solving one problem, you'll discover the other was totally irrelevant, or a special case of the other. (At least for me, "Turns out I didn't need this to be perfect" doesn't carry much weight, whereas "Turns out I didn't need this at all" is quite convincing.)

[richardkmichael]

This happens to me often as well. Although, I am not certain I would describe it as something I need to fix. All my practical experience, going back 15+ years has formed this way.

As a recent example, I just spent a long time on a tiny plugin for CarrierWave (the Ruby/Rails file uploading gem). This took me through most of their source code (always good to read code, and I could probably PR on their project now), new Rails internals and techniques, and mocking and stubbing with RSpec (in the process, I turned up a bug with RSpec, which they promptly fixed; and I have a better understanding of mocking best practices and clean RSpec).

I do empathize with feeling I'm "not getting enough done/shipped" (if you feel that way?). To alleviate it, I try to cut corners and just get something out.

Time-boxing helps me do this -- "accomplish X in 1 hour". This doesn't happen too often, however. I know, like you, I prefer the learning itself; and, I view all my spent time as building experience and a critical mass to accomplish work faster in the future.

Also, I find pair programming helps immensely. I am sensitive to the other person's time, and thus naturally refrain from digression when pairing.

As for motivating myself to "schlep". I'm not sure what you mean -- menial tasks? I do those when I'm tired or having down time.

Finally, as in your case, I do this on personal projects and not on "work". That said, it can make it hard to get your startup and product launched - your own "work". Again, disciplined time-boxing helps. I have not mastered this, and find myself regularly looking for good time-boxing tools.

[Sujan]

I write a ticket for someone else to take care of it. Even if someone else is only future me, it's enough for my mind to accept that it is taken care of and I can proceed with the really important stuff.

Most of these tickets get resolved as "doesn't matter / won't fix" two or three weeks later - but myself.

[a3_nm]

I agree with this. I do theoretical CS research, and whenever I find a general problem that I feel must have been solved by someone but seemingly hasn't, the only way I can get the problem off my head is phrase it as a StackExchange question (usually on cstheory) and ask it.

First, it forces me to write down clearly the question that's obsessing me, occasionally leading me to realize that doesn't actually make sense or isn't that interesting -- all the more so because it is published so my standards are higher that if I just wrote it in a text file of mine. Second, it gives me the impression that people will work on it now so I can stop thinking about it for a while (even if no one actually cares about the question). Third, in any case the question is documented so it won't be lost.

In some cases I get insightful answers (sometimes, brilliant answers) that save me the time to hunt down a solution in an unfamiliar field or to work it out by myself, other times I end up replying to my own question because I stumbled on the solution later. In any case, it totally defuses the urge to think compulsively about a problem.

[misleading_name]

Catalog your rabbit holes if possible, and then review them before working on them.

What I mean is, if you see a problem to solve, and you are able to keep working on your current task and to solve the problem later, then do so... just note the problem. This will immediately make you much more focused.

Then at the end of the day, review your list of rabbit holes and try and determine which ones are necessary for the current project, which ones would be educational / you want to do, and which ones can be discarded.

Basically rabbit holes are a problem because they are long and narrow and do not offer an overview of the entire grounds, so before jumping down a rabbit hole force yourself to survey the big picture and to see if you can step over it instead.

[rizz0]

You apparently enjoy the interesting stuff, and love the fact that there is actual practical benefit - though initially very limited - from doing something that's theoretically interesting. Letting that chance slip would feel like a waste.

It might help to limit going down the rabbit hole too much, by researching what could be improved without actually doing it right then. Save it in a list for later, do the schlep, then when things get bigger chances are you'll actually need to take on some of the challenges on that list. And the more useful they become, the more satisfying it'll be.

[kronholm]

Kind of sounds like procrastination to me. You should have a clear goal, and work towards it, allowing no or few of these side-distractions to take you off the path. A real life deadline helps immensively as well.

[shurcooL]

Yeah, I would suggest you make sure to read the top answer at www.quora.com/Life-Advice/How-do-I-get-over-my-bad-habit-of-procrastinating

I found that to be quite helpful in maximizing your happiness by getting done what you most care about in the long term.

[aaronbrethorst]

What else do you enjoy? Do you derive satisfaction from solving problems for users? Maybe you're trying to perfect something that's less interesting to you than another problem on your plate.

If positive user feedback 'gets you off,' as it were, try releasing something that isn't as perfect as you'd like it to be and see if there's any measurable difference in sentiment. I bet there won't be. Do it a couple more times and hopefully it'll break the pattern.

For the other case, try time boxing yourself so you can get back to the more interesting challenge. Make that other thing 'perfect' if you won't.

[handzhiev]

Never happens to me. I'm always eager to finish, release and go on. Good enough for the users? Publish. Not so good? Publish. Some will bear it, the rest will swear it and I'll know what to fix.

I'd very much prefer to pick up an apple that isn't perfect than having no apple at all because of looking for the perfect one forever.

[benzesandbetter]

A couple possibilities. If you really enjoy doing these things for their own sake, you can treat your side projects as a hobby and stop beating yourself up about it. Spend all the time you want going down the rabbit hole, and enjoy that it provides a chance to think deeply, making something as close to perfection as possible, and be fully present in the experience of creating.

The other approach I'd suggest is at the complete opposite end of the spectrum. Start treating your side projects like a real business, and create some process that connects real incentives to them. There are harder and softer ways to approach this. You can quit your job and now you have the focus in your side projects that you need to ship to get paid. You could also keep your day job and set up some automatic funds transfers into a semi-liquid asset like a CD or 401k, such that you leave yourself enough to cover the basic expenses, but so that the marginal income from your side project becomes significant enough to be motivational. Revenue is a great feedback loop getting you to focus on what matters, and letting go of what really doesn't.

You could also bring in another person, perhaps as a business partner, or simply have one of your friends agree to call you up once a week and have you give them a report. The key here is to clarify and draw attention in your mind to the work that you're avoiding, and the outcome that you want. In this process, it's important to vividly color the positive outcome be emphasizing the real changes it will create in your life.

For example: "This week, instead of shipping my product which will enable me to quit working for other people, spend more time with my family, own the high-performance german cars I've always wanted, and visit 20+ countries a year, I instead spend 6.5 hours working on a sort algorithm, and 5.4 hours fiddling with the presentation styles on the country-list drop-down menu in my app."

I used all of the techniques above when I was starting my first company, and they made a big difference in helping me focus on what was important in getting to market, and not OCD-ing out on things that were, ultimately, minor details.

[msutherl]

I think the key is to:

1. Set goals and fixate on them. Imagine what it will be like to have achieved the goal. Get excited about it and keep reminding yourself. When you're not making progress toward the goal, make yourself imagine the situation where you spend 6 months making unimportant things perfect and never achieve the goal. Imagine all the other goals you won't even be able to set because you're wasting your time.

2. Make other commitments. Make plans to meet a friend for drinks at 9pm. You only have 2 hours after work to get anything done – don't waste those 2 hours! If you can stay consistently busy, you'll notice quite quickly that not using your time effectively will lead you nowhere.

[mblake]

I've been very affected by this myself.

It happened the most while attempting to study research papers.

A lot of times I would be unable to finish reading what I had planned, because I had stumbled upon an interestingly-looking concept and then proceeded opening up another paper on that subject and so on.

And then, of course, having returned back to the original paper, I would have to re-start reading from beginning to freshen up my memory.

Terribly exhausting process, so I can perfectly understand your frustration.

The only solution I found was to un-clutter my computer 'work-space' as much as possible: close any non-essential apps, unplug internet cable and each time I have the urge to stop reading and go research a newly discovered subject, I remind myself that I am only allowed to do it after finishing what I'm currently reading.

Another helpful thing for me is to remind myself what I'm trying to accomplish with whatever I'm doing at that particular time and what my long-term goals are.

For instance, if I'm working on a project for a client, the goal is to get the work done as soon as possible and obviously get paid. I am not working on said project to primarily enrich my knowledge, but to make money.

I can use the time after the project is delivered to draw conclusions from the experience or do further research.

This may sound trivial, but it really does help to constantly remind yourself of what your goals are, it keeps you in check.

For extra effect, every time you have the urge to let your mind wander too much, try imagining the possible consequences of not completing your task (on time).

This can be particularly effective if you're doing client work. Imagine how embarrassing/unprofessional would have to explain to your client/employer that you won't be able to deliver on time because of something that you could've prevented.

Side-note: you have not provided enough info, but you may have obsessive-compulsive disorder, which is nothing to be ashamed of, but you can only 'solve' this with medication, so you would need to see a doctor.

[jimboconway]

Two ideas: (1) Find a partner for each side project, or at least 1 target user (real person you know) you commit to serving by a hard date. It's an uphill battle to maintain accountability without another human. (2) I draw a box on my whiteboard and write in it the essence of the project and only the 2-3 deliverables essential to achieving it. When I take on a new activity, I challenge myself as to whether it goes in the box or represents a branch from it. Forces me to visually acknowledge how I am spending my time (and thus define and stay to the essence of the project).