Although it's a fun read, it's a classic example of diving into coding without giving a project it's due diligence. There's nothing that's derailed my projects more consistently when I started out as not understanding the user's needs. They'll never tell you what they want, only what they don't want after you deliver something.
I think that's the key difference to an experienced dev/BA. One who can actually sit with the stakeholders and build the system on paper and go through each of the problems as the diagrams connect. What you end up with is the stated requirements (tip) and the unstated assumptions (iceberg).
These types of projects are easily spotted as they're often called "quick" or "easy", which in layman's means no one's really thought about it yet.
Kind of agree with you and OP, so let me add to the discussion with the caveat that YMMV etc. Some of these points may overlap with comments by fellow HNers in these threads.
In any organization as a prospective customer, there are three forces at work that matter to a consultant: users, decision makers and finance folks. Ideally, the decision makers appoint a dedicated person (or team) who mediates users and finance guys to create a coherent road map for the system to be built. What tends to happen in practice is a combination of chance, human errors and conflicts in priorities.
To survive, we must understand ourselves as consultants, the terrain and the opposing forces (anything that prevents money getting deposited in the bank).
Consulting is a business => the individual consultant must shoulder quite a lot on himself:
* Advertising himself
* Finding work
* Meeting prospective customers
* Writing proposals
* Closing the deal
* Writing a contract
* Defining the scope of work
* Nailing what is to be done and what is to be avoided
* Constructing components of software
* (In worst cases) baby-sitting customer's devs / ops
* Testing his modules
* Getting money in his account
So, when we talk with customers, there is inherent pressure in closing the deal versus precisely defining everything. Icing on the cake: back of the mind, you might worry about milkman's bills and the birthday gifts to be brought for your kid's friends. Always have a good runway: six months seems optimum. Technically, even three months should suffice, but it is insufficient for the edge case when these three things hold true simultaneously:
* You've to walk away from a customer in the middle of a project, and
* Dry sales pipeline, and
* Unexpected crisis at the personal front: very close friend in need of money for surgery etc.
God be with you.
Now, back to the main track. The key is to first understand the priorities of stakeholders. The contradictions are easier to grasp once we put ourselves in the individual's shoes:
* [Convenience] The users want everything including the kitchen sink,
* [Accountability for the risk] They want it yesterday but won't commit to it on paper since you're in early sales stage,
* [Cost] The finance folks want stability and the most economical price,
* [Availability] The decision maker prefers to spend the least of her time to be spent on this activity, except when it's her pet project.
These roles may be played by different people or the same person.
Again, we don't have to agree, but acknowledge that things are meant to be imperfect. Most people are actually decent and rational. Now, somehow, the consultant must figure out a good path. To begin, he should have a good handle on:
* The industry in which the customer operates, e.g. BFSI, Energy, online marketplaces, food, fashion etc.
* The nature of the market: oligopoly, crowded, emerging et al.
* The level of Govt. regulation: HIPAA? PCI?
* The customer's position: among top tier, mid-size, small firms.
* Customer's customers: helps to track the wind.
* Standard pain areas in the market.
* How customer's competition and partners survive the pain, specifically the differences among the customer's approach and the competitor's ones.
* The financial health of the customer: if possible, try to talk with their existing suppliers in other arenas (e.g. the guy who supplies printer cartridges, the cleaning lady etc.).
Based on that, assuming we've landed a hot lead, say that we want to convert it to money. There will be a lot of conversations over the phone, e-mail, chats, face to face; what is to be left out of those conversations is really context sensitive. The important parts are:
* What problem we're going to solve for all stakeholders?
* How we're going to solve it?
* Communicating our approach and progress on a timely basis.
* Improving Vulcan skills: like money, expression of anger is a great servant but a terrible master.
I leave out the details on the development process, since it depends on the individual.
Here are a few things which worked for me marvellously in certain contexts and doomed in others:
* Setting expectations by explaining the cone of uncertainty in the estimation process. Always be prepared for some customers, who deliberately mislead you that they don't understand the mumbo-jumbo, projecting ignorance as a shield so that they can ruin you on payments. Mostly you should refuse such customers, unless you're in an extreme crisis on multiple fronts.
* Writing use cases for the modules / systems I was supposed to work. Most customers say that they don't want any time spent on "documentation," to which I reply that writing a document is not "typing" but "nailing the thoughts to deliver higher quality work."
* Creating mock ups using sketching tools (even Excel/LibreOffice can be good enough, if one is pressed too much for time) in the absence of UX designers on the team.
* Writing test cases before writing my code. Helps to ease the integration.
* Optimizing lazily: progressively refining the artefact (document / mock up / test / code).
* Knowing the expected value of each task:
(amount of money * satisfaction * improvement in reputation) / (e-mail tennis * stakeholder's understanding of requirements * payment latency * time spent)
Finally, reducing it to just one sentence: be nice, believe in yourself and humanity, be multi-dimensional and please heed Guy Kawasaki's advice about reading voraciously and omnivorously.
The flip side of this is that when stakeholders/customers don't know what they really want or need, you can labor in the "requirements phase" for months. My team spent 900 hours last year just getting a functional spec and high level SRS together for a project that hasn't happened yet. Why hasn't it happened? Because at the end of the 900 hours, we presented our estimate (about 1100hrs) to the customer and he balked at the cost & time. Then my boss bitched at us for spending so much time on requirements in the first place.
<8 months pass>
Now we're looking at commercial solutions in an attempt to not have to build this thing and deal with this department anymore.
I agree with you 100%, but the number of people astute and experienced enough to quickly and deeply understand the problem to be solved (and the quirks of the customer) are few and far between. In my experience, perhaps 5-10% of any large IT organization fall into this category -- everyone knows who the superstars are and their time is always at a premium. Drives me bonkers.
The tricky part is often convincing a client that pre-planning is a necessary part if the process and, though it is billable time, will ultimately save money. Many client don't appreciate that programming is not just writing code.
Also you can wind up meeting and planning yourself right out of a gig sometimes - which is fine if the gig wasn't truly necessary. But can be bad for your business if you spend hundreds of hours doing this for no compensation. This is a real situation that happens - you get a client that wants a big project and after many meeting and calls you determine that a Wordpress install will suit them perfectly. Their network guy does the install and for all the money you saved them your reward is that you don't get the gig!
The real issue is sales...you make the sale because you have to. Then you stick the project on some poor PM/Ba/Dev who will fight with them forever over this stuff.
Maybe I'm just bitter about the situation that I continually get stuck in.
For those who don't know about the author of this post. His name is Mark Jason Dominus, the author of one the most awesome programming book that one can ever read.
Higher Order Perl, is available for free download. If you read it you will see some amazing insights into programming techniques most people would have never heard of encountered in MegaCorp jobs. You will also grow a great appreciation for Perl in general and understand how it can be an amazing language of choice for a wide variety of problems.
Sorry if it reduces self-esteem, but we (software developers) are not engineers.
I've been brainwashed for many years by my brother, who is a Phd civil engineer, that professions that hijack the term engineer are kidding themselves.
I know it sounds cool, and my current title is 'Senior Software Engineer', but it is all a scam.
I had a "Senior Software Engineer" title after only about 2-3 years of real-world programming experience, at age 23 or so. I had shipped a couple games as lead by then, so it wasn't entirely unearned.
US corporations tend to do that. Instead of a big raise, small raise and "senior". That can even be negotiated in hiring, so you may join as senior for a position just created in the market.
I'm surprised nobody, not even the author, has said that the program was useful anyway. But instead of publishing it in the web, it should have been used by the toll-free number operators.
I never cease to be amazed that the more things change, the more they stay the same. If you removed the dates from this story, you'd be hard pressed to fix when this happened.
(Except the static HTML idea is a bit of a give away, no web developer would ever dare suggest a static page in our brave new world of Web X.0)
So true. Before leaving to focus on more interesting things I've spent a chunk of my career working developing software services for some big insurers. This tale from 1995 could easily be a story of the challenges we faced in 2012/13.
Data managed in spreadsheets, layer upon layer of disconnected management, no engagement with users, very little willingness to understand software development. More spreadsheets...
> These days I would handle this easily; after the first or second iteration I would explain the situation: I had based my estimate on certain expectations of how much work would be required; I had not expected to clean up dirty data in eight different formats; they had the choice of delivering clean data in the same format as before, renegotiating the fee, or finding someone else to do the project.
Great advice for dealing with issues we did not consider in our estimate.
Had he charged hourly instead of a fixed price, would this project have been less shitty?
I worked at Prudential about 10 years ago, as a FTE. Our small division mainly ran on a bunch of custom Access reporting applications. It wasn't quite cutting it, because Access, so it was decided that we would build a portal on the company's intranet. The only problem was that we, as accidental web developers, were not allowed to run development web servers on our dev machines, because they were locked down by corporate. We had to use an extra PC that, by some miracle, had IIS, and develop against that remotely. Good times.
The title reminds me of the shittiest software job/project I ever worked on- short and sweet: Got hired off craigslist. It was all php. Their main competitor was "the spreadsheet". Worked directly next to a cold caller that repeated the same phrase over and over again, fake laugh and all. They were all from the same church group and tried to convert me multiple times. Paid minimum wage.
In addition to being funny in retrospect, it was a good lesson to me to learn that no matter how shitty your current situation, you can always improve it.
The punch line here is great, but even if the specs necessitated something more than a static page, then it'd still be a hard job.
If the most critical part of a data heavy project is speccing it out, I'd say the next most important part is the data munging process...and sadly, both of these things are the most overlooked.
I'm sure this is the very tough first lesson any new consultant with limited experience would learn. The summary is that the specs are incredibly important, they should be expensive to produce and they should protect you and your client.
tl;dr - A very long, difficult, and complex calculation multiplied by zero is zero. Or in this case, a very long, difficult, and complex database manipulation culminated in an invariant "Call 1-800-555-5555".
Somehow reminds me of a co-worker wandering into my cube with a dazed look, mumbling "I just ran an 8-hour database job with the wrong data...I just wasted $64,000...".
That's not bad at all, it's sounds more like business as usual. I think it's my regular workday.
Btw. Did the customer require extensive documentation, escrow, you to fix their data when they can't get it done (like invalid post area codes linked to wrong addresses, fixing post number / city information based on post area code) etc or looking it up based on street adress or so. Been there, done that. Did you spend several weeks in meetings where they can't decide how their stuff works, and you'll just keep wondering if they'll ever decide what they actually want etc. Of course they're going to completely change that week later etc. They don't have a clue how things should technically work, or even what the actuall business process is. Because they have bought an integration, everything must just automatically work, right? We need to know what females in age range 20-25 have bought during last month. Err, but we don't record customer age or sex? Well, but our management team needs that information. It's sure alarm sign, that they want to know how much "this" project will cost, but nobody really knows what "this" is. Also it's needs to be completed by end of the month. I have declined so many projects, and clearly told customers why I'm not going to do anything for them at all. Unless they accept it as "agile" project, with unlimited budget. Then I'll promise them that I'll personally see that it gets done, but it's going to be expensive. - 15 years of ERP/POS/BI/CRM integration programming & consulting.
I am working on quite a tedious project right now. It involves 10 years old, quite extensive, Visual Basic 6 programs. No source control was used. In our company it is practice to hire interns for 3-month periods to work on production software. A mix of programming styles can be found in this project. Some functions return 0 when they fail. Others return 1. Or -1. Or False. Or "False". I love it!
"In 1995 I quit my regular job as senior web engineer for Time-Warner and became a consultant developing interactive content for the World-Wide Web, which was still a pretty new thing at the time."
I am confused the person stated that they were a senior web engineer but quit to work on a new thing the WWW. How can you be a senior web engineer for something brand new?
i dont think the problem lies with the consultant tho - how could he have figured out the actual business problem, if no body bothered to tell him? He was hired to do a particular task, and that task was pre-determined by some "stakeholders" who dont actually know what they want.
I can highly recommend Flawless Consulting by Peter Block.
He covers all these issues and many others. Self awareness is critical to success. If you are inexperienced you need to be able to recognize/acknowledge your inexperience. Then read everything available on the subject and interview every expert you can identify. Clint Eastwood said it best, "A man's got to know his limitations". http://www.youtube.com/watch?v=_VrFV5r8cs0
Fascinating! As somebody who is starting to do freelance work a story like this is not only an interesting read but provides insight on what to avoid and when to speak up as I start freelancing.
The shittiest project that I worked is just finished where we are not allowed to write any test cases cause we don't have time. I just can't believe that I finished fairly big project without writing a single test case. I hate my company, my role and managers, sales people who negotiate tight deadlines. Over all don't work for any consulting companies out there. They care less of the code quality. They just want money and no ethics.
[+] [-] redact207|12 years ago|reply
I think that's the key difference to an experienced dev/BA. One who can actually sit with the stakeholders and build the system on paper and go through each of the problems as the diagrams connect. What you end up with is the stated requirements (tip) and the unstated assumptions (iceberg).
These types of projects are easily spotted as they're often called "quick" or "easy", which in layman's means no one's really thought about it yet.
[+] [-] shailesh|12 years ago|reply
In any organization as a prospective customer, there are three forces at work that matter to a consultant: users, decision makers and finance folks. Ideally, the decision makers appoint a dedicated person (or team) who mediates users and finance guys to create a coherent road map for the system to be built. What tends to happen in practice is a combination of chance, human errors and conflicts in priorities.
To survive, we must understand ourselves as consultants, the terrain and the opposing forces (anything that prevents money getting deposited in the bank).
Consulting is a business => the individual consultant must shoulder quite a lot on himself:
* Advertising himself
* Finding work
* Meeting prospective customers
* Writing proposals
* Closing the deal
* Writing a contract
* Defining the scope of work
* Nailing what is to be done and what is to be avoided
* Constructing components of software
* (In worst cases) baby-sitting customer's devs / ops
* Testing his modules
* Getting money in his account
So, when we talk with customers, there is inherent pressure in closing the deal versus precisely defining everything. Icing on the cake: back of the mind, you might worry about milkman's bills and the birthday gifts to be brought for your kid's friends. Always have a good runway: six months seems optimum. Technically, even three months should suffice, but it is insufficient for the edge case when these three things hold true simultaneously:
* You've to walk away from a customer in the middle of a project, and
* Dry sales pipeline, and
* Unexpected crisis at the personal front: very close friend in need of money for surgery etc.
God be with you.
Now, back to the main track. The key is to first understand the priorities of stakeholders. The contradictions are easier to grasp once we put ourselves in the individual's shoes:
* [Convenience] The users want everything including the kitchen sink,
* [Accountability for the risk] They want it yesterday but won't commit to it on paper since you're in early sales stage,
* [Cost] The finance folks want stability and the most economical price,
* [Availability] The decision maker prefers to spend the least of her time to be spent on this activity, except when it's her pet project.
These roles may be played by different people or the same person.
Again, we don't have to agree, but acknowledge that things are meant to be imperfect. Most people are actually decent and rational. Now, somehow, the consultant must figure out a good path. To begin, he should have a good handle on:
* The industry in which the customer operates, e.g. BFSI, Energy, online marketplaces, food, fashion etc.
* The nature of the market: oligopoly, crowded, emerging et al.
* The level of Govt. regulation: HIPAA? PCI?
* The customer's position: among top tier, mid-size, small firms.
* Customer's customers: helps to track the wind.
* Standard pain areas in the market.
* How customer's competition and partners survive the pain, specifically the differences among the customer's approach and the competitor's ones.
* The financial health of the customer: if possible, try to talk with their existing suppliers in other arenas (e.g. the guy who supplies printer cartridges, the cleaning lady etc.).
Based on that, assuming we've landed a hot lead, say that we want to convert it to money. There will be a lot of conversations over the phone, e-mail, chats, face to face; what is to be left out of those conversations is really context sensitive. The important parts are:
* What problem we're going to solve for all stakeholders?
* How we're going to solve it?
* Communicating our approach and progress on a timely basis.
* Improving Vulcan skills: like money, expression of anger is a great servant but a terrible master.
I leave out the details on the development process, since it depends on the individual.
Here are a few things which worked for me marvellously in certain contexts and doomed in others:
* Setting expectations by explaining the cone of uncertainty in the estimation process. Always be prepared for some customers, who deliberately mislead you that they don't understand the mumbo-jumbo, projecting ignorance as a shield so that they can ruin you on payments. Mostly you should refuse such customers, unless you're in an extreme crisis on multiple fronts.
* Writing use cases for the modules / systems I was supposed to work. Most customers say that they don't want any time spent on "documentation," to which I reply that writing a document is not "typing" but "nailing the thoughts to deliver higher quality work."
* Creating mock ups using sketching tools (even Excel/LibreOffice can be good enough, if one is pressed too much for time) in the absence of UX designers on the team.
* Writing test cases before writing my code. Helps to ease the integration.
* Optimizing lazily: progressively refining the artefact (document / mock up / test / code).
* Knowing the expected value of each task: (amount of money * satisfaction * improvement in reputation) / (e-mail tennis * stakeholder's understanding of requirements * payment latency * time spent)
Finally, reducing it to just one sentence: be nice, believe in yourself and humanity, be multi-dimensional and please heed Guy Kawasaki's advice about reading voraciously and omnivorously.
Edit: formatting.
[+] [-] greenyoda|12 years ago|reply
Reminds me of an old saying: "If you don't know how to do it, you don't know how to do it on a computer."
[+] [-] eitally|12 years ago|reply
<8 months pass>
Now we're looking at commercial solutions in an attempt to not have to build this thing and deal with this department anymore.
I agree with you 100%, but the number of people astute and experienced enough to quickly and deeply understand the problem to be solved (and the quirks of the customer) are few and far between. In my experience, perhaps 5-10% of any large IT organization fall into this category -- everyone knows who the superstars are and their time is always at a premium. Drives me bonkers.
[+] [-] misterjangles|12 years ago|reply
Also you can wind up meeting and planning yourself right out of a gig sometimes - which is fine if the gig wasn't truly necessary. But can be bad for your business if you spend hundreds of hours doing this for no compensation. This is a real situation that happens - you get a client that wants a big project and after many meeting and calls you determine that a Wordpress install will suit them perfectly. Their network guy does the install and for all the money you saved them your reward is that you don't get the gig!
[+] [-] grumps|12 years ago|reply
The real issue is sales...you make the sale because you have to. Then you stick the project on some poor PM/Ba/Dev who will fight with them forever over this stuff.
Maybe I'm just bitter about the situation that I continually get stuck in.
[+] [-] Agathos|12 years ago|reply
The formation of this sentence seems odd to me. Why can't we say it's a fun read because it's a classic example etc.?
[+] [-] perfunctory|12 years ago|reply
If everybody was so picky the size of the IT industry would diminish by 90%.
[+] [-] kamaal|12 years ago|reply
Higher Order Perl, is available for free download. If you read it you will see some amazing insights into programming techniques most people would have never heard of encountered in MegaCorp jobs. You will also grow a great appreciation for Perl in general and understand how it can be an amazing language of choice for a wide variety of problems.
[+] [-] angrow|12 years ago|reply
They needed a competitor.
[+] [-] fusiongyro|12 years ago|reply
You had the job title "senior web engineer" when the web was 4 years old. That's pretty cool.
[+] [-] adamconroy|12 years ago|reply
I've been brainwashed for many years by my brother, who is a Phd civil engineer, that professions that hijack the term engineer are kidding themselves.
I know it sounds cool, and my current title is 'Senior Software Engineer', but it is all a scam.
[+] [-] SomeCallMeTim|12 years ago|reply
[+] [-] gcb0|12 years ago|reply
[+] [-] narag|12 years ago|reply
[+] [-] nekopa|12 years ago|reply
(Except the static HTML idea is a bit of a give away, no web developer would ever dare suggest a static page in our brave new world of Web X.0)
[+] [-] jcoder|12 years ago|reply
[0] http://kylerush.net/blog/meet-the-obama-campaigns-250-millio...
[+] [-] thekingshorses|12 years ago|reply
Page is static. JSON is a static file with a phone number. And use jQuery to load the JSON file.
[+] [-] benjamta|12 years ago|reply
Data managed in spreadsheets, layer upon layer of disconnected management, no engagement with users, very little willingness to understand software development. More spreadsheets...
[+] [-] ams6110|12 years ago|reply
[+] [-] kabisote|12 years ago|reply
Great advice for dealing with issues we did not consider in our estimate.
Had he charged hourly instead of a fixed price, would this project have been less shitty?
Edit: Fixed grammar. Thanks b0z0. :)
[+] [-] LordHumungous|12 years ago|reply
[+] [-] unknown|12 years ago|reply
[deleted]
[+] [-] b0z0|12 years ago|reply
[+] [-] LargeWu|12 years ago|reply
I worked at Prudential about 10 years ago, as a FTE. Our small division mainly ran on a bunch of custom Access reporting applications. It wasn't quite cutting it, because Access, so it was decided that we would build a portal on the company's intranet. The only problem was that we, as accidental web developers, were not allowed to run development web servers on our dev machines, because they were locked down by corporate. We had to use an extra PC that, by some miracle, had IIS, and develop against that remotely. Good times.
[+] [-] the_cat_kittles|12 years ago|reply
In addition to being funny in retrospect, it was a good lesson to me to learn that no matter how shitty your current situation, you can always improve it.
[+] [-] chetanahuja|12 years ago|reply
[+] [-] danso|12 years ago|reply
If the most critical part of a data heavy project is speccing it out, I'd say the next most important part is the data munging process...and sadly, both of these things are the most overlooked.
[+] [-] dsugarman|12 years ago|reply
[+] [-] mjd|12 years ago|reply
[+] [-] ovoxo|12 years ago|reply
[+] [-] ctdonath|12 years ago|reply
Somehow reminds me of a co-worker wandering into my cube with a dazed look, mumbling "I just ran an 8-hour database job with the wrong data...I just wasted $64,000...".
[+] [-] alayne|12 years ago|reply
[+] [-] cranklin|12 years ago|reply
[+] [-] zeckalpha|12 years ago|reply
[+] [-] Sami_Lehtinen|12 years ago|reply
[+] [-] donquichotte|12 years ago|reply
[+] [-] wil421|12 years ago|reply
I am confused the person stated that they were a senior web engineer but quit to work on a new thing the WWW. How can you be a senior web engineer for something brand new?
Just nitpicking...
[+] [-] Moto7451|12 years ago|reply
[+] [-] ibudiallo|12 years ago|reply
[+] [-] tobiasbischoff|12 years ago|reply
- not enough questions asked - not listened to the customer
Do not ever start building something or proposing a solution if you don't understand the customers problem deeply enough.
[+] [-] chii|12 years ago|reply
[+] [-] sfbsfbsfb|12 years ago|reply
[+] [-] jetti|12 years ago|reply
[+] [-] grumps|12 years ago|reply
[+] [-] scriptstar|12 years ago|reply
My next job will be a product based company.