His distinction between 'engineer' and 'developer' is artificial and easily criticized. That isn't what is interesting in this article. There are a lot of things that are.
Their method for avoiding the proliferation of different tools is interesting.
His discussion of the kind of people they want, once you ignore the semantics, is interesting. They want people who take responsibility and are always pushing the boundary of their knowledge into new fields. That's interesting.
My favorite part of the article was right at the end: "instead of asking questions about “why did something fail,” we want to ask why something succeeded, which is really easy to skip over."
I agree that the use of the 'engineer' vs 'developer' terms in his context was not important.
What does irk me is that in real life, software developers who meet the Etsy CTO's standard often can't call themselves engineers. In many countries, misrepresenting oneself as an engineer without accreditation is against the law! [1]
In the U.S., there's no official occupational category for Software Engineer. It's just Software Developer (Applications), Software Developer (Systems), and Computer Programmer.
It is possible to get accreditation (http://ncees.org/exams/pe-exam/, under Software), but it didn't appear very useful to me aside from being able to call yourself an Engineer.
Will I be rewarded for responsibility of imperfect things? Will the company provide compensation and the necessary environment (trial equipment and software) for training to push the boundary of my knowledge?
Programming is a liberal art. As in: it involves language and explication of processes. The amount of actual engineering involved in making a successful application (like Etsy) is actually quite low. The amount of clear code that focuses relentlessly on the mission at hand (an admittedly squishy metric — after all, that's the nature of art) also seems to be quite low in the typical application, but naturally needs to be much higher.
For instance just putting a platform like Etsy entirely on Google App Engine would allow Etsy to focus on the liberal arts side of the problem. Because GAE "solves" most all the engineering problems that typically have plagued software practices on the web...
I am sympathetic to the distinction. I had a bunch of applicants for a position claiming to be "Distributed Systems Engineers" mostly because they'd stood-up, maintained, and or used a Hadoop or Spark cluster in their current or prior job.
But to me that was more of an advanced (one hopes) end-user. Someone who could take a bunch of large, mostly-complete logical components that somebody else engineered and then use them to stitch together a solution by integrating these existing frameworks that already provide the first 80+% of the technical solution to carry the last ~20% toward a domain-specific use-case.
What I was looking for wasn't somebody who knew how to use something like HDFS. I was looking for someone who could build something as good or better than HDFS from nothing if they had to. A lot of what passes for "engineering" today, at least by marketplace label, tends to resemble the former rather than the latter.
There's definitely all kinds of space for both kinds of builders/creators depending on the needs and the project, but it certainly doesn't help that the English language and it's colloquial application to the problem space has grossly blurred the distinction.
> But to me that was more of an advanced (one hopes) end-user. Someone who could take a bunch of large, mostly-complete logical components that somebody else engineered and then use them to stitch together a solution by integrating these existing frameworks...
I don't find this distinction very useful. We're all end-users at different layers in the stack. Building HDFS from scratch is also mostly taking others' components and ideas and stitching them together. That's what progress and innovation looks like. I think you're looking for engineers at a lower level in the stack than the applicants you received.
Additionally, if you're building the next distributed filesystem, you'll be much more successful if you're also an end-user of existing distributed filesystems, so you know the strengths, weaknesses, user preferences, etc. of the existing products. If you're building something without knowing how it's going to be used...well...you're probably not going to build the right thing.
I've literally seen the opposite interviewing as distributed computing person, most people ask for a distributed engineer and then want someone to stitch together Hadoop and Spark or how to effectively leverage HDFS. When you start talking about concurrency issues, Lamport clocks, consensus algorithms, etc. their eyes glaze over and they ask how to efficiently rotate an array. Clearly something is broken but I'm not sure it is how we use language.
> take a bunch of large, mostly-complete logical components that somebody else engineered and then use them to stitch together a solution by integrating these existing frameworks that already provide the first 80+% of the technical solution to carry the last ~20% toward a domain-specific use-case
This seems like a strange objection, considering that a) getting technology to fit the use case is all virtually everybody wants, and b) having to do 20% of the solution from scratch--rather than, say, the last 0.10%--would be an _enormous_ undertaking. Or don't you consider the silicon, microcode, network, servers, physical protocol, wire protocol, operating system, standards, tools, language, and compiler in that equation? If not then where do you draw the line?
I don't think the skills/abilities you're looking for would be denoted merely by looking for an "engineer" of some kind. In fact, I think looking for an "engineer" likely obfuscates your actual needs since it has become such a catchall term in computing fields.
> I was looking for someone who could build something as good or better than HDFS from nothing if they had to.
Well, for 90 percent of job offers, what a company claims to need is not what it actually needs (i.e. the typical "10 years experience with 5 year old technology" bullshit). If your company is not a very unique snowflake or in an academic setting, believing that it makes zero economic sense to completely reinvent the wheel from first principles is a valid assumption for applicants to make.
> A lot of what passes for "engineering" today, at least by marketplace label, tends to resemble the former rather than the latter.
We can keep reclassifying what it takes to earn that label until we've eliminated all but the geniuses of the software world. The titles (and seniority) are, frankly, useless because they aren't legally enforced because no one has a good and popular way to test for competency. If they did, that would be the technical interview and then market forces could once again weed out people who don't make it.
There really isn't any push back when you have an opinion on what a software engineer ought to be when you're hiring, so naturally, a lot of people have their own opinion. Figuring out which direction to go if you are one of those supposed engineers is pretty much a crap shoot but still better than just not learning anything new.
What I was looking for wasn't somebody who knew how to use something like HDFS. I was looking for someone who could build something as good or better than HDFS from nothing if they had to.
Did you try articulating that distinction in your job ads?
Or if not, can you really blame people for applying when your ads read like 99% of help wanted ads in the fin-de-boom era; like, you know:
We've got the coolest office in the Mission, with a climbing wall and jamming room, we do beer bongs every Thursday and play lawn bowling together on weekends! And of course you can bring your dog in everyday, too! Keywords: Node, Python, D3, Spark, Hadoop, HDFS
Joel Spolsky wrote an essay on "leaky abstractions," and from reading the interview with the Etsy CTO, the two seem related. Allspaw is making the distinction between engineers and developers this way. To him, an engineer is willing and able to get under the hood, if necessary; whereas a developer is limited to using libraries and frameworks as-is, and filing bug reports on them when things don't go as planned, or feature requests if he bumps up against some limitation. I think that's what he means when he characterizes the non-engineer as having a "not my job" mentality.
I've always felt that in the various jobs I've held the degree to which I could "get under the hood" was determined by people higher up the chain of command than me.
Run into a complex bug in library X? Well, we don't want you spending a week to debug it, write a patch and try to upstream it. Work around it for now instead. Or we'll defer that feature since we don't want you to spend that much time on "non-company" projects. Etc.
I think this is usually all the more true when your employer expects you to be a generalist of some kind and I've become a bit jealous of people whose employers let them really dig into things and even deviate to ecosystem projects instead of focusing on keeping the internal hamster wheel spinning.
On another tangent, one thing I found very frustrating early in my career and which I still feel is problematic in our field is that there's kind of a double standard when it comes to approaching things "under the hood."
At my first job I'd always ask my supervisor when when things lead there and then get told not to go there (despite wanting to) while a coworker of similar experience hired at around the same time would constantly get everything done behind schedule because he just did the things under the hood I made the mistake of asking about, but he'd get applauded for it continually. (Perhaps I learned from this, but it was pretty frustrating and disheartening at the time.)
As someone who likes to "get under the hood", I'm used to grumbling about developers these days who just glue together existing frameworks.
But the real cause of my dissatisfaction is that I find it harder to glue together existing frameworks -- I don't know where to begin, until I've dug in a bit to find out how something works underneath. Meanwhile other people whose approaches I think of as simplistic are able to leap in and get things done more quickly than I can.
(Similarly I find it almost impossible to learn much of a foreign language without studying the grammar, while some other people can immerse themselves and become quite conversant without ever thinking about grammar.)
Maybe the "non-engineer" developers are just working at a higher, and more immediately useful, level of interpretation.
Oh man, the Law of Leaky Abstractions is an amazing read, and an incredibly useful general theory. It's so applicable that I've taught my non-technical friends and family as it related to university structures, the insurance industry, management, technical support, etc.
In the paragraph on "Engineering vs. Development" I read what engineering is
"Engineering, as a discipline and as an activity, is multi-disciplinary. It’s just messy. And that’s actually the best part of engineering. It’s not about everyone knowing everything. It’s about paying attention to the shared, mutual understanding. "
But the author doesn't contrast that with what development is. I used the words interchangeably like film vs movie.
Moreover, there seems to be a contradiction there: they are saying engineering is not inherently domain-specific, but to seek out domain experts when the need arises. Okay? That network engineer isn't REALLY an engineer until he wants to learn about databases? Would a "network engineer" be a non-sequitur in itself?
"Software engineering," as the term is usually used, is really a joke.
In my mind, engineering is about rigor: process, measurability and discipline. The hallmark of a well-engineered system, in my opinion, is reliability. Software is anything but reliable.
Too much of software development is throwing things at the wall and hoping that sticks, because there is not a good understanding (or willful neglect) of how the different parts of the stack may adversely affect your application. Add that to ever-expanding requirements scope, poorly designed/maintained/understood code artifacts, and developer churn, and the typical software project is rather frail.
There are considerations you can make for more reliable software: a testing regimen and release planning, conservative resource estimates and knowing your bottlenecks, strategies for degraded operating conditions, fallback and error mitigation, scaling, consistent documentation, and basically knowing the seams of your software, where things might break, under what conditions, and what corrective action could be done.
Most software projects either move too quickly or are simply not important enough to hit these points. There are exceptions, of course (most well-known software we use would qualify), but those are not the rule: most YC companies certainly would not qualify as doing "software engineering." In fact, that almost seems to be the antithesis of a fast-moving startup.
Imagine your civil engineer did not take shear, vibration, bedrock, joint and material strength, etc into account when designing a structure, or allowed a good design to be constructed with shoddy labor, duct-taped together. That is exactly what we see from most software "engineering" today. Move fast and break things, indeed.
I would also like them to go into detail about the differences, and how they see the people they brought up from the hacker schools fit into that model.
> Engineering, as a discipline and as an activity, is multi-disciplinary. It’s just messy.
I would qualify that and say "software engineering," as we understand it today is messy. That's because it's not engineering as a recognized discipline (and seems to be the reason why we can get away with calling ourselves engineers and not be sued).
If your process involves thinking immediately about how to write the solution in code you're doing a very sloppy form of engineering. If you're the kind to write notes down about your design and possibly share some kind of diagram/written specification... you're doing a very weak form of engineering. I don't think it would pass at Lockheed Martin or JPL. You need to be using formal methods and mathematically modelling, checking, and possibly even proving enabled actions on your state and the invariants you hope to maintain. You need to start thinking at a higher level and have liability and all of those other things that drive you to get more guarantees and rigor out of your process.
My theory is that "formal methods" are not out of the reach of hackers, developers, and the wider industry. With a smattering of complicated-sounding things like predicate calculus and temporal logic you can get a psuedo-language for exhaustively checking your designs before you write the code to make sure you haven't forgotten important things like building the right solution.
It's really cool stuff... I'm learning TLA+ right now and loving it. I hope more people will find it as useful as I do.
> That's because it's not engineering as a recognized discipline (and seems to be the reason why we can get away with calling ourselves engineers and not be sued).
I'm consistently surprised at how little programmers/developers and software engineers are aware of the progress made here over the past 5 years. Software engineering most definitely is a recognized engineering discipline nationally and in most states:
We can debate whether or not these developments are good for our profession (good, in my view). However, it's no longer disputable, at least in the US, that software engineering is a recognized engineering discipline. At least unless you don't recognize the authority of the IEEE/NCEES in this matter (in which case you don't recognize the credentials of most US engineers from any engineering discipline).
That said, I strongly agree with your emphasis on formal methods. I'm disappointed that software developers/engineers don't put more emphasis on these tools, even if they fail to convince their managers to actually use them. Yes, we don't need to use formal methods for a few jQuery scripts on a web page, but there are lots of places where they could find good use.
As one example, for general application development, Ada/Spark is an excellent example of an engineering-focused language and environment, and I wish other languages took this approach. For embedded designs, solutions like Ivory (and Ada, too) provide a fairly rigorous approach to software development. TLA+ is another interesting tool (that I know relatively little about). These are the sorts of tools we need to emphasis.
Unfortunately, most clients/companies aren't really interested in these methods, even when they're building applications where security matters. There are obvious exceptions, like some medical and avionics software, but even the automotive software I've looked into seems to have been developed in a sloppy, ad-hoc manner (and surprisingly, often by people trained as traditional engineers).
High-assurance software and security field have been doing real engineering for decades. Here's a nice presentation with a tie-in to another high-assurance discipline:
They warranty their code like Cleanroom teams used to. Still in business so the method works. ;) Their SPARK tool is now GPL with lots of learning resources available. So, those are a few examples of engineering software vs just developing it. The results in terms of predictability, reliability and low defects speak for themselves.
I don't understand why he's drawing this line in the sand.
Engineering is applying scientific principles to solve a problem. Development is the process of improving something.
I don't see why they should be at odds; the concepts seem orthogonal.
Instead of insulting rigorous, creative professionals who prefer the name "developer" over "engineer", maybe he could have said that he wants to hire people that won't go CYA when problems happen.
I recently applied for a position at Etsy, and one of the main points I make in my self summary is that I am most motivated by craftsmanship and an attention to quality (I think a lot of developers feel this way, and more should practice speaking up about it for interviews).
I was told I would have a phone interview with someone from the hiring team. Weeks went by and I got zero response. I figured I had just been rejected with usual company non-response b.s.
But after about three weeks I received an impromptu message from HR saying they were sorry about the delay, and also that unfortunately someone internal had put in a request for the job I would have interviewed for, and they wanted to proceed with their internal candidate.
I was grateful for the message and thanked that HR representative. I think it spoke well of Etsy that they bothered to give me some reply about my status.
But I also thought, as far as engineering goes, that it sounded less than ideal to me that their hiring process could go like that. Why wouldn't internal candidates have been vetted prior to spending time on the application process and phone interviews with others?
They said they thanked me for my understanding and would be in touch about future roles. I have never heard back. So it's puzzling that they seem to want to hire developers with a focus on craftsmanship, but aren't willing to consider me after previously at least feeling that I was good enough for a phone screen.
It's not so much that this is "bad" or "good" for Etsy, but more that it gives an overwhelming feeling of "What on earth is going on inside Etsy? What are they doing?"
It's probably some poorly implemented HR policy that if there's a job opening they still have to publicly post it even if there's someone internally who fits the bill and wants it. I ran into something similar at my first job out of college (which was a shitty place to work in many ways). There was a job opening, they posted it, and even as an existing employee they wouldn't interview me for it because they already had someone in mind internally that was essentially going to be a shoe-in.
Requiring a public posting is a good idea in theory, but in reality it seems most places are just wasting people's time because external candidates are never seriously considered.
Why wouldn't internal candidates have been vetted prior to spending time on the application process and phone interviews with others?
It could be that the internal process started simultaneously, just that the internal candidate "grabbed the thread", and either the HR contact wasn't informed, or didn't think / wasn't coached to notify the external candidates as to the change in status. Until they went back to clean up their inbox.
"What on earth is going on inside Etsy? What are they doing?"
I've come to accept that even when all parties involved have the best of intentions, the workflow of candidate vetting, combined with the inherent game theoretic aspects[1] of finding the best available candidate on the market (when one generally has no idea what market looks like until one has churned through a goodly number of candidates), all come together to make the process more complex (and time consuming) than people generally anticipate.
Unfortunately what gets added to this is that seldom do people stop to think, "Wait, if we don't treat ALL candidates nicely during the process, that means the good ones won't apply again. And they won't recommend their friends to apply to join us, either." And so, "Maybe we should put a little more thought into candidate experience, and make a point of treating them better."
Again, this is under the best of circumstances, and assuming that the parties involve seem to have put some degree of thought into, and to have some sense of discipline behind their vetting and communication process. Sadly, this isn't very often the case.
I interviewed with Etsy for an internship position last year. After the on-site was finished, our group of interns was told we'd hear back within a few days (but likely the very next day.) After two weeks of silence, I emailed them asking if they had an update or a new timeline and was told they'd get back to me in a week or so. This happened two or three more times over the next couple months, until I eventually gave up. Like you, I was more confused than anything. Either way, I can't really say I recommend interviewing with them.
I'm not sure there's a point in trying to create a distinction between titles or trying to make "developer" a pejorative.
At the end of the day companies want to hire people who love what they do, a fiery passion to for continuous self-improvement, extremely competent at their job, and keenly aware of how their role impacts the overall success of the company. Simply changing 'developer' to 'engineer' in your recruiting efforts sure as hell doesn't guarantee those attributes.
>Given the choice between a software engineer who has demonstrated incredible abilities in algorithms, or an engineer or data scientist who has demonstrated both some fluency and, more importantly, curiosity on the financial and legal implications of the work that they’re doing, we will always choose the latter.
Bullshit. Complete and utter bullshit. If you can't do the algorithm dance on the whiteboard you're "not technical enough" - after all, these companies get such a high volume of applicants that there has to be some way to filter them out. Curiosity isn't a measurable metric - it's a "nice-to-have".
Disagree with you about curiosity being a "nice-to-have".
But something else has occurred to recently about the never-ending discussion on "whiteboarding" and other interview techniques:
What's perhaps wrong about the overemphasis on the "algorithm dance" (be it at the whiteboard, over a website, or over the phone) is that people revert to it because it (at least appears to be) measurable, and can be roughly assessed within a reasonably short timeframe (and more or less reproducibly so), and at low cost/risk to the company.
While more nuanced, and arguably much more valuable skills -- like the ability to manage complexity; being generally ego-divorced, and immune to silly hangups about one's platform or style choices, or those of others; seeing the forest for the trees (but the trees also), generally; not being a jerk; and yes, curiosity -- basically can't be measured in an interview setting, or in any way other than through shared experience on actual work projects. At least not reliably, and certainly not reproducibly.
The old "searching for the car keys under the lamp post, because that's where the light is" fallacy, in other words.
More I see cross-engineer that do backend to fronend and more I see problem. There is no way a database guru can create a super nice UI with CSS and JavaScript. Same thing for an expert in JavaScript and web cannot be also an expert in AI, database and xyz stuff. I am all about having people touching a little bit of everything, but at the end, everyone has an expertise and they should do 80% of their time there.
Multi-disciplinary is over preached and create average quality across a system instead of having best practices and efficient development everywhere.
I agree with the sentiment that people should specialize and spend the majority of the time working what they are proficient at.
That said, I personally like working with engineers who at least try to participate in the design/front-end process instead of just passing off to me like they don't give a shit. Their opinions can be informative and it's also a way for me to help teach them a thing or two. Plus sometimes knowing small UI things can help alleviate my workload so that I'm not doing really tiny tasks associated with a project that's primarily driven by a backend engineer.
As a longtime designer/UI guy, I have no illusions of ever being an elite backend programmer. But I actually enjoy learning backend on the side. I feel like it's a way of me having a better understanding how the teams entire codebase works and allows me to have a slightly better understanding of what my team members discuss during meetings. And even for side projects, it's just empowering to be able to implement a product idea from end to end (even though yes, it's likely not stellar code). Overall, just having a growing understanding of how it all connects together is truly rewarding.
Being very-multi-disciplinary makes sense for development managers (who manage a team of specialized people), not for individual contributors, at least not in the common case.
I think Eric's article was able to articulate differences between a programmer and a developer. Reading this interview, I don't think Etsy's CTO is able to clarify what makes an engineer different from a developer.
An engineer engineers solutions, frequently from scratch or very small pieces.
A developer develops solutions, often from larger pieces, focusing on implementing them to solve business needs.
It's subtle but I think there is some differentiation. I think the vast majority of companies only require developers, or mostly developers, including Etsy.. because let's be honest what is Etsy doing that hasn't been done before?
Feels like we're chasing our tails on this one. We went from "programmers" to "developers" to "engineers". Every time it started well, because only people with the right mindset referred to themselves and the people they looked for that way, but pretty soon the rest catches up and the term devalues.
We should just call ourselves codemonkeys and focus on doing the job well regardless.
I still default to calling myself a programmer or a coder... I'm happy to ride the social status train of "Software Engineer", but I don't pretend it's anything else, and it could swiftly be taken away if some Congress Critters got it into their heads that because in general none of us are licensed PEs we don't deserve to call ourselves engineers. I sometimes think it might be good for that to happen so that some of us have less-inflated views of ourselves and our profession, but judging from other countries that have such restrictions, we'd probably get a significant salary cut to go along with it while the business people with MBAs reap even larger shares of whatever company's profits which are only made possible by code monkeys...
And are you willing to pay your "software engineers" what they are worth? because that's often the issue at end. Plenty of good "engineers". Not that many startup willing to pay them fairly. You want the best people but you don't want to pay them the "best" money? you get what you pay for.
At least for me, developer is solely focused on shipping code, although it's unavoidable to meet problems that require engineering. When facing such problem, the developer will focus on 'making it work', instead of focusing on answering 'how should I proper architecture it?'.
The curiosity of answering the later question comes from an engineer that also wants to know about networking, architecture and better ways to develop systems.
In simple terms this distinction answers if the professional is curious about development only (developer) or curious about the whole stack plus development (software engineer).
One of the best pieces of sleight of hand that modern companies are pulling on modern engineering job seekers is the complete demise of the job description.
Job descriptions act like a constraint on the human complexity of a team or organization. They help codify the way people conceive of modularizing known workflows and discretizing for the purposes of having employees solve them.
Crucially, a job description defines the boundaries of what an employee can say no to -- "that's not in my job description."
In the modern hiring landscape, this would be immediately met with vapid criticisms, like you are not a team player or you won't "wear many hats" or whatever.
But the benefit of job description boundaries is not related to entitling programmers to be whiny and complacent about tasks they don't like. That's naive. The reason it's good to empower your workers to tell you no, especially knowledge workers, is because it preserves organization structure and planning. It highlights bottlenecks for you and points out the discrepancy between what you need and what you have. Otherwise, while your machine learning expert is repressing her fury over being assigned to clean up some legacy Rails codebase, you might be off playing golf thinking you've got your Rails maintenance needs correctly covered, when really you don't at all (and you're leaving money on the table by not extract the full value from your machine learning engineer that you otherwise could).
This, by the way, is perhaps the biggest hallmark of a good manager. Good managers will act like double-sided adapters who process arbitrary inputs from the business-problem-stream on one side, and turn them into collaborations with subordinates that fully respect those subordinates' specialities, aptitudes, and goals. That is a damn hard job, and when done properly it is big justification for the increased compensation and status usually given to management-level employees. Unfortunately, many modern firms do not expect managers to do this, and instead they seek to build teams of so-called "full-stack" developers, for everything, so that managers are only receiving business problem inputs and never doing the actually difficult part of turning them into workflows that show respect for subordinates (... you know, managing them).
In fact, I would argue that this is exactly what it means to even have a business model at all. If your "business model" involves hiring people and then screaming at them to do whatever they are told, instead of what they are good at or motivated to do, because "that's just how jobs work" then in effect you actually don't have a business model. You have not yet done the hard part -- figuring out that double-sided adapter layer that translates real world business concerns into things that disparate specialists want to do and won't refuse to do.
When I read this by the Etsy CTO, it strikes me as a big, whiny excuse to try to hire more people who are "arbitrary work receptacles." That seems to be what he is trying to describe as "engineering" -- He uses words like multi-disciplinary and messy, when really he's trying to convey "I want you to do whatever arbitrary work I say, regardless of how poorly it matches your aptitudes, interests, or goals."
I feel developers should more strongly demand legitimate job descriptions, and ask during the hiring process for specifics on the ways that managers will be held accountable for protecting the job description, instead of forcing employees to bottomlessly compromise on their goals in order to be arbitrary work receptacles.
It's not about mere displeasure over having to do arbitrary tasks. It's about the fact that a lack of planning or modularity codified through job descriptions is bad for business. Properly respecting employee aptitudes is a sign of a healthy business. Demanding arbitrary cross-functionality all the time is a sign of needless chaos. We can see this clearly in software itself: if all of your classes make everything public, silently mutate each other's state, and everything grows purely by attrition with no attention paid to e.g. Single Responsibility Principle, it's a huge red flag of bad code and impending system failure. But we bury our heads in the sand when the same ideas become apparent when managing people complexity instead of software complexity.
[+] [-] agarden|10 years ago|reply
Their method for avoiding the proliferation of different tools is interesting.
His discussion of the kind of people they want, once you ignore the semantics, is interesting. They want people who take responsibility and are always pushing the boundary of their knowledge into new fields. That's interesting.
My favorite part of the article was right at the end: "instead of asking questions about “why did something fail,” we want to ask why something succeeded, which is really easy to skip over."
[+] [-] linkregister|10 years ago|reply
What does irk me is that in real life, software developers who meet the Etsy CTO's standard often can't call themselves engineers. In many countries, misrepresenting oneself as an engineer without accreditation is against the law! [1]
In the U.S., there's no official occupational category for Software Engineer. It's just Software Developer (Applications), Software Developer (Systems), and Computer Programmer.
It is possible to get accreditation (http://ncees.org/exams/pe-exam/, under Software), but it didn't appear very useful to me aside from being able to call yourself an Engineer.
[1] http://www.canadianconsultingengineer.com/engineering/quebec...
[+] [-] ta0967|10 years ago|reply
semantics == meaning. once you ignore what he's trying to say, the discussion ends, no?
edit: yes, i could care less about semantics of semantics, but i can't bring myself to it.
[+] [-] gech|10 years ago|reply
[+] [-] nickbauman|10 years ago|reply
For instance just putting a platform like Etsy entirely on Google App Engine would allow Etsy to focus on the liberal arts side of the problem. Because GAE "solves" most all the engineering problems that typically have plagued software practices on the web...
[+] [-] Kinnard|10 years ago|reply
[+] [-] im_down_w_otp|10 years ago|reply
But to me that was more of an advanced (one hopes) end-user. Someone who could take a bunch of large, mostly-complete logical components that somebody else engineered and then use them to stitch together a solution by integrating these existing frameworks that already provide the first 80+% of the technical solution to carry the last ~20% toward a domain-specific use-case.
What I was looking for wasn't somebody who knew how to use something like HDFS. I was looking for someone who could build something as good or better than HDFS from nothing if they had to. A lot of what passes for "engineering" today, at least by marketplace label, tends to resemble the former rather than the latter.
There's definitely all kinds of space for both kinds of builders/creators depending on the needs and the project, but it certainly doesn't help that the English language and it's colloquial application to the problem space has grossly blurred the distinction.
[+] [-] cle|10 years ago|reply
I don't find this distinction very useful. We're all end-users at different layers in the stack. Building HDFS from scratch is also mostly taking others' components and ideas and stitching them together. That's what progress and innovation looks like. I think you're looking for engineers at a lower level in the stack than the applicants you received.
Additionally, if you're building the next distributed filesystem, you'll be much more successful if you're also an end-user of existing distributed filesystems, so you know the strengths, weaknesses, user preferences, etc. of the existing products. If you're building something without knowing how it's going to be used...well...you're probably not going to build the right thing.
[+] [-] bitcrusher|10 years ago|reply
[+] [-] sk5t|10 years ago|reply
This seems like a strange objection, considering that a) getting technology to fit the use case is all virtually everybody wants, and b) having to do 20% of the solution from scratch--rather than, say, the last 0.10%--would be an _enormous_ undertaking. Or don't you consider the silicon, microcode, network, servers, physical protocol, wire protocol, operating system, standards, tools, language, and compiler in that equation? If not then where do you draw the line?
[+] [-] Glyptodon|10 years ago|reply
[+] [-] _pmf_|10 years ago|reply
Well, for 90 percent of job offers, what a company claims to need is not what it actually needs (i.e. the typical "10 years experience with 5 year old technology" bullshit). If your company is not a very unique snowflake or in an academic setting, believing that it makes zero economic sense to completely reinvent the wheel from first principles is a valid assumption for applicants to make.
[+] [-] FLUX-YOU|10 years ago|reply
We can keep reclassifying what it takes to earn that label until we've eliminated all but the geniuses of the software world. The titles (and seniority) are, frankly, useless because they aren't legally enforced because no one has a good and popular way to test for competency. If they did, that would be the technical interview and then market forces could once again weed out people who don't make it.
There really isn't any push back when you have an opinion on what a software engineer ought to be when you're hiring, so naturally, a lot of people have their own opinion. Figuring out which direction to go if you are one of those supposed engineers is pretty much a crap shoot but still better than just not learning anything new.
[+] [-] kafkaesq|10 years ago|reply
Did you try articulating that distinction in your job ads?
Or if not, can you really blame people for applying when your ads read like 99% of help wanted ads in the fin-de-boom era; like, you know:
We've got the coolest office in the Mission, with a climbing wall and jamming room, we do beer bongs every Thursday and play lawn bowling together on weekends! And of course you can bring your dog in everyday, too! Keywords: Node, Python, D3, Spark, Hadoop, HDFS
[+] [-] facepalm|10 years ago|reply
[+] [-] mariodiana|10 years ago|reply
[+] [-] Glyptodon|10 years ago|reply
Run into a complex bug in library X? Well, we don't want you spending a week to debug it, write a patch and try to upstream it. Work around it for now instead. Or we'll defer that feature since we don't want you to spend that much time on "non-company" projects. Etc.
I think this is usually all the more true when your employer expects you to be a generalist of some kind and I've become a bit jealous of people whose employers let them really dig into things and even deviate to ecosystem projects instead of focusing on keeping the internal hamster wheel spinning.
On another tangent, one thing I found very frustrating early in my career and which I still feel is problematic in our field is that there's kind of a double standard when it comes to approaching things "under the hood."
At my first job I'd always ask my supervisor when when things lead there and then get told not to go there (despite wanting to) while a coworker of similar experience hired at around the same time would constantly get everything done behind schedule because he just did the things under the hood I made the mistake of asking about, but he'd get applauded for it continually. (Perhaps I learned from this, but it was pretty frustrating and disheartening at the time.)
[+] [-] cannam|10 years ago|reply
But the real cause of my dissatisfaction is that I find it harder to glue together existing frameworks -- I don't know where to begin, until I've dug in a bit to find out how something works underneath. Meanwhile other people whose approaches I think of as simplistic are able to leap in and get things done more quickly than I can.
(Similarly I find it almost impossible to learn much of a foreign language without studying the grammar, while some other people can immerse themselves and become quite conversant without ever thinking about grammar.)
Maybe the "non-engineer" developers are just working at a higher, and more immediately useful, level of interpretation.
[+] [-] rqebmm|10 years ago|reply
http://www.joelonsoftware.com/articles/LeakyAbstractions.htm...
[+] [-] nogbit|10 years ago|reply
"The Law of Leaky Abstractions is dragging us down." - Joel Spolsky (2002)
[+] [-] mtmail|10 years ago|reply
"Engineering, as a discipline and as an activity, is multi-disciplinary. It’s just messy. And that’s actually the best part of engineering. It’s not about everyone knowing everything. It’s about paying attention to the shared, mutual understanding. "
But the author doesn't contrast that with what development is. I used the words interchangeably like film vs movie.
[+] [-] jallmann|10 years ago|reply
"Software engineering," as the term is usually used, is really a joke.
In my mind, engineering is about rigor: process, measurability and discipline. The hallmark of a well-engineered system, in my opinion, is reliability. Software is anything but reliable.
Too much of software development is throwing things at the wall and hoping that sticks, because there is not a good understanding (or willful neglect) of how the different parts of the stack may adversely affect your application. Add that to ever-expanding requirements scope, poorly designed/maintained/understood code artifacts, and developer churn, and the typical software project is rather frail.
There are considerations you can make for more reliable software: a testing regimen and release planning, conservative resource estimates and knowing your bottlenecks, strategies for degraded operating conditions, fallback and error mitigation, scaling, consistent documentation, and basically knowing the seams of your software, where things might break, under what conditions, and what corrective action could be done.
Most software projects either move too quickly or are simply not important enough to hit these points. There are exceptions, of course (most well-known software we use would qualify), but those are not the rule: most YC companies certainly would not qualify as doing "software engineering." In fact, that almost seems to be the antithesis of a fast-moving startup.
Imagine your civil engineer did not take shear, vibration, bedrock, joint and material strength, etc into account when designing a structure, or allowed a good design to be constructed with shoddy labor, duct-taped together. That is exactly what we see from most software "engineering" today. Move fast and break things, indeed.
[+] [-] unknown|10 years ago|reply
[deleted]
[+] [-] Arzh|10 years ago|reply
[+] [-] agentultra|10 years ago|reply
I would qualify that and say "software engineering," as we understand it today is messy. That's because it's not engineering as a recognized discipline (and seems to be the reason why we can get away with calling ourselves engineers and not be sued).
If your process involves thinking immediately about how to write the solution in code you're doing a very sloppy form of engineering. If you're the kind to write notes down about your design and possibly share some kind of diagram/written specification... you're doing a very weak form of engineering. I don't think it would pass at Lockheed Martin or JPL. You need to be using formal methods and mathematically modelling, checking, and possibly even proving enabled actions on your state and the invariants you hope to maintain. You need to start thinking at a higher level and have liability and all of those other things that drive you to get more guarantees and rigor out of your process.
My theory is that "formal methods" are not out of the reach of hackers, developers, and the wider industry. With a smattering of complicated-sounding things like predicate calculus and temporal logic you can get a psuedo-language for exhaustively checking your designs before you write the code to make sure you haven't forgotten important things like building the right solution.
It's really cool stuff... I'm learning TLA+ right now and loving it. I hope more people will find it as useful as I do.
[+] [-] jnbiche|10 years ago|reply
I'm consistently surprised at how little programmers/developers and software engineers are aware of the progress made here over the past 5 years. Software engineering most definitely is a recognized engineering discipline nationally and in most states:
http://ncees.org/about-ncees/news/ncees-introduces-pe-exam-f...
http://insight.ieeeusa.org/insight/content/careers/97473
We can debate whether or not these developments are good for our profession (good, in my view). However, it's no longer disputable, at least in the US, that software engineering is a recognized engineering discipline. At least unless you don't recognize the authority of the IEEE/NCEES in this matter (in which case you don't recognize the credentials of most US engineers from any engineering discipline).
That said, I strongly agree with your emphasis on formal methods. I'm disappointed that software developers/engineers don't put more emphasis on these tools, even if they fail to convince their managers to actually use them. Yes, we don't need to use formal methods for a few jQuery scripts on a web page, but there are lots of places where they could find good use.
As one example, for general application development, Ada/Spark is an excellent example of an engineering-focused language and environment, and I wish other languages took this approach. For embedded designs, solutions like Ivory (and Ada, too) provide a fairly rigorous approach to software development. TLA+ is another interesting tool (that I know relatively little about). These are the sorts of tools we need to emphasis.
Unfortunately, most clients/companies aren't really interested in these methods, even when they're building applications where security matters. There are obvious exceptions, like some medical and avionics software, but even the automotive software I've looked into seems to have been developed in a sloppy, ad-hoc manner (and surprisingly, often by people trained as traditional engineers).
[+] [-] nickpsecurity|10 years ago|reply
http://web.cecs.pdx.edu/~hook/cs491sp08/AssuranceSp08.ppt
Here's an old method, Cleanroom, that was cost-effective for business with low, defect rate even on first use:
http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=E10....
Altran[-Praxis] is a modern company engineering solutions with their Correct by Construction methodology:
http://www.sis.pitt.edu/jjoshi/Devsec/CorrectnessByConstruct....
They warranty their code like Cleanroom teams used to. Still in business so the method works. ;) Their SPARK tool is now GPL with lots of learning resources available. So, those are a few examples of engineering software vs just developing it. The results in terms of predictability, reliability and low defects speak for themselves.
[+] [-] mattlutze|10 years ago|reply
Engineering is applying scientific principles to solve a problem. Development is the process of improving something.
I don't see why they should be at odds; the concepts seem orthogonal.
Instead of insulting rigorous, creative professionals who prefer the name "developer" over "engineer", maybe he could have said that he wants to hire people that won't go CYA when problems happen.
[+] [-] p4wnc6|10 years ago|reply
I was told I would have a phone interview with someone from the hiring team. Weeks went by and I got zero response. I figured I had just been rejected with usual company non-response b.s.
But after about three weeks I received an impromptu message from HR saying they were sorry about the delay, and also that unfortunately someone internal had put in a request for the job I would have interviewed for, and they wanted to proceed with their internal candidate.
I was grateful for the message and thanked that HR representative. I think it spoke well of Etsy that they bothered to give me some reply about my status.
But I also thought, as far as engineering goes, that it sounded less than ideal to me that their hiring process could go like that. Why wouldn't internal candidates have been vetted prior to spending time on the application process and phone interviews with others?
They said they thanked me for my understanding and would be in touch about future roles. I have never heard back. So it's puzzling that they seem to want to hire developers with a focus on craftsmanship, but aren't willing to consider me after previously at least feeling that I was good enough for a phone screen.
It's not so much that this is "bad" or "good" for Etsy, but more that it gives an overwhelming feeling of "What on earth is going on inside Etsy? What are they doing?"
[+] [-] beeboop|10 years ago|reply
Requiring a public posting is a good idea in theory, but in reality it seems most places are just wasting people's time because external candidates are never seriously considered.
[+] [-] kafkaesq|10 years ago|reply
It could be that the internal process started simultaneously, just that the internal candidate "grabbed the thread", and either the HR contact wasn't informed, or didn't think / wasn't coached to notify the external candidates as to the change in status. Until they went back to clean up their inbox.
"What on earth is going on inside Etsy? What are they doing?"
I've come to accept that even when all parties involved have the best of intentions, the workflow of candidate vetting, combined with the inherent game theoretic aspects[1] of finding the best available candidate on the market (when one generally has no idea what market looks like until one has churned through a goodly number of candidates), all come together to make the process more complex (and time consuming) than people generally anticipate.
Unfortunately what gets added to this is that seldom do people stop to think, "Wait, if we don't treat ALL candidates nicely during the process, that means the good ones won't apply again. And they won't recommend their friends to apply to join us, either." And so, "Maybe we should put a little more thought into candidate experience, and make a point of treating them better."
Again, this is under the best of circumstances, and assuming that the parties involve seem to have put some degree of thought into, and to have some sense of discipline behind their vetting and communication process. Sadly, this isn't very often the case.
[1] https://en.wikipedia.org/wiki/Stable_marriage_problem
[+] [-] ExtremePopcorn|10 years ago|reply
[+] [-] meritt|10 years ago|reply
At the end of the day companies want to hire people who love what they do, a fiery passion to for continuous self-improvement, extremely competent at their job, and keenly aware of how their role impacts the overall success of the company. Simply changing 'developer' to 'engineer' in your recruiting efforts sure as hell doesn't guarantee those attributes.
[+] [-] vdnkh|10 years ago|reply
Bullshit. Complete and utter bullshit. If you can't do the algorithm dance on the whiteboard you're "not technical enough" - after all, these companies get such a high volume of applicants that there has to be some way to filter them out. Curiosity isn't a measurable metric - it's a "nice-to-have".
[+] [-] kafkaesq|10 years ago|reply
But something else has occurred to recently about the never-ending discussion on "whiteboarding" and other interview techniques:
What's perhaps wrong about the overemphasis on the "algorithm dance" (be it at the whiteboard, over a website, or over the phone) is that people revert to it because it (at least appears to be) measurable, and can be roughly assessed within a reasonably short timeframe (and more or less reproducibly so), and at low cost/risk to the company.
While more nuanced, and arguably much more valuable skills -- like the ability to manage complexity; being generally ego-divorced, and immune to silly hangups about one's platform or style choices, or those of others; seeing the forest for the trees (but the trees also), generally; not being a jerk; and yes, curiosity -- basically can't be measured in an interview setting, or in any way other than through shared experience on actual work projects. At least not reliably, and certainly not reproducibly.
The old "searching for the car keys under the lamp post, because that's where the light is" fallacy, in other words.
[+] [-] daok|10 years ago|reply
Multi-disciplinary is over preached and create average quality across a system instead of having best practices and efficient development everywhere.
[+] [-] toomuchtodo|10 years ago|reply
* Must do backend
* Must do frontend
* Must do devops
* Must do database schemas like a wizard
* Must know all of AWS inside and out
* Do you like customer support? You'll be doing customer support.
* You'll be doing marketing disguised as engineering blog posts.
Completely unrealistic.
[+] [-] colmvp|10 years ago|reply
That said, I personally like working with engineers who at least try to participate in the design/front-end process instead of just passing off to me like they don't give a shit. Their opinions can be informative and it's also a way for me to help teach them a thing or two. Plus sometimes knowing small UI things can help alleviate my workload so that I'm not doing really tiny tasks associated with a project that's primarily driven by a backend engineer.
As a longtime designer/UI guy, I have no illusions of ever being an elite backend programmer. But I actually enjoy learning backend on the side. I feel like it's a way of me having a better understanding how the teams entire codebase works and allows me to have a slightly better understanding of what my team members discuss during meetings. And even for side projects, it's just empowering to be able to implement a product idea from end to end (even though yes, it's likely not stellar code). Overall, just having a growing understanding of how it all connects together is truly rewarding.
[+] [-] johansch|10 years ago|reply
[+] [-] chrisbennet|10 years ago|reply
-Robert A. Heinlein
[+] [-] solutionyogi|10 years ago|reply
http://ericsink.com/No_Programmers.html
I think Eric's article was able to articulate differences between a programmer and a developer. Reading this interview, I don't think Etsy's CTO is able to clarify what makes an engineer different from a developer.
[+] [-] jonesb6|10 years ago|reply
An engineer engineers solutions, frequently from scratch or very small pieces.
A developer develops solutions, often from larger pieces, focusing on implementing them to solve business needs.
It's subtle but I think there is some differentiation. I think the vast majority of companies only require developers, or mostly developers, including Etsy.. because let's be honest what is Etsy doing that hasn't been done before?
[+] [-] makeitsuckless|10 years ago|reply
We should just call ourselves codemonkeys and focus on doing the job well regardless.
[+] [-] whitegrape|10 years ago|reply
[+] [-] iamleppert|10 years ago|reply
Not every problem needs to be engineered. Sometimes you just need a good enough solution developed, and an engineered one would be overkill.
[+] [-] LawrenceHecht|10 years ago|reply
Based on our analysis of LinkedIn, there are 1.7 developers for each SW engineer. Raw data here: bit.ly/1QOobbN
[+] [-] aikah|10 years ago|reply
[+] [-] ninjakeyboard|10 years ago|reply
[+] [-] andersonmvd|10 years ago|reply
At least for me, developer is solely focused on shipping code, although it's unavoidable to meet problems that require engineering. When facing such problem, the developer will focus on 'making it work', instead of focusing on answering 'how should I proper architecture it?'.
The curiosity of answering the later question comes from an engineer that also wants to know about networking, architecture and better ways to develop systems.
In simple terms this distinction answers if the professional is curious about development only (developer) or curious about the whole stack plus development (software engineer).
[+] [-] p4wnc6|10 years ago|reply
Job descriptions act like a constraint on the human complexity of a team or organization. They help codify the way people conceive of modularizing known workflows and discretizing for the purposes of having employees solve them.
Crucially, a job description defines the boundaries of what an employee can say no to -- "that's not in my job description."
In the modern hiring landscape, this would be immediately met with vapid criticisms, like you are not a team player or you won't "wear many hats" or whatever.
But the benefit of job description boundaries is not related to entitling programmers to be whiny and complacent about tasks they don't like. That's naive. The reason it's good to empower your workers to tell you no, especially knowledge workers, is because it preserves organization structure and planning. It highlights bottlenecks for you and points out the discrepancy between what you need and what you have. Otherwise, while your machine learning expert is repressing her fury over being assigned to clean up some legacy Rails codebase, you might be off playing golf thinking you've got your Rails maintenance needs correctly covered, when really you don't at all (and you're leaving money on the table by not extract the full value from your machine learning engineer that you otherwise could).
This, by the way, is perhaps the biggest hallmark of a good manager. Good managers will act like double-sided adapters who process arbitrary inputs from the business-problem-stream on one side, and turn them into collaborations with subordinates that fully respect those subordinates' specialities, aptitudes, and goals. That is a damn hard job, and when done properly it is big justification for the increased compensation and status usually given to management-level employees. Unfortunately, many modern firms do not expect managers to do this, and instead they seek to build teams of so-called "full-stack" developers, for everything, so that managers are only receiving business problem inputs and never doing the actually difficult part of turning them into workflows that show respect for subordinates (... you know, managing them).
In fact, I would argue that this is exactly what it means to even have a business model at all. If your "business model" involves hiring people and then screaming at them to do whatever they are told, instead of what they are good at or motivated to do, because "that's just how jobs work" then in effect you actually don't have a business model. You have not yet done the hard part -- figuring out that double-sided adapter layer that translates real world business concerns into things that disparate specialists want to do and won't refuse to do.
When I read this by the Etsy CTO, it strikes me as a big, whiny excuse to try to hire more people who are "arbitrary work receptacles." That seems to be what he is trying to describe as "engineering" -- He uses words like multi-disciplinary and messy, when really he's trying to convey "I want you to do whatever arbitrary work I say, regardless of how poorly it matches your aptitudes, interests, or goals."
I feel developers should more strongly demand legitimate job descriptions, and ask during the hiring process for specifics on the ways that managers will be held accountable for protecting the job description, instead of forcing employees to bottomlessly compromise on their goals in order to be arbitrary work receptacles.
It's not about mere displeasure over having to do arbitrary tasks. It's about the fact that a lack of planning or modularity codified through job descriptions is bad for business. Properly respecting employee aptitudes is a sign of a healthy business. Demanding arbitrary cross-functionality all the time is a sign of needless chaos. We can see this clearly in software itself: if all of your classes make everything public, silently mutate each other's state, and everything grows purely by attrition with no attention paid to e.g. Single Responsibility Principle, it's a huge red flag of bad code and impending system failure. But we bury our heads in the sand when the same ideas become apparent when managing people complexity instead of software complexity.