> I’ve found it a real struggle to get our team to adopt writing tests.
If you're struggling to judge the engineering culture of a company that you're considering joining, consider this indicative of a poor one. It isn't definitive, but it's something you should ask about and probe further. Ask to see their CI dashboard and PR comments over the last few days. When they talk about Agile, ask what _engineering_ techniques (not process!) they leverage. These things will tell you if you're joining a GM or a Toyota; a company that sees quality and efficiency as opposing forces, or one that sees them as inseparable.
When it comes to tests, there are two types of people: those who know how to write tests, and those who think they're inefficient. If I had to guess what happened here, I'd say: the company had a lack of people who knew how to write effective tests combined with a lack of mentoring.
That's why you ask to see recent PR comments and find out if they do pair programming. Because these two things are decisive factors in a good engineering culture.
PR comments I agree with, but after believing in unit tests for years I'm drifting slowly into the "waste of time" camp.
I'm convinced that unit tests don't usually find bugs. IMO, most bugs are edge cases that were an oversight in the design. If the dev didn't handle the case in code they're not going to know to test for it. Fuzzing is a much better approach.
At my current position I have the opportunity to work with two large code-bases, built by different teams in different offices. One of the projects has ~70% code coverage, the other doesn't have a single test. Exposure to both of these systems really bent my opinion on unit tests and it has not recovered.
The project with high code coverage is basically shit and has so many bugs that we regularly cull anything marked less than "medium" severity as "not worth fixing". This project was written by a team that loves "patterns", so you can find all sorts of gems like Aspect Based Programming, CQRS, N-tier, etc. well mixed into a featureless grey Java EE goo. We get so many bug reports that its someones job to go through them.
The other project with no tests is a dream to work on. Not a single file over a few hundreds lines, everything linted and well documented. Almost no methods that don't fit on the screen, no recursion. No bullshit "layering" or "patterns". I can't remember the last time we had a bug report, since our monitoring picks up any exception client and server side. Every bug I've worked on was identified by our monitoring and fixed before anyone noticed.
Whats the difference between teams that developed these vastly different applications?? I've worked with both teams for a while, and honestly, the engineers that wrote no tests are of far higher caliber. Use Linux at home, programming since they can remember, hacking assembler on the weekends and 3D printing random useless objects they could easily buy. The other team went to school to program, and they do it because it pays the bills. Most of the bad programmers know what they're doing is wrong, but they do it anyways so they can pad their resume with more crap and tell the boss how great they are for adding machine learning to the help screen that nobody has ever opened.
If your developers are great then tests would hardly fail and be fairly useless, and if they're terrible tests don't save you. Maybe there's some middle ground if you have a mixed team or a bunch of mediocre devs??
This stems from an unwillingness to make it a job requirement.
There are several things you as a software engineer are expected to do as a part of your job: write code, write tests, participate in code reviews, ensure successful deployment, work effectively with various groups, etc.
It's really simple: state the job requirements up front in the position description and during the hiring process. Make testing part of the code review process, and use it as an opportunity to educate about what makes a good test. Make it part of the performance review and tie raises to it (and, if it goes on long enough, continued employment).
Need to write tests for existing untested areas of the code? Have the team create a developer rotation so they dedicate someone to it for part of each sprint.
To be fair, testing is to some extent an unsolved problem. The joys of testing were being extolled long before test frameworks were actually usable. Now that they are, and you can glue Travis, GitHub and the test lib of your choice together pretty easily you have solved about 30% of what needs to be tested. If, say, you are developing an Office add-in on a Mac, and you want to test it on Word 2013 on Windows 7, there is no easy way to automate this task, and certainly no "write once, run everywhere" solution.
In my GitHuby life, I write tests obsessively. In my enterprisey-softwarey life I don't, because there is no sensible way to do it.
Tests or unit tests? When people refer to writing tests, they usually mean unit tests. In the last four years, our unit tests have caught maybe one or two bugs. The time and resources spent on fixing those bugs after they were in production would have been a fraction of the time and resources spent on writing unit tests. Unit tests simply don't catch bugs and spending time on them is time wasted. Judging a company as having a bad engineering culture because they don't do pointless, unnecessary, and superfluous work that doesn't benefit them seems to be more a reflection of your ideas than the company itself. I'd say that reflects quite well on the company and its engineering culture. If you're talking about other automated tests, they may or may not make sense depending on your team size and product.
> I’ve found it a real struggle to get our team to adopt writing tests.
I find this hard to believe. Do others CTOs / team leads find this to be the case?
I've been a CTO of two small startups with 3-7 developers. We've had resistance to tests at some points (myself included). We've solved it fairly simply. All pull requests (PRs) require tests. PRs are rejected immediately without tests. If a PR doesn't have tests and it is critical to get in, we open a new ticket to track adding tests. It isn't fool proof, but it does result in a high degree of test coverage.
And once developers understand how and where to write tests, they usually see the benefit quickly and want to write more tests.
I'm not a CTO but I do lead the dev team at our agency (was previously 16 devs, but we've slimmed down to 7 currently). I want to preface this by saying that at an agency, your biggest enemy is always time; sales teams have to sell projects for the absolute minimum in order to get a contract, so you can't waste time on non-essentials for most projects.
That said, the biggest resistance I have found is "this feature is due in three days, I need two and a half to finish, and then we have another half to review and find bugs." In the end, the biggest issue is that we have time to test on the spot or write tests, but not both. You can scrape by with just manual testing, but I don't think anyone would ever rely on automated tests 100%.
Our larger projects are test-backed, and our largest even reaches 90% coverage, but the only reasons we wrote tests for those was because we knew we would be working on them for 2-3 years and it was worth the time in that case. I wish this wasn't the case, but I've found it's always the argument against automated tests in my corner of the market
It's much easier to convince people that tests are necessary if you're starting a new project. Problems arise when you're working on a legacy codebase that never had tests to begin with. Often, the code isn't testable at all.
IME, there are far too many "senior" devs (who absolutely should know better) who never worked on any testing-heavy teams that just don't see the point. After all, there's QA, and it's not like THIS code should break THAT code in a seemingly-unrelated part of the codebase...
I'm CTO at a bootstrapped agency that has now 30+ ppl, and I couldn't agree more with the author.
Sure thing that you can use your authority to force ppl, but should you?
Smart people are hard to come by but once you have them you should let them work, and when you tell them how to do their job you implicitly assuming that you know better. Besides if you force them you achieve nothing but some brain-dead tests that are going to hunt you later and getting a budget to "rewrite tests" is a fairytale.
The art here is to build a culture that embrace the test as a powerful tool. So newcomers are quickly seeing benefits and start to write the tests in right places, not for the sake of an artificial metric.
Besides, there are plenty of places where having high coverage is going to be a waste of time:
- throwaway prototypes,
- heavy UI code full of animations - they need to look right that is hard to test,
- infrastructure code if you have just a few servers of a particular type,
- customer projects with unreasonable deadlines that are not going to be touched again,
So having your team that writes tests is a hard job and using PR policy won't help much.
The things that worked for me were:
- write tests that make sense your self in early stages of the project
- pair with your employees and write tests with them,
- do peer reviews and suggest what could be tested and why it make sense.
Getting a team to write tests is change management 101:
People are resistant to change when they don't know how it benefits them directly and immediately.
My suggestions have been:
- By giving developers slight nudges every time they get frustrated with developing when tests aren't present is a good way to help them see the benefit. "Imagine how much easier it would be to write this piece of code if you had tests in places where this function calls other things".
- Enforcing it during commits (as you suggest, using PRs)
- Reminding your whole organization that while you migrate to implementing more testing that velocity of development will be impacted. This is really important, because it means people outside of the dev team also need to see the benefit.
- Eliminating "deadline cultures" first and then implementing unit testing
At my old telecom job, we had weekly MOP (method of procedure) meetings, basically the gate process from lab to production.
One of the MOP checkoff boxes, test results.
So many times you could tank someone by asking "Where are the test results" and they would have to reschedule their maintenance window. If you pissed some ops engineer off, expect the question "Where are the test results" every MOP meeting.
Team Lead/Manager here. I frequently have the opposite problem: people who think code coverage (quantity) is as important as or worse more important than test quality. I'd much rather have low percentage high quality coverage than high-percentage low quality coverage.
How many tests does a PR need? One? Five? When would you not write tests for something(that something not necessarily being a PR, but maybe a unit or feature)?
Tests, like any process, should be serving you and your goals.. You shouldn't be serving your processes or testing practices. This sort of un-nuanced thinking isn't indicative of a high performing startup or CTO IMHO. Perhaps your policies are not directly indicative of your real thoughts on the matter?
As an engineer, I would now be very wary of joining any company unless "Do they write tests for their code?" as part of the hiring criteria. If you want to have something be part of your culture, it needs to be part of the judgement exercised by humans in the hiring/performance evaluation process. I say "by humans" because you do need someone exercising actual judgement rather than checking a box.
What you outline seems reasonable, at least in an environment where you sometimes have hard deadlines (eg. Ticket sales for this festival go live next week). Outside of that, I'm curious what cases there are where you can have a PR which is both critical to merge and doesn't need tests. When I review a PR, I look at the tests as one way of thinking through "what edge cases have already been accounted for here?"
I've never found developers resisted reasonable unit tests, though teams may squabble about what _real_ unit tests are, or may have been burned by poor approaches to unit testing in the past. If you can find the cause of resistance, all teams I've worked with have been happy (even excited) to get better testing in place. It makes them more productive and more successful when done properly.
What I find more common is for the business to be unprepared to make lateral changes to a product. Even rational unit tests are a medium term investment. You need to spend time developing features customers don't see, and apply those tools for some time, to see quality differences. That can be difficult to justify in a number fairly normal business scenarios (low cashflow/reserves, high tech debt/regret, etc.).
To help offset the cost (and delayed benefits), I've always suggested phasing in unit test strategically. Pick a module or cross-section of the product that is suffering from bugs that customers see (i.e., affecting revenue) and add the minimum viable tests to that. Repeat as needed, and within months/years, you'll have coverage that fits the business needs well.
I haven't had this problem either. We also tend to hire more experienced folks though. A lot of startups are hiring junior folks who might not know the best way to structure things. There aren't a ton of incentives to write tests at startups and it's hard to get right. Very different when you write an infra product though :). It likely depends on the kind of product you're shipping.
Consumer might not need that as much as enterprise.
Can you recommend good resources (article/pdf, lynda, etc.) on writing tests? Less about the how (though important too) - but more about when, why, and for what parts/functionality. I find when I do write tests some are so trivial and then I stop doing them altogether.
I've actually had a bit of luck in the past with the idea that every PR required a test in some unrelated bit of code. This accomplished two things: coverage went up, and a junior Dev had to explore the code base and learn
I have not had this problem as a CTO. I try to lead by example with comprehensive tests for everything I commit. We also track test coverage with CodeClimate and make checking tests a part of our human code reviews.
Very hard to write tests in ES5 and angular 1 and karma. I am testing way more with React, jest with snapshots and enzyme, redux, my team wouldn't have 100% coverage without easy to test code
Now say I have a CTO who doesn’t like (automated) tests be cause he thinks they’re a waste of time and mostly something a type system should do for you (never mind we’re in dynamic loosetyped land). How do I introduce tests? Tried many times, always end up not getting adopted.
So hiring is pretty hard but I kinda disagree with most of the points there..
* only hire when desperate
Strong talent is so hard to get you should probably always be hiring. If you're hiring too many people your bar is probably too low.
* only hire to keep up with growth
You need to be at least a little preemptive. The hiring process itself can take months, plus the time to train even good new hires is at least a few months, AND you need your most sr. engineers to help interview so that is time they aren't writing features when you're trying to hit that critical milestone.
* Don’t hire someone to do something you’ve not yet figured out
This is probably also a mistake as software engineering has become pretty specialized. Specialized Frontend, Devops, or Data engineers can bang out solutions even a strong generalist would take ten times longer to even approximate (and most likely anything they build will be throw-away). There is huge low hanging fruit in engineering productivity /business value to getting at least a decent 80% solution for most of these areas that it's worth hiring at least one strong specialist to help Shepard development.
> Don’t hire someone to do something you’ve not yet figured out
I think this is not an indictment of hiring for something you do not know how to do, so much as it is of hiring someone before you have a defined job for them to do.
When you’re hiring an engineer, presumably you’ll be placing them onto a team that is responsible for some well-defined part of the stack. So you should know what skills you’re looking for when you’re interviewing. This should make interviewing easier; if you know what capabilities you need a new hire to have, then you know exactly what to test for in new candidates.
(This is yet another reason why generic whiteboard interviews make no sense. They’re optimizing for solving problems that could be wholly unrelated to the problems your company faces on a daily basis. I’m surprised more companies do not give interviews that focus more specifically on their relevant problem domains.)
If you don’t know what the new hire is going to do when he or she starts work, then you have no idea what skills to measure in the interview, and end up settling for the “least common denominator” of whiteboard coding ability.
In startups, specialised engineer would not necessarily add a good value due to how fast things are changing and mostly lack of proper spec. Specialised engineer can find a good solution with a proper structure which startups lack.
Enjoyed reading this article, all valid points. However, the one thing that stood out to me in this area was how light I was in effective principles of management and leadership. As a CTO of an organization of more than a handful of people you eventually "get things done" largely via other people rather than being hands on yourself. Had to read a lot of Harvard Business Review to gain the skills and confidence for that. Just like programming, there are indeed tangible skills to learn. It's not just common sense and you're not just born with it.
I have quit jobs because we kept bad hires too long and then didn’t fight to keep good hires from walking away. I think grooming and retaining talent is just as important as providing technical leadership. You need to be strong in both areas.
What always stands out about startup reflections like these is how utterly undefined, freeform, and rapidly evolving the roles can be.
The old fire hose saying is true, but it’s not just that you’re drinking from a fire hose, it’s that you often don’t know what’s coming out of the hose next. One minute deep technical decisions, the next minute helping to establish hiring philosophy, and cashflow and growth always on a background thread.
After a few years of this I think my experience is not uncommon. If you exit and through whatever circumstance (success or failure), come back inside an F500 company, you realize that trial by fire has force fed you a vast amount of new skills without even realizing it.
On one hand, the realization is really empowering, the realization you feel comfortable taking on various high impact tasks without much thought that you could have never jumped right into before. On the other hand, it can feel limiting, because F500 companies tend not to encourage even the most talented technical people to cross roles and help define company wide hiring practices.
It’s an invaluable education, but I don’t know if MBA is quite the correct ananlogy, not sure what a better comparison is.
CTO positions are much more about technology vision (e.g. choosing frameworks/technologies that can last + serve your needs today and tomorrow) and hiring/retaining talent. Everything else is gravy.
"Only hire when you feel you’re completely desperate for the role". Maybe for a tiny, extremely lean startup. But for anyone else if you wait until you are desperate you will end up hiring the first person that you think might do the job. That doesn't sound like the right way to go to me. But maybe "desperate" is relative.
Although I'm not a founder, the rule I espoused in the early days (<20 people) was to not hire for a role unless 50% of a person's time was collectively being spent on that role across the company. This was a great way to be disciplined in determining what was actually a bottleneck for our growth.
> ...it’s a blessing that my predilection for hipster technologies has not caused any serious problems.
It's entirely possible that this was the primary source of his problems with hiring, firing, testing, and a lot more.
The technology you choose determines which technologists you attract. And it's not a superficial thing, it actually says everything about the CTO's own technical skill, judgement, and experience.
As time goes on, the CTO becomes a pretty flexible position, somewhat analogous to that of a COO. This article was useful for me to figure out the kind of options I had as a CTO, in terms of specializing, as the company got progressively bigger: https://www.linkedin.com/pulse/five-flavors-being-cto-matt-t...
Early on, like OP discovered, you pretty much have to do it all, but you slowly remove yourself away from a lot of those tasks as you find better people to replace you in those areas.
I like reading posts like this one. May be they serve as a form of therapy for me that I’m not in this alone. There are others in a similar boat, fighting the good fight, making similar mistakes, and having the same realizations.
Very well; now, I can go back to work with my head up high. :)
> "I appreciate now that technologies have a surprisingly short lifespan"
This fact alone makes me so glad that I stuck with older tech that has withstood the test of time for our own SaaS. I know that we have users from bleeding edge tech companies sign up for our service, then run away when they glean the 'ancient' tech that it runs on - but then again, I think we have outlasted many other new tech frameworks/languages that have rocketed on high, then fizzled out into obscurity in that same time.
> Don’t hire someone to do something you’ve not yet figured out (some exceptional candidates can bring new capabilities to companies, but often the most reliable route is for some “founder magic” to re-assemble the company until it can perform the new thing)
I'm curious what this "founder magic" bit means. Is this advice largely because of the difficulty of trying to find a qualified expert to bring new capabilities to your company when you personally aren't familiar with that area? E.g., it's hard to not get the wool pulled over your eyes by someone who talks well but can't deliver?
>Of the list, AngularJS and MySQL have been the only ones to give us scaling problems. Our monolithic AngularJS code-bundle has got too big and the initial download takes quite a while and the application is a bit too slow. MySQL (in RDS) crashes and restarts due to growing BI query complexity and it’s been hard to fix this.
It's funny you mention that you had difficulty having your team write tests. At my company, the CTO has difficulty writing tests and the team has consistently written adequate test coverage.
I fixed this in a new project by starting with jest [1] and failing the CI if the test coverage wasn't at 100%.
Re: getting your team more interested in testing. This is not an easy thing to get momentum on if people aren't used to it. Yes to getting the test time down (and keeping it down)
Also, try defining (maybe in collaboration with the team) the tests you want people to write rather than leaving it up to them or (hopefully not) expecting 100% coverage. I wrote this on my thoughts a while back https://getcorrello.com/blog/2015/11/20/how-much-automated-t...
We had some success with increasing testing using that and code review so others could check tests were being written. Still not total buy in to be honest but a big move in the right direction :)
One surprising thing was that after years of thinking I was encouraging my team to write tests, the main feedback on why they didn't was that the didn't have time. Making it an explicit part of the process and importantly defining what tests didn't need to be maintained forever really helped.
[+] [-] latch|8 years ago|reply
If you're struggling to judge the engineering culture of a company that you're considering joining, consider this indicative of a poor one. It isn't definitive, but it's something you should ask about and probe further. Ask to see their CI dashboard and PR comments over the last few days. When they talk about Agile, ask what _engineering_ techniques (not process!) they leverage. These things will tell you if you're joining a GM or a Toyota; a company that sees quality and efficiency as opposing forces, or one that sees them as inseparable.
When it comes to tests, there are two types of people: those who know how to write tests, and those who think they're inefficient. If I had to guess what happened here, I'd say: the company had a lack of people who knew how to write effective tests combined with a lack of mentoring.
That's why you ask to see recent PR comments and find out if they do pair programming. Because these two things are decisive factors in a good engineering culture.
[+] [-] forgotpw1123|8 years ago|reply
I'm convinced that unit tests don't usually find bugs. IMO, most bugs are edge cases that were an oversight in the design. If the dev didn't handle the case in code they're not going to know to test for it. Fuzzing is a much better approach.
At my current position I have the opportunity to work with two large code-bases, built by different teams in different offices. One of the projects has ~70% code coverage, the other doesn't have a single test. Exposure to both of these systems really bent my opinion on unit tests and it has not recovered.
The project with high code coverage is basically shit and has so many bugs that we regularly cull anything marked less than "medium" severity as "not worth fixing". This project was written by a team that loves "patterns", so you can find all sorts of gems like Aspect Based Programming, CQRS, N-tier, etc. well mixed into a featureless grey Java EE goo. We get so many bug reports that its someones job to go through them.
The other project with no tests is a dream to work on. Not a single file over a few hundreds lines, everything linted and well documented. Almost no methods that don't fit on the screen, no recursion. No bullshit "layering" or "patterns". I can't remember the last time we had a bug report, since our monitoring picks up any exception client and server side. Every bug I've worked on was identified by our monitoring and fixed before anyone noticed.
Whats the difference between teams that developed these vastly different applications?? I've worked with both teams for a while, and honestly, the engineers that wrote no tests are of far higher caliber. Use Linux at home, programming since they can remember, hacking assembler on the weekends and 3D printing random useless objects they could easily buy. The other team went to school to program, and they do it because it pays the bills. Most of the bad programmers know what they're doing is wrong, but they do it anyways so they can pad their resume with more crap and tell the boss how great they are for adding machine learning to the help screen that nobody has ever opened.
If your developers are great then tests would hardly fail and be fairly useless, and if they're terrible tests don't save you. Maybe there's some middle ground if you have a mixed team or a bunch of mediocre devs??
[+] [-] mratzloff|8 years ago|reply
There are several things you as a software engineer are expected to do as a part of your job: write code, write tests, participate in code reviews, ensure successful deployment, work effectively with various groups, etc.
It's really simple: state the job requirements up front in the position description and during the hiring process. Make testing part of the code review process, and use it as an opportunity to educate about what makes a good test. Make it part of the performance review and tie raises to it (and, if it goes on long enough, continued employment).
Need to write tests for existing untested areas of the code? Have the team create a developer rotation so they dedicate someone to it for part of each sprint.
[+] [-] darkerside|8 years ago|reply
I just wanted to say that this was beautifully stated. I've been looking for better words to explain this concept to the people around me.
[+] [-] fergie|8 years ago|reply
In my GitHuby life, I write tests obsessively. In my enterprisey-softwarey life I don't, because there is no sensible way to do it.
[+] [-] sagichmal|8 years ago|reply
[+] [-] mnm1|8 years ago|reply
[+] [-] doubleocherry|8 years ago|reply
This is fantastic advice.
[+] [-] mkarazin|8 years ago|reply
I find this hard to believe. Do others CTOs / team leads find this to be the case?
I've been a CTO of two small startups with 3-7 developers. We've had resistance to tests at some points (myself included). We've solved it fairly simply. All pull requests (PRs) require tests. PRs are rejected immediately without tests. If a PR doesn't have tests and it is critical to get in, we open a new ticket to track adding tests. It isn't fool proof, but it does result in a high degree of test coverage.
And once developers understand how and where to write tests, they usually see the benefit quickly and want to write more tests.
[+] [-] apocalyptic0n3|8 years ago|reply
That said, the biggest resistance I have found is "this feature is due in three days, I need two and a half to finish, and then we have another half to review and find bugs." In the end, the biggest issue is that we have time to test on the spot or write tests, but not both. You can scrape by with just manual testing, but I don't think anyone would ever rely on automated tests 100%.
Our larger projects are test-backed, and our largest even reaches 90% coverage, but the only reasons we wrote tests for those was because we knew we would be working on them for 2-3 years and it was worth the time in that case. I wish this wasn't the case, but I've found it's always the argument against automated tests in my corner of the market
[+] [-] drspacemonkey|8 years ago|reply
IME, there are far too many "senior" devs (who absolutely should know better) who never worked on any testing-heavy teams that just don't see the point. After all, there's QA, and it's not like THIS code should break THAT code in a seemingly-unrelated part of the codebase...
[+] [-] piotr001|8 years ago|reply
Sure thing that you can use your authority to force ppl, but should you?
Smart people are hard to come by but once you have them you should let them work, and when you tell them how to do their job you implicitly assuming that you know better. Besides if you force them you achieve nothing but some brain-dead tests that are going to hunt you later and getting a budget to "rewrite tests" is a fairytale.
The art here is to build a culture that embrace the test as a powerful tool. So newcomers are quickly seeing benefits and start to write the tests in right places, not for the sake of an artificial metric.
Besides, there are plenty of places where having high coverage is going to be a waste of time: - throwaway prototypes, - heavy UI code full of animations - they need to look right that is hard to test, - infrastructure code if you have just a few servers of a particular type, - customer projects with unreasonable deadlines that are not going to be touched again,
So having your team that writes tests is a hard job and using PR policy won't help much.
The things that worked for me were: - write tests that make sense your self in early stages of the project - pair with your employees and write tests with them, - do peer reviews and suggest what could be tested and why it make sense.
[+] [-] mbesto|8 years ago|reply
People are resistant to change when they don't know how it benefits them directly and immediately.
My suggestions have been:
- By giving developers slight nudges every time they get frustrated with developing when tests aren't present is a good way to help them see the benefit. "Imagine how much easier it would be to write this piece of code if you had tests in places where this function calls other things".
- Enforcing it during commits (as you suggest, using PRs)
- Reminding your whole organization that while you migrate to implementing more testing that velocity of development will be impacted. This is really important, because it means people outside of the dev team also need to see the benefit.
- Eliminating "deadline cultures" first and then implementing unit testing
[+] [-] IronWolve|8 years ago|reply
One of the MOP checkoff boxes, test results.
So many times you could tank someone by asking "Where are the test results" and they would have to reschedule their maintenance window. If you pissed some ops engineer off, expect the question "Where are the test results" every MOP meeting.
Good times.
[+] [-] sidlls|8 years ago|reply
[+] [-] Rapzid|8 years ago|reply
Tests, like any process, should be serving you and your goals.. You shouldn't be serving your processes or testing practices. This sort of un-nuanced thinking isn't indicative of a high performing startup or CTO IMHO. Perhaps your policies are not directly indicative of your real thoughts on the matter?
[+] [-] afarrell|8 years ago|reply
What you outline seems reasonable, at least in an environment where you sometimes have hard deadlines (eg. Ticket sales for this festival go live next week). Outside of that, I'm curious what cases there are where you can have a PR which is both critical to merge and doesn't need tests. When I review a PR, I look at the tests as one way of thinking through "what edge cases have already been accounted for here?"
[+] [-] edmack|8 years ago|reply
[+] [-] robotpony|8 years ago|reply
What I find more common is for the business to be unprepared to make lateral changes to a product. Even rational unit tests are a medium term investment. You need to spend time developing features customers don't see, and apply those tools for some time, to see quality differences. That can be difficult to justify in a number fairly normal business scenarios (low cashflow/reserves, high tech debt/regret, etc.).
To help offset the cost (and delayed benefits), I've always suggested phasing in unit test strategically. Pick a module or cross-section of the product that is suffering from bugs that customers see (i.e., affecting revenue) and add the minimum viable tests to that. Repeat as needed, and within months/years, you'll have coverage that fits the business needs well.
[+] [-] agibsonccc|8 years ago|reply
Consumer might not need that as much as enterprise.
[+] [-] crdoconnor|8 years ago|reply
[+] [-] miketery|8 years ago|reply
[+] [-] paulie_a|8 years ago|reply
[+] [-] colinbartlett|8 years ago|reply
[+] [-] ojr|8 years ago|reply
[+] [-] Swizec|8 years ago|reply
[+] [-] tyingq|8 years ago|reply
Sure, it will pay off, but not right away. Something needs to cover the interim.
[+] [-] siliconc0w|8 years ago|reply
* only hire when desperate
Strong talent is so hard to get you should probably always be hiring. If you're hiring too many people your bar is probably too low.
* only hire to keep up with growth
You need to be at least a little preemptive. The hiring process itself can take months, plus the time to train even good new hires is at least a few months, AND you need your most sr. engineers to help interview so that is time they aren't writing features when you're trying to hit that critical milestone.
* Don’t hire someone to do something you’ve not yet figured out
This is probably also a mistake as software engineering has become pretty specialized. Specialized Frontend, Devops, or Data engineers can bang out solutions even a strong generalist would take ten times longer to even approximate (and most likely anything they build will be throw-away). There is huge low hanging fruit in engineering productivity /business value to getting at least a decent 80% solution for most of these areas that it's worth hiring at least one strong specialist to help Shepard development.
[+] [-] chatmasta|8 years ago|reply
I think this is not an indictment of hiring for something you do not know how to do, so much as it is of hiring someone before you have a defined job for them to do.
When you’re hiring an engineer, presumably you’ll be placing them onto a team that is responsible for some well-defined part of the stack. So you should know what skills you’re looking for when you’re interviewing. This should make interviewing easier; if you know what capabilities you need a new hire to have, then you know exactly what to test for in new candidates.
(This is yet another reason why generic whiteboard interviews make no sense. They’re optimizing for solving problems that could be wholly unrelated to the problems your company faces on a daily basis. I’m surprised more companies do not give interviews that focus more specifically on their relevant problem domains.)
If you don’t know what the new hire is going to do when he or she starts work, then you have no idea what skills to measure in the interview, and end up settling for the “least common denominator” of whiteboard coding ability.
[+] [-] nine_k|8 years ago|reply
[+] [-] mwseibel|8 years ago|reply
[+] [-] Oras|8 years ago|reply
[+] [-] ripberge|8 years ago|reply
[+] [-] mberning|8 years ago|reply
[+] [-] WhitneyLand|8 years ago|reply
The old fire hose saying is true, but it’s not just that you’re drinking from a fire hose, it’s that you often don’t know what’s coming out of the hose next. One minute deep technical decisions, the next minute helping to establish hiring philosophy, and cashflow and growth always on a background thread.
After a few years of this I think my experience is not uncommon. If you exit and through whatever circumstance (success or failure), come back inside an F500 company, you realize that trial by fire has force fed you a vast amount of new skills without even realizing it.
On one hand, the realization is really empowering, the realization you feel comfortable taking on various high impact tasks without much thought that you could have never jumped right into before. On the other hand, it can feel limiting, because F500 companies tend not to encourage even the most talented technical people to cross roles and help define company wide hiring practices.
It’s an invaluable education, but I don’t know if MBA is quite the correct ananlogy, not sure what a better comparison is.
[+] [-] mbesto|8 years ago|reply
CTO positions are much more about technology vision (e.g. choosing frameworks/technologies that can last + serve your needs today and tomorrow) and hiring/retaining talent. Everything else is gravy.
[+] [-] hartator|8 years ago|reply
Hum. I would say the reverse. Bring people that are smarter and know more than you.
[+] [-] dumbfounder|8 years ago|reply
[+] [-] paladin314159|8 years ago|reply
[+] [-] staunch|8 years ago|reply
It's entirely possible that this was the primary source of his problems with hiring, firing, testing, and a lot more.
The technology you choose determines which technologists you attract. And it's not a superficial thing, it actually says everything about the CTO's own technical skill, judgement, and experience.
[+] [-] akurilin|8 years ago|reply
Early on, like OP discovered, you pretty much have to do it all, but you slowly remove yourself away from a lot of those tasks as you find better people to replace you in those areas.
[+] [-] zeeshanm|8 years ago|reply
Very well; now, I can go back to work with my head up high. :)
[+] [-] cyberferret|8 years ago|reply
This fact alone makes me so glad that I stuck with older tech that has withstood the test of time for our own SaaS. I know that we have users from bleeding edge tech companies sign up for our service, then run away when they glean the 'ancient' tech that it runs on - but then again, I think we have outlasted many other new tech frameworks/languages that have rocketed on high, then fizzled out into obscurity in that same time.
[+] [-] fergie|8 years ago|reply
Is this a thing? How can my company get $100,000 of AWS on credit?
[+] [-] majormajor|8 years ago|reply
I'm curious what this "founder magic" bit means. Is this advice largely because of the difficulty of trying to find a qualified expert to bring new capabilities to your company when you personally aren't familiar with that area? E.g., it's hard to not get the wool pulled over your eyes by someone who talks well but can't deliver?
[+] [-] perfmode|8 years ago|reply
[+] [-] shenli3514|8 years ago|reply
Maybe they should try TiDB(https://github.com/pingcap/tidb). It is a MySQL drop-in replacement that scales.
[+] [-] senoroink|8 years ago|reply
I fixed this in a new project by starting with jest [1] and failing the CI if the test coverage wasn't at 100%.
[1] : https://facebook.github.io/jest/
[+] [-] tribby|8 years ago|reply
how long have they been using perl5 over at craiglist?
[+] [-] robinwarren|8 years ago|reply
Also, try defining (maybe in collaboration with the team) the tests you want people to write rather than leaving it up to them or (hopefully not) expecting 100% coverage. I wrote this on my thoughts a while back https://getcorrello.com/blog/2015/11/20/how-much-automated-t... We had some success with increasing testing using that and code review so others could check tests were being written. Still not total buy in to be honest but a big move in the right direction :)
One surprising thing was that after years of thinking I was encouraging my team to write tests, the main feedback on why they didn't was that the didn't have time. Making it an explicit part of the process and importantly defining what tests didn't need to be maintained forever really helped.