Our plan is to continually experiment with new ways to measure technical ability, that aren't as adversarial as most standard practices today. We'd love to hear any ideas for more experiments we could run.
Something I've been thinking about for a long time - I'll add this here because you and others at HN are more likely to be able to act on it than I am.
It seems to me that hiring teams of programmers that are known to work well together and have proven track records is a smart idea. The reason is that hiring individual programmers, who may or may not get along with one another, and whose individual hire decisions may or may not cover the breadth needed or may result in redundancies - is an inefficient and strange practice.
What if you needed an IT department and could hire groups of programmers who have, collectively, proven track records of managing IT well? Need a website done? Yeah, you could hire a designer, a backend server woman, a database guy, a CMS guru - and hope they all get along well and work together.
Or we could imagine hiring a team of people who manage themselves to get the result you're asking for.
> We'd love to hear any ideas for more experiments we could run.
As much as possible, persist the interview results per-candidate and make them reusable so people don't have to re-do the same work over and over and over again. One of the worst parts of interviewing is answering the same basic CS 101 question in every interview because everybody reads the same "how to interview a programmer" post written 15 years ago. Yes, 22 year old interviewer, I know what a compiler is, what an open source license is, what linux is, and how to write a linked list. Also, I've written code for NASA, but you have no basis to evaluate that against your limited knowledge and life experience, so keep asking things you learned two years ago.
I looked into doing something like this a while ago. The long term goal is obviously to become a candidate marketplace where you hold all the quality cards and charge a fee to companies who want a pre-vetted, guaranteed ("our programmers are warrantied against hip dysplasia!") employee pool.
The end result is essentially credentialing/certification where companies only want to hire people "Certified by X." Then, those people can also job hop easier because they are "certified" and don't have to repeat the same interview process everywhere again. Mild speed bumps in credentialing end up being: re-certification (sure I know all this now, but I may not remember any of it 6 months later), bleeding-edge certification (do you even react-goober-swift?), and maintenance certification ("I know this 20 year old technology nobody else knows!").
The reason special people will run away screaming: we don't want to be treated as interchangeable cogs. We are unicorn pony snowflakes. We want our $30k/week contracts and don't need to be evaluated by another programmer's static scripts pseudo-determining what they think is a fair evaluation of our bespoke, artisanal knowledge and abilities.
I think the idea of judging people primarily on strengths is key. But I think you actually understate this, when you say on the FAQ that 'Everyone's bad at something' -- in the sense that the amount of skills, knowledge, and abilities that any one person lacks is essentially infinite.
People analytics is a fascinating problem and I think it's high time someone tried to introduce concrete data to it. Several companies are tackling it in their own ways.
At http://InterviewKickstart.com, for example, we're starting to find some interesting correlations between personal confidence-level of an engineer coming in, and their employability. Yet another correlation is between academic achievements and employability. Small data, but interesting still.
Have you any plans to extend beyond the Technical assessment to help quantify "a good fit with the mission and team"? I'm working with a couple of companies doing interesting things in that behavioural space - and would be fascinated to see outcomes in that field from your data-driven approach. Good luck to you Guillaume and Ammon.
> First, track decisions as quantitatively as possible
First, doesn't every single programmer based ATS do this currently? Second, as soon as this quantitative information becomes symmetrical (it always does, Google "google interview prep"), how do you prevent coders from gaming ("hacking") the system?
I know you are just getting started but it would great to hear if you plan to extend this to technical management and different skillsets that would entail.
Just doing a shotgun application to a bunch of startups seems like a hilariously bad idea. But maybe I'm biased because I'm currently recovering from burnout from working at a startup that I joined for the wrong reasons.
There is only one reason to join a startup as an employee: you really really really believe in the mission and you reallyx3+ believe that the team can execute. Anything else and you aren't doing the team or yourself any favours.
So, what are you really applying for?
So you can have a "worked at a YC startup" on your resume:
Just go work for google or facebook or amazon: you'll actually learn a tonne more about engineering/product dev at any of those places and they are a much better brandname for the rest of your career. Think you wont get into those companies? Spend a month preparing for the rigamarole that is a technical interview and you'll be fine.
So you can get investor contact:
Just apply to YC/techstars/500startups. Or email investors directly. Any of those options work better. You aren't getting any meaningful investor contact by being an employee.
So you can learn how a startup works to start your own:
Anyone who tells you this is deluded or conning you. There ain't nothing to it but to do it. The only way to learn how to do your startup is to do it. Nothing else comes close. Anything else is an excuse.
Working at a startup definitely isn't the right decision for everyone and I agree your reasons are the best ones to do so. We're specifically not taking a shotgun application approach here. We work with engineers to help them figure out five startups they're interested in for specific reasons to avoid a situation where they join a startup they wouldn't enjoy working at.
> There is only one reason to join a startup as an employee: you really really really believe in the mission and you reallyx3 believe that the team can execute.
I'd say this is fairly reasonable.
> So, what are you really applying for?
For the opportunity to get in the funnel of startups using this system, so you can evaluate which ones have missions and teams you believe in and find one to work at. Presuming more than a few use this system, even if they don't use it exclusively, that's potentially a more efficient approach than research startups individually, finding ones you would want to work at, and then navigating their own individual hiring systems (and, to the extent you might miss some good startups that aren't using this system, there's no reason you can't also do that as well as the shotgun system.)
> There is only one reason to join a startup as an employee: you really really really believe in the mission and you reallyx3 believe that the team can execute.
...what if you really need the money (and never finished college, so no bigcorps are interested)?
I quite liked the application process, it was very smooth. I'm interested to see how the rest of the process goes – since the two interviews are just chatting about projects I've worked on and having someone watch me code, it doesn't seem to stressful. (Let's be honest – it's sometimes hard to get me to shut up about projects I've worked on or want to work on...)
I'm particularly interested in this as a startup filter. There are a lot of startups out there, many that I don't know about. I'm wondering if Triplebyte could help me find something I'm interested in by filtering out startups that I'm unlikely to be interested in. If they can do that, I'll be a huge fan, and if they can't (because I'm too picky), then it's not too time-consuming or stressful a process to have tried it.
I really like the idea, I just completed the programming questions. I understand that you can't have the test in every conceivable language but I feel like a few of the questions were too pythonic for really no reason at all.
My favorite part of this is that it reduces the n*m engineers-to-employers search problem to an n+m problem (to an extent).
Right now, engineers search through many companies to find a job, and companies search through many engineers to fill a position. The traditional way this is done is an interview gauntlet, whose primary purpose is to verify technical ability. Since that's something that should only be done once per engineer, TripleByte seems to be providing that intermediary service. Then a much shorter set of interviews can be used to check fit, thus saving each candidate and company an incredible amount of time.
hired.com is another company trying to do something like this, albeit using a slightly different methodology.
As a engineer who is not a startup founder, my initial reaction is that of "Don't use these guys if you are a jobseeker!" Detailed data on how you did in a specific set of tests (which may have nothing to do with what you do on the job) will be benchmarked not only during your initial application, but also throughout your job at the company, and at future companies that use TripleByte.
> "But this is a horizontal technical HR layer that spreads across many companies, instead of being contained inside one enormous corporation."
Perhaps being pessimistic, but I have no wish for a the future where I go from a company that is using TripleByte to another one that is also doing the same, and being told "We know your TripleByte score on your last job was 315, we won't give you $X unless you raise that to 400 in your first year". I also don't understand how this won't create a collusion of sorts amongst companies who all use TripleByte (since TripleByte knows exactly how much they pay you).
[Please feel free to correct me if I have missed out on some fundamental different in the way the company works. I have enormous respect for the founder (Harj) personally, but this seems like a bad idea for the majority of the workforce. ]
Thanks for the thoughts! We have no desire to create something akin to a credit score that could be used against people. What we're trying to do is build a process that makes it easy for applicants to show their strengths, that's the kind of information we want to share with employers vs a numerical score. Our focus is entirely on helping discover talented people who might be overlooked by current processes.
> We don't care where you went to school or which companies you've worked at. We only care if you can code.
Filtering candidates by pedigree is one of the biggest mistakes companies make, particularly in Silicon Valley, but filtering by coding ability (as measured primarily by online tests) is just as naive.
There are tons of people who can code themselves out of a maze but struggle to ship code that solves real problems and creates real value.
A lot of startups would do better with mediocre engineers who can see the big picture than superb engineers who can't see beyond their monitor.
> Companies should not have to make recruiting a core competency.
This is incredibly flawed, especially for startups. Recruiting is a two-way street. If an early-stage startup can't effectively sell itself to candidates, chances are it won't be able to sell itself to customers, partners, etc.
> As for their revenue model, TripleByte takes a 25 percent cut of an engineer’s first-year salary, which is a fairly typical model for recruiting agencies.
Hey Harj,
Do you think that by charging a flat % of the first year salary that TripleByte might be adding downward pressure on the potential salary for an engineer?
I think the upwards pressure from companies not being able to hire enough good engineers is likely a much stronger force. That being said, we're not tied to the % model. We plan to experiment with both our selection process and pricing to figure out what works best.
If you don't have the network and you need a recruiter (as I have in the past and will again), the 25% is how much you give up for the help getting a job.
If a jobseeker won't see that kind of value from a recruiter, they can always take it on themselves.
Founder here. Yeah, sorry about that. We totally intend to work with people without visas (it's actually where the idea started). We're just doing a phased launch. Talking to lots of people on the phone is hard to scale. We'll be removing the restriction as soon as we can.
EDIT
People without visas are free to register now. They just can't book phone calls. We'll let everyone who registers know when they can book a call.
I gotta say, this is a great move for founders + companies: Commoditizing your components will lead to downward price pressure on engineering wages. Also, it's superficially great for employees too: apply to a common app and get prequalified for N companies rather than 1. But I doubt any of those companies will ACTUALLY change their hiring funnels, which means that this is just an extra, gratuitous step in the process. Perhaps there will be the benefit of being introduced to many hiring funnels at once, but that could have been achieved by an "are you hiring" email, or by checking out those companies' hiring websites.
Harj is not really interested in creating a service that "measures" a person's ability or "predicting" their future performance, because that's impossible to do.
What they're really doing is creating a cargo cult tool that gives the illusion of being "objective" and using "data", so that they can sell the serivce based on this appearance, and later get bought out by LinkedIn or something.
Perhaps they're doing a startup themselves because they had problems finding jobs due to hiring practices such as these.
If TripleByte’s software was even trained on a sufficient volume of data, Taggar imagines it could even make the hiring decision if technical ability is the sole criteria.
When is technical ability ever the sole criteria? Granted it would be incredibly useful to know up front if a candidate is technically strong enough but you still need to establish if a candidate would work well with your team. Technical ability is worthless if the candidate has a horrible attitude.
That being said, the idea of accumulating significant data over a significant period of time and actually defining what makes a successful engineer is intriguing.
Hey, founder here. Yes, we totally agree that things other than technical ability matter. Some of these things (productivity, ability to work on a team, communication ability) we think we can evaluate. Others (company culture) we can't. As you say, however, getting better at identifying good engineers (outside of any company's specific requirements) is valuable.
I'd love if there could be more information about what YC companies are participating in hiring through this process. Or have I missed that info somewhere?
Heh, the HR at my old company sent me too many bullshit candidates. We had already tried a solution where you'd solve multiple problems and then get assigned a score, but it sucked, every single one of them was an algorithm question.
I hacked out a quick page (that'd inform you it'll log everything, but gave you the problem to solve only after you accepted), that litterally asked you to solve fizzbuzz in any language you like.
But it logged timestamped keystrokes to the server and if the tab lost focus.
Of course I wrote a small player that would replay the logs so you could watch the candidate come up with the solution and watch them writing it :)
I just had too many candidates googling questions in a phone interview, so we changed our process.
They'd take the online fizzbuzz test, and then we'll invite them for an onsite interview if they pass, in my opinion it worked surprisingly well.
Too bad I can't try out how these guys are doing it.
With that approach, anyone who immediately switches to a programmer's editor to write the code and pastes it in when done (either manually or via something like It's All Text) will look identical to a candidate who searched for the answer and pasted it in. The former is a property you'd want to select for, while the latter is one you'd want to select against.
TripleByte, like many other people and companies, refers to "software engineers" as if they are a monolithic group. Is this another case where there is an implicit "web and mobile" attached to "engineer", or is the process really flexible enough for engineers of all skillsets?
Congrats on the launch. This feels a little bit "code monkey" to me. We don't care who you are, or what you're like, as long as you can write code. Do you plan to account for the "soft skills"? I'd expect YC companies to hire for more than raw engineering abilities.
We definitely recognize there's more to a person than just engineering ability :) For now that's what we're focusing on because it's more quantifiable and we think companies are better placed to assess soft skills, since those are usually more specific to them.
While that is also human to do, and company cultures obviously develop flavors over time, when it comes to technical hiring, we sorely need an effort to first prove and then to reduce biases, based on data. This is a great effort.
At http://InterviewKickstart.com, we're trying to do this indirectly in a small way. The better you are at solving problems, the less the biases affect you.
While it seems time-saving, I don't think putting all your eggs in one basket is a good idea when it comes to interviewing. If you bomb the interview (it can happen to great engineers), then you've effectively shut yourself out of every YC company? I'd much rather interview 5 times and be 99% sure I'll get one, than go all or nothing and potentially get screwed.
[+] [-] Harj|10 years ago|reply
Our plan is to continually experiment with new ways to measure technical ability, that aren't as adversarial as most standard practices today. We'd love to hear any ideas for more experiments we could run.
[+] [-] xnull6guest|10 years ago|reply
It seems to me that hiring teams of programmers that are known to work well together and have proven track records is a smart idea. The reason is that hiring individual programmers, who may or may not get along with one another, and whose individual hire decisions may or may not cover the breadth needed or may result in redundancies - is an inefficient and strange practice.
What if you needed an IT department and could hire groups of programmers who have, collectively, proven track records of managing IT well? Need a website done? Yeah, you could hire a designer, a backend server woman, a database guy, a CMS guru - and hope they all get along well and work together.
Or we could imagine hiring a team of people who manage themselves to get the result you're asking for.
Dunno. Zany idea.
[+] [-] seiji|10 years ago|reply
As much as possible, persist the interview results per-candidate and make them reusable so people don't have to re-do the same work over and over and over again. One of the worst parts of interviewing is answering the same basic CS 101 question in every interview because everybody reads the same "how to interview a programmer" post written 15 years ago. Yes, 22 year old interviewer, I know what a compiler is, what an open source license is, what linux is, and how to write a linked list. Also, I've written code for NASA, but you have no basis to evaluate that against your limited knowledge and life experience, so keep asking things you learned two years ago.
I looked into doing something like this a while ago. The long term goal is obviously to become a candidate marketplace where you hold all the quality cards and charge a fee to companies who want a pre-vetted, guaranteed ("our programmers are warrantied against hip dysplasia!") employee pool.
The end result is essentially credentialing/certification where companies only want to hire people "Certified by X." Then, those people can also job hop easier because they are "certified" and don't have to repeat the same interview process everywhere again. Mild speed bumps in credentialing end up being: re-certification (sure I know all this now, but I may not remember any of it 6 months later), bleeding-edge certification (do you even react-goober-swift?), and maintenance certification ("I know this 20 year old technology nobody else knows!").
The reason special people will run away screaming: we don't want to be treated as interchangeable cogs. We are unicorn pony snowflakes. We want our $30k/week contracts and don't need to be evaluated by another programmer's static scripts pseudo-determining what they think is a fair evaluation of our bespoke, artisanal knowledge and abilities.
[+] [-] Alex3917|10 years ago|reply
[+] [-] soham|10 years ago|reply
At http://InterviewKickstart.com, for example, we're starting to find some interesting correlations between personal confidence-level of an engineer coming in, and their employability. Yet another correlation is between academic achievements and employability. Small data, but interesting still.
[+] [-] JacobAldridge|10 years ago|reply
Have you any plans to extend beyond the Technical assessment to help quantify "a good fit with the mission and team"? I'm working with a couple of companies doing interesting things in that behavioural space - and would be fascinated to see outcomes in that field from your data-driven approach. Good luck to you Guillaume and Ammon.
[+] [-] mbesto|10 years ago|reply
First, doesn't every single programmer based ATS do this currently? Second, as soon as this quantitative information becomes symmetrical (it always does, Google "google interview prep"), how do you prevent coders from gaming ("hacking") the system?
[+] [-] newman314|10 years ago|reply
[+] [-] mildbow|10 years ago|reply
There is only one reason to join a startup as an employee: you really really really believe in the mission and you reallyx3+ believe that the team can execute. Anything else and you aren't doing the team or yourself any favours.
So, what are you really applying for?
So you can have a "worked at a YC startup" on your resume:
Just go work for google or facebook or amazon: you'll actually learn a tonne more about engineering/product dev at any of those places and they are a much better brandname for the rest of your career. Think you wont get into those companies? Spend a month preparing for the rigamarole that is a technical interview and you'll be fine.
So you can get investor contact:
Just apply to YC/techstars/500startups. Or email investors directly. Any of those options work better. You aren't getting any meaningful investor contact by being an employee.
So you can learn how a startup works to start your own:
Anyone who tells you this is deluded or conning you. There ain't nothing to it but to do it. The only way to learn how to do your startup is to do it. Nothing else comes close. Anything else is an excuse.
[+] [-] Harj|10 years ago|reply
[+] [-] dragonwriter|10 years ago|reply
I'd say this is fairly reasonable.
> So, what are you really applying for?
For the opportunity to get in the funnel of startups using this system, so you can evaluate which ones have missions and teams you believe in and find one to work at. Presuming more than a few use this system, even if they don't use it exclusively, that's potentially a more efficient approach than research startups individually, finding ones you would want to work at, and then navigating their own individual hiring systems (and, to the extent you might miss some good startups that aren't using this system, there's no reason you can't also do that as well as the shotgun system.)
[+] [-] derefr|10 years ago|reply
...what if you really need the money (and never finished college, so no bigcorps are interested)?
[+] [-] unknown|10 years ago|reply
[deleted]
[+] [-] Harj|10 years ago|reply
If you have any feedback or thoughts, we'd love to hear them.
[+] [-] PieSquared|10 years ago|reply
I'm particularly interested in this as a startup filter. There are a lot of startups out there, many that I don't know about. I'm wondering if Triplebyte could help me find something I'm interested in by filtering out startups that I'm unlikely to be interested in. If they can do that, I'll be a huge fan, and if they can't (because I'm too picky), then it's not too time-consuming or stressful a process to have tried it.
[+] [-] tejasm|10 years ago|reply
[+] [-] quadrature|10 years ago|reply
Do most YC startups use python ?
[+] [-] klinker|10 years ago|reply
[+] [-] paragpatelone|10 years ago|reply
[+] [-] jzila|10 years ago|reply
Right now, engineers search through many companies to find a job, and companies search through many engineers to fill a position. The traditional way this is done is an interview gauntlet, whose primary purpose is to verify technical ability. Since that's something that should only be done once per engineer, TripleByte seems to be providing that intermediary service. Then a much shorter set of interviews can be used to check fit, thus saving each candidate and company an incredible amount of time.
hired.com is another company trying to do something like this, albeit using a slightly different methodology.
edit: I accidentally a word.
[+] [-] parennoob|10 years ago|reply
> "But this is a horizontal technical HR layer that spreads across many companies, instead of being contained inside one enormous corporation."
Perhaps being pessimistic, but I have no wish for a the future where I go from a company that is using TripleByte to another one that is also doing the same, and being told "We know your TripleByte score on your last job was 315, we won't give you $X unless you raise that to 400 in your first year". I also don't understand how this won't create a collusion of sorts amongst companies who all use TripleByte (since TripleByte knows exactly how much they pay you).
[Please feel free to correct me if I have missed out on some fundamental different in the way the company works. I have enormous respect for the founder (Harj) personally, but this seems like a bad idea for the majority of the workforce. ]
[+] [-] Harj|10 years ago|reply
[+] [-] 7Figures2Commas|10 years ago|reply
Filtering candidates by pedigree is one of the biggest mistakes companies make, particularly in Silicon Valley, but filtering by coding ability (as measured primarily by online tests) is just as naive.
There are tons of people who can code themselves out of a maze but struggle to ship code that solves real problems and creates real value.
A lot of startups would do better with mediocre engineers who can see the big picture than superb engineers who can't see beyond their monitor.
From https://triplebyte.com/manifesto:
> Companies should not have to make recruiting a core competency.
This is incredibly flawed, especially for startups. Recruiting is a two-way street. If an early-stage startup can't effectively sell itself to candidates, chances are it won't be able to sell itself to customers, partners, etc.
[+] [-] wasd|10 years ago|reply
Hey Harj,
Do you think that by charging a flat % of the first year salary that TripleByte might be adding downward pressure on the potential salary for an engineer?
[+] [-] Harj|10 years ago|reply
[+] [-] bgilroy26|10 years ago|reply
If a jobseeker won't see that kind of value from a recruiter, they can always take it on themselves.
[+] [-] walshemj|10 years ago|reply
[+] [-] lukasm|10 years ago|reply
[+] [-] ammon|10 years ago|reply
EDIT People without visas are free to register now. They just can't book phone calls. We'll let everyone who registers know when they can book a call.
[+] [-] beyti|10 years ago|reply
[+] [-] ryanSrich|10 years ago|reply
[+] [-] _lex|10 years ago|reply
[+] [-] smil|10 years ago|reply
What they're really doing is creating a cargo cult tool that gives the illusion of being "objective" and using "data", so that they can sell the serivce based on this appearance, and later get bought out by LinkedIn or something.
Perhaps they're doing a startup themselves because they had problems finding jobs due to hiring practices such as these.
[+] [-] egusa|10 years ago|reply
[+] [-] Peroni|10 years ago|reply
When is technical ability ever the sole criteria? Granted it would be incredibly useful to know up front if a candidate is technically strong enough but you still need to establish if a candidate would work well with your team. Technical ability is worthless if the candidate has a horrible attitude.
That being said, the idea of accumulating significant data over a significant period of time and actually defining what makes a successful engineer is intriguing.
[+] [-] ammon|10 years ago|reply
[+] [-] danielrakh|10 years ago|reply
[+] [-] mavdi|10 years ago|reply
[+] [-] kwi|10 years ago|reply
[+] [-] lawl|10 years ago|reply
I hacked out a quick page (that'd inform you it'll log everything, but gave you the problem to solve only after you accepted), that litterally asked you to solve fizzbuzz in any language you like. But it logged timestamped keystrokes to the server and if the tab lost focus.
Of course I wrote a small player that would replay the logs so you could watch the candidate come up with the solution and watch them writing it :)
I just had too many candidates googling questions in a phone interview, so we changed our process.
They'd take the online fizzbuzz test, and then we'll invite them for an onsite interview if they pass, in my opinion it worked surprisingly well.
Too bad I can't try out how these guys are doing it.
[+] [-] JoshTriplett|10 years ago|reply
[+] [-] vonmoltke|10 years ago|reply
[+] [-] tyke|10 years ago|reply
[+] [-] Harj|10 years ago|reply
[+] [-] soham|10 years ago|reply
While that is also human to do, and company cultures obviously develop flavors over time, when it comes to technical hiring, we sorely need an effort to first prove and then to reduce biases, based on data. This is a great effort.
At http://InterviewKickstart.com, we're trying to do this indirectly in a small way. The better you are at solving problems, the less the biases affect you.
[+] [-] _vn5r|10 years ago|reply
[+] [-] ipince|10 years ago|reply
[+] [-] nedwin|10 years ago|reply
[+] [-] m3talridl3y|10 years ago|reply