[elptacek]

No, we never really got far enough that we were able to collect that kind of data. I'm glad it was fun; certainly doing it for a living is entertaining. Mostly.

IMO, the messiness of hiring processes and the lack of adoption of work sample testing have a lot to do with each other. One of the questions I regularly ask in interviews (and have for a long time) is, "What are the first 5 things you'll have me work on?" During the last round of interviews, no one I talked to could answer that. I even had a couple of interviewers who didn't even know the name of the hiring manager or the position. Then you have other artificial filters like preferred certifications where the assumption seems to be that whatever training was required to pass can be mapped onto every organisation... or that simply possessing a cert magically grants insight into how most companies do stuff. The point is that companies don't really seem to have a good idea of what they're actually hiring for, but they get by hiring more senior people who have enough experience (hopefully) that they can figure it out on their own.

One of the things we discussed early on was avoiding having Starfighter turn into a certification process and maybe that's one of the reasons why we stumbled around a bit. I am aware that certifications have their place, but they're not really a substitute for industry experience and they definitely aren't a guarantee that a candidate can quickly figure out how a company's tech is glued together. IMO that's the power of work sample testing: you're giving a candidate a challenge where they can demonstrate aptitude. But if you don't know what actually needs to be done, you can't create that challenge, much less a rubric for describing expectations.