In a world where chat gpt exists I think even work samples are untrustworthy sources of excellence unfortunately.
It would seem one has to either watch the process or devise unique and difficult to game tests.
Work samples aren't all straight coding; in fact, only a small subset of ours are.
Later
I want to add: I fully believe LLMs can reliably knock out the code for most programming challenges, but a good coding work sample is as much about what you choose to code as it is about the code itself.
If you're asking people to invert binaries trees or whatever, that's not a work-sample test, it's a coding quiz. Work-sample tests mirror the actual work you're doing; that's what makes them predictive. Writing a function to do X, Y, or Z is usually the easy part about building software; the hard part is deciding which X, which Y, and which Z, and how they fit together.
We didn't deliberately set out to design work samples where the decisions we're grading are things LLMs don't just do for you; we do a lot of systems programming work and those kinds of decisions just dominate systems programming.
tptacek|2 years ago
Later
I want to add: I fully believe LLMs can reliably knock out the code for most programming challenges, but a good coding work sample is as much about what you choose to code as it is about the code itself.
If you're asking people to invert binaries trees or whatever, that's not a work-sample test, it's a coding quiz. Work-sample tests mirror the actual work you're doing; that's what makes them predictive. Writing a function to do X, Y, or Z is usually the easy part about building software; the hard part is deciding which X, which Y, and which Z, and how they fit together.
We didn't deliberately set out to design work samples where the decisions we're grading are things LLMs don't just do for you; we do a lot of systems programming work and those kinds of decisions just dominate systems programming.
xena|2 years ago