I worked in Admissions at a large state school from 2005-2007. Our method for determining admissibility reminded me of the heat map chart in this article - ie GPA on one axis, SAT/ACT score on the another axis, trace your finger along both paths to see if the applicant was in or out.
I probably reviewed 20-30 applications a day. If high school transcripts were universally formatted, decisions could have been instantaneous but alas, we live in the real world and some human-in-the-loop normalizing had to be done over-and-over.
With all that reviewing, patterns emerged, namely that SAT and ACT scores strongly correlate to GPA. Now, I’m the kind of person that roots for the genius to overcome his grades and emerge a genius on the SAT/ACT. But in two years, it probably happened only twice. Before calculating a normalized GPA, I could look at the test scores and predict “admit” or “deny”.
While the author is correct to say “the irrelevance of test scores is greatly exaggerated”, in my experience, whether or not something is irrelevant has very little to do with what universities do.
I’d recommend only using test scores. Or, only go with GPAs. Only test scores is more efficient. Only GPAs looks better on press releases.
So one of the best standardized tests I’ve ever taken is the Subject GRE. When I was applying for PhD programmes I and a few of my colleagues took it. It was perfect - it tested both knowledge and critical thinking skills and experimental design. It was obscure so no material was available for specific training.
In the end the scores were so accurate in predicting eventual success - a decade later, the folks who got 96% plus are all either professors or deliberately chose not to, while the rest just took industry jobs (and in my opinion because they realized that’s best for them).
Here’s the kicker - every institution explicitly said they will NOT consider these scores as part of admission process. None of the people who got in the top percentiles (4 I know) made it to a top 10 institution in the US, while a bunch of others did because they had paper authorship in their undergrad. But paper authorship in undergrad has NO correlation with your actual scientific skills! It just meant you were connected and or a hustler. It’s sad that such an awesome test is deliberately ignored by these institutions.
The irony was that these tests were also not elitist. All you needed to do was thoroughly read Lodish and Lehninger and you’re good. We did study in a semi-premium institution in India, but by no means were we privileged by any special facilities or help (at least in the context of this test). The only barrier might be the test fee itself if anything.
Respectfully, I call bullshit. You can't quantify success in a PhD with a single variable. There are a billion ways research can go right or wrong -- irrespective of your personal pedigree. Your ideas might be too early or late for your community to grasp, maybe you appeal to the wrong audience, maybe you're unaware of an application of your research, maybe you don't have the right set of collaborators or need a perspective that, often times, emerges out of a lucky encounters with someone. I respect your experience but I don't want it to give people the wrong idea about research success...
> Paper authorship had NO correlation with research success
I think paper authorship demonstrates that you're willing to put in a non-trivial amount of work to persue a problem. That seems to be atleast one attractive skill in a PhD, wouldn't you agree?
>But paper authorship in undergrad has NO correlation with your actual scientific skills!
This seems like a bad take because the goal of PhDs in the US is to produce people who can publish a lot of papers, whether that's a good ideal or not. So having produced papers as an undergrad gives you a powerful signal that you are such a person.
>Here’s the kicker - every institution explicitly said they will NOT consider these scores as part of admission process.
I know in my field, Physics, the subject GRE was very important for admissions and outside of the recent changes due to Covid I believe every top program explicitly requires the Physics GRE for admissions and its not until you get to much lower ranked programs that it becomes optional or not used at all.
>while a bunch of others did because they had paper authorship in their undergrad. But paper authorship in undergrad has NO correlation with your actual scientific skills! It just meant you were connected and or a hustler
uh, the whole point of a phd is learning how to produce scientific knowledge usually in the form of a paper? I can't think of a better measure for that then if you have experience publishing papers because that means you've contributed to the production of some scientific knowledge? sure there's tons of variability there due to circumstance, but I think my point holds
and finding connections is bad? the scientific process is one of cooperation, teams building off the work of other teams. the lone polymath is a long dead myth.
and again, having hustle is bad? I think you'd want someone who can be self motivated enough to get involved with research going on at their university
I feel like standardized tests are way more fair than GPA.
I personally struggled a lot in school due to health and other issues. I also hated doing homework which is BS work, and I always had high scores on my midterms and finals which I think matter way more.
Why does busywork play such a big role? It should be exempted. If I can get a 95+ on midterm and final I obviously understand the material. Why did I need to spend hours of my day doing BS exercises when I already understood the material? Nonsense.
So I get a B at best overall because I get a F on homework? What a joke. I really think school is a waste of time when it’s done that way. Learn stuff, move on. I wish that when I have kids I can afford to send them somewhere they can actually grow and thrive, not just sit down and do a bunch of homework problems.
The implication of the “test scores don’t matter” argument is that we have the wrong people staffing virtually every credential-based position. There’s lots of kids with 4.0 GPAs from high school at directional state university getting education degrees. Test scores is what distinguishes those kids from the ones at Stanford. And Stanford is where everyone from Big Tech to Big Banks to Big Law to Big Consulting goes to hire. Who is working at Pfizer or Astra Zeneca doing drug development? It’s the people with high test scores. Every single Supreme Court justice in the last half century got there based on test scores. (The nominee whose test scores led her to a law school ranked outside the top 50, Harriet Miers, has to withdraw because of that.) Warren Buffett got into Wharton based on test scores. Charlie Munger got into Caltech based on his scores on the army’s intelligence test. Bill Gates got an 1590 on the SAT. Even the professors saying test scores don’t matter got where they are because of test scores.
Maybe that’s all true. But it’s quite a remarkable claim that would turn the world as we know it upside down.
If you structured society such that only the top 0.5% of households by income could send their kids to college, you'd find that a lot of your successful entrepreneurs, judges, scientists, etc. went to college and had a high income growing up.
But I don't know that proposing something else, like standardized testing, as an alternative to this would be quite a remarkable claim. You're just saying that the scoring function produces a lot of false negatives.
So a test score opens doors. It’s more of a statement about how our society and institutions are structured. That implication says nothing about the quality and acumen of a person once they step through those doors. Test scores do not determine qualities and acumen, only that you meet a standard (acumen and standard is not always one-to-one). All those you named got the opportunity to demonstrate their qualities and acumen because a test score opened a door for them to do so. So I don’t believe test scores have ever mattered to society or institutions other than that: opening a door and not a guarantee of any sort of success once you step through it.
I don't think test scores are a predictor of success, but I do think they are one of the best metrics we have, and actually help poor kids to "make it" to good schools. Rich kids might have access to better teachers, but Math is the same for everyone, a poor kids using materials freely available in a public library can do very well in exams, allowing him to get into an elite school, otherwise only reserved for the elites.
Test scores are accurate at predicting your ability to succeed at further testing, I don't know if this is all that controversial.
What people are getting up in arms about is the validity of testing as a whole regarding real life outcomes, and this is an effective argument as it is damn hard to grade people by any metric that doesn't get gamified to death and resulting in people wasting a ton of time on playing the game.
I can't believe I had to scroll this far down to find this comment! People are just assuming that positively correlating testing with further testing somehow implies that the way we test is indicative of future work performance or even your quality as a human (whatever that means). They may be correlated, but a person's ability to sit exams moves the needle much more significantly. More so for universities — I've taught many students who graduate with great scores that I would never ever hire.
Using the same methods to measure wildly diverse people is a tragedy that causes individuals who could otherwise contribute to society in very meaningful ways to have fewer doors open to them.
A student who scores highly on a Calculus exam may be generally good at Calculus, or maybe they know a couple exam-specific algorithms to succeed and got a little lucky besides.
A student who fails a Calculus exam is nearly guaranteed to be poor at Calculus.
To say exams have no predictive power is to ignore the obvious.
Barring fraud, standardized tests are not gameable, unlike GPA, references, etc. You can study and improve your score, of course (just like you can study and improve GPA) but that's not "gameing" in the subversive way you mean it. Their ungameability is inconvenient for people that are looking to obfuscate the real admission criteria, which is why there is enormous effort to downplay their predictive performance.
College graduation rate is a real world outcome, which is author is referring to. You don't have to be particularly good at testing to graduate. Cs get degrees. A lot of courses are project or paper-based. If all else fails, you can take GPA boosting classes.
Of course, the far left will claim that "real world outcomes" are also systemically biased when they don't align with their priors.
> Test scores are accurate at predicting your ability to succeed at further testing
Very well put. The whole question of "do test scores predict college graduation" is easily conflated with "do test scores predict success/ability". In reality, college is a big set of tests. It would be interesting to compare with other metrics.
My take is that standardized tests are a road to social mobility. People can bullshit their way to higher grades but there are some rich kids who can't test their way out of a paper bag and the one way we can stop them from laundering their parents accomplishments is standardized testing.
Except you can buy your way into tutoring too, and pay for test accommodations, and so forth and so on. It might not amount to a huge effect, but if you're on the boundary area, it can make the difference. Someone might also argue that although bullshitting with grades makes a difference, it usually isn't going to turn a D into an A either (speaking as a university professor, with many colleagues at different universities, in those situations it's usually a matter of being pressured to turn an F into a D).
I see it the opposite. You can hire people to prep you for the test and pass it while having done zilch during HS; whereas bs'ing your way to higher grades over a period of several years is harder.
Cute. Disclosure, I make the data (transcripts, test scores) flow to decision makers as part of my day job at a Big 10 University. We just went "test optional".
I view "test optional" as a signaling exercise. In brief, how closely can you read the instructions and infer what you are not being told? The key is "test optional". If you sit for the test, you get more than just admissions, as merit-based financial awards are unlocked. The University Admissions does not require a test, but many of the donor contracts that fund scholarships do require those tests.
> I view "test optional" as a signaling exercise. In brief, how closely can you read the instructions and infer what you are not being told?
Sorry, what does that mean - who is the "you" in that sentence, and what are they inferring?
It sounds like what you mean is that that your school went test optional (ostensibly to help level the playing field), but those who are lucky or privileged enough to have the secret decoder ring can decipher the instructions and find the key to optimizing their chances of admission. If so (as someone who grew up with immigrant parents who were educated but definitely would not have had the awareness to navigate this), that seems really fucked up...
Interesting, and good to know! But wouldn't you say that students are better off not submitting their scores if they scored low on the tests? Only given a sufficiently high enough score does it become a "nothing to lose" situation.
Unless the issue is students just don't want to take the test to begin with.
I see zero discussion of test scores and people with learning disabilities. Those who cannot be properly accommodated will be unfairly represented and often have to put up with a lot more barriers (such as frequently messed up accommodations on test day leading to anxiety). Usually these accommodations help normalize the specific testing scenario and people with disabilities do not need as drastic accommodations in real job setups. The tests are designed for the average test taker but have lots of hiccups for those that deviate from the average.
It’s a difficult problem to solve, but it’s unfair if a student gets easily distracted or anxious when taking a test around others in a time constrained setting. Those types of external factors generally shouldn’t be represented in a student’s test results yet often are. If tests are to be a good indicator for all students, I think accommodations have a ways to go.
If tests are indeed a generally good indicator of a student’s success and are continued to be used in such a manner, I feel it’s important that the experience is fair for all types of test takers. Many US tests such as the CPA exam do a great job of handling accommodations. However, many others (like the PSAT) have a lot of room for improvement.
> I see zero discussion of test scores and people with learning disabilities. Those who cannot be properly accommodated will be unfairly represented…
I have long wondered how these interact / can support people who are a couple of sigma out from what we have decided is the “norm”. I can see how additional test time can help someone with anxiety or distraction issues, which can help someone with other strengths shine.
But how can this work outside the academy? A trial lawyer has only so much time to work on the case before it goes to trial or before some response to a filing is required. An engineer building a rocket still has to get something designed before assembly. Arent such time accommodations are hard to implement in the “real world”?
There are nice counter examples to my question. For example dyslexic people struggle with all sorts of cases, yet technology can help many of them (e.g. new kinds of fonts), and not just current students but people already in the work force.
I can also see the school accommodation helping someone who is separately working on their anxiety issues — though mental health cost support is quite poor, at least in the US.
But at their root: do these accommodations help or do they provide an unrealistic hope to the student?
NB: I want everyone to have a fair shake, and am a fan of ADA accommodation and affirmative action and related diversity efforts. I ask this question within that context.
... so here's the thing (speaking as someone who has taken many of these tests, been in admissions committees, and researches these tests). Take learning disabilities, and now expand it out to consider the similar factors, but with things like cultural background, life circumstances, and so forth and so on. Learning disabilities are not the same as those other things -- I don't want to equate them -- but they share some of the same issues with the test being normed on a certain standard population, and once you get outside of that, weird things happen.
Those density gradient plots in the linked blog post are interesting and useful to think about, but they're kinda hiding the fact that the vast majority of the data going into them is based on test-takers with a certain standard background, characteristics, and so forth. That in turn shapes the contours of what is error in those plots and analyses. The problem isn't necessarily that the tests are useless per se, it's that it's hard to interpret them in a way that accommodates people in nonstandard situations, or even accommodates the idiosyncracies of differences between people in general, differences that wouldn't matter in the real world.
The errors of using ACT-only, GPA-only, and so forth in the post is pretty interesting, but the author is missing the fact that that table has never really been the point of contention. The point of contention is whether that table's patterns apply equally across divisions of gender, race, SES background, age, disability status, and so forth and so on. They raise the point that the improvement in error from switching from ACT to GPA is comparable to (or better than) adding ACT to GPA, but isn't the real question whether doing so increases certain types of "predictable error variance", in the sense that you could predict the residual from things like SES, race, and so forth and so on?
I think GPA is seen as more acceptable than ACT because it's exchangeable in an important sense with the criterion being predicted. That is, if you want to know college GPA, maybe secondary school GPA is a little flawed, but at least it's ostensibly similar in terms of what it actually is. I think people have a sense that, say, you aren't using standardized tests as the criterion for college graduation, so why use it for prediction? Why have college and college GPA at all? Why not just let people take standardized tests and skip the whole degree program thing? There's reasonable arguments for doing that, but also reasonable arguments for not doing it, and many of those are the same arguments for and against using the test for admissions.
Tangent - I think that my particular disability (ADHD) helped me with my test scores. I was terrible at studying, but on the tests I would hyper-focus and do extremely well.
On the other hand, on the GRE, the writing portion was last and I could not bring myself to care. I scored in the 50th percentile, which for someone who regularly scores in the 99th percentile on standardized tests (including ones with writing portions), is pretty bad.
This whole concept of measuring SAT/ACT scores or GPAs and using them to "predict" graduation rates is bonkers.
Sounds a lot like the precogs in Philip K. Dick's Minority Report.
Several decades ago, I had a high-school History teacher. We had an elite private high-school in the same city as I lived (India has both private and public schools). They had an extremely rigorous selection process, including infamous interviews with parents for admission to kindergarten. Our history teacher once remarked, "They take horses and send them out as horses. We take horses, asses and everything else and send them out as horses".
The SAT/ACT/GPA filter seems to be focused on selecting horses and proudly declaring all horses graduated.
Correct. Although there are huge benefits of a gate-kept horse-only environment. Competing against each other, the horses become more excellent. This is why the parents are paying the big bucks and putting up with the process. It is about gaining access to a horse-only environment.
> We take horses, asses and everything else and send them out as horses.
This is just envy at the easier job of the private school teachers, not indicative of their actual transformational capabilities of the public school teacher. The best that these teachers can do (which is no small feat) is to manage the classroom such that the donkeys and the horses do not fight and the horses can stay motivated despite having fewer peers.
Test scores have already been irrelevant for certain segments for over a decade, speaking as someone of East Asian ethnicity.
Personal anecdote: 3.7 GPA in average high school with 8 APs (some of which were self-study), 2400 SAT, 36 ACT, lots of state-level top 3 finishes in multiple academic competitions, varsity track, musical instruments, first generation immigrant, etc.
Ended up going to what I thought was my safety school and rejected/waitlisted at every reach school I applied to. I sometimes wish colleges mandated name/ethnicity-blind application reviews - not to sound ungrateful, but I’m still incredulous to this day that the best I could do was a public state school (still top 50 admittedly).
It ended up being a great experience in that it was a forcing function and made me realize most of these rules and expectations around admissions were meaningless. But I probably could have saved years in high school exploring things that mattered rather than optimizing for a college application process that didn’t ultimately end up feeling very fair.
I graduated with the highest SAT scores in my school and the second highest in my city back in the early 90s. I’m not bragging, it was a relatively small town in the south with not great school systems and a total graduating class of 1500 across all of the schools.
What are the chances that happen because I was so smart or because I had a mother who was not only a high school math teacher but also spent years volunteering to teach SAT prep classes?
Second story: my step son didn’t have the ACT scores to get into the college he wanted to go to. Four months later and with the help of a private tutor who was a teacher (not in his school) that we paid $100/hour for 12 hours, he suddenly raised his scores more than enough. Did he gain aptitude in 12 weeks or did he learn test taking techniques?
Yeah, I hear this, but in 1995 I just bought an SAT prep book for $12 and spent a couple months of Saturdays practicing and did very well. My family lived well below the poverty line. SAT prep is not confined to the privileged.
> What are the chances that happen because I was so smart or because I had a mother who was not only a high school math teacher but also spent years volunteering to teach SAT prep classes?
It sounds like you are saying that good parenting shouldn't have any effect of kids success in life.
Look, we know that poor parenting results in kids with poor outlooks, but penalising good parenting by taking away the effect of good parenting is probably not a good way to go about helping those kids with poor parenting.
Don’t you think a mother able to teach those classes would be in general a better test taker than the average person, and so her children likely would be too?
And nobody is denying that test scores can be raised. That’s likely a fixed amount so just amounts to constant noise. I think you’re as likely to find a way to consistently raise a 600 SAT to 1600 as you are trying to train any couch potato into an Olympic sprinter.
Why is it that the anti test score argument almost always tries to chalk it up to hiring tutors? They consistently try to make it sound like people "buy" higher test scores with tutoring in order to discredit tests like the SAT
There is research on this. I believe it was Thomas Espenshade, social scientist at Princeton, who had data on test prep access, modeled along with SES, etc. Test prep is worth about 40 points on the SAT. Not trivial, but not a game changer.
It's true that people who are otherwise well off are more likely to have test prep, and those people's test scores are often several standard deviations higher than low-income students from bad schools without test prep. But the counterfactual of (same person) +/- test prep = about 40 SAT points.
Agreed! Although, I admittedly didn’t read the post carefully enough to evaluate how accurate its analysis of the original paper. But all else being equal, we’d live in a better world if papers that get a lot of popular attention are subjected to this scrutiny with an eye towards explaining what works/ doesn’t work for a general audience
It'a sunken cost effect at paly, little more than that.
Oh, and parents pressure, least we forget the role of family.
You can test my claim empirically by asking yourself: "are all college graduates I met more intelligent than me?".
Admission test measure how well you can prepare for admission tests, nothing more.
There are too many confounding factors and co-factors that play in the career of a relatively fresh semi-adult aged 17-18, that reducing it to a single number is just moronic or bad-faith utilitaristic escape route.
Universities only care that you graduate so they can make money, therefore they use the simplest KPI to measure that. Let's not forget Universities are staffed with people and are a business.
> The full dataset isn’t available, but since we have the number of students in each ACT / GPA bin above, we can create a “pseudo” dataset, with just a small loss of precision.
Big oof. This is an example of terminal data science poisoning. The data in the paper already proved the point of the article! Using it to create some chart gore where you reproduce the same plot in uncanny valley levels of resolution doesn't add to the argument. This is using very sophisticated techniques to achieve what you could by simply blurring an image of the first chart.
It is not that they are irrelevant, it is that they are ableist, selecting for a narrow skill and ability combination, and creating a barrier for anyone else who lacks it, for whatever reason.
[+] [-] ajwinn|3 years ago|reply
I probably reviewed 20-30 applications a day. If high school transcripts were universally formatted, decisions could have been instantaneous but alas, we live in the real world and some human-in-the-loop normalizing had to be done over-and-over.
With all that reviewing, patterns emerged, namely that SAT and ACT scores strongly correlate to GPA. Now, I’m the kind of person that roots for the genius to overcome his grades and emerge a genius on the SAT/ACT. But in two years, it probably happened only twice. Before calculating a normalized GPA, I could look at the test scores and predict “admit” or “deny”.
While the author is correct to say “the irrelevance of test scores is greatly exaggerated”, in my experience, whether or not something is irrelevant has very little to do with what universities do.
I’d recommend only using test scores. Or, only go with GPAs. Only test scores is more efficient. Only GPAs looks better on press releases.
[+] [-] ramraj07|3 years ago|reply
In the end the scores were so accurate in predicting eventual success - a decade later, the folks who got 96% plus are all either professors or deliberately chose not to, while the rest just took industry jobs (and in my opinion because they realized that’s best for them).
Here’s the kicker - every institution explicitly said they will NOT consider these scores as part of admission process. None of the people who got in the top percentiles (4 I know) made it to a top 10 institution in the US, while a bunch of others did because they had paper authorship in their undergrad. But paper authorship in undergrad has NO correlation with your actual scientific skills! It just meant you were connected and or a hustler. It’s sad that such an awesome test is deliberately ignored by these institutions.
The irony was that these tests were also not elitist. All you needed to do was thoroughly read Lodish and Lehninger and you’re good. We did study in a semi-premium institution in India, but by no means were we privileged by any special facilities or help (at least in the context of this test). The only barrier might be the test fee itself if anything.
[+] [-] muds|3 years ago|reply
> Paper authorship had NO correlation with research success
I think paper authorship demonstrates that you're willing to put in a non-trivial amount of work to persue a problem. That seems to be atleast one attractive skill in a PhD, wouldn't you agree?
[+] [-] adamsmith143|3 years ago|reply
This seems like a bad take because the goal of PhDs in the US is to produce people who can publish a lot of papers, whether that's a good ideal or not. So having produced papers as an undergrad gives you a powerful signal that you are such a person.
>Here’s the kicker - every institution explicitly said they will NOT consider these scores as part of admission process.
I know in my field, Physics, the subject GRE was very important for admissions and outside of the recent changes due to Covid I believe every top program explicitly requires the Physics GRE for admissions and its not until you get to much lower ranked programs that it becomes optional or not used at all.
[+] [-] thomasahle|3 years ago|reply
> All you needed to do was thoroughly read Lodish and Lehninger and you’re good.
How can these be true at the same time? If you can just read a book and get a good score, why don't everyone just get the good score?
[+] [-] ausbah|3 years ago|reply
uh, the whole point of a phd is learning how to produce scientific knowledge usually in the form of a paper? I can't think of a better measure for that then if you have experience publishing papers because that means you've contributed to the production of some scientific knowledge? sure there's tons of variability there due to circumstance, but I think my point holds
and finding connections is bad? the scientific process is one of cooperation, teams building off the work of other teams. the lone polymath is a long dead myth.
and again, having hustle is bad? I think you'd want someone who can be self motivated enough to get involved with research going on at their university
[+] [-] okay_dude_q|3 years ago|reply
[deleted]
[+] [-] mradek|3 years ago|reply
I personally struggled a lot in school due to health and other issues. I also hated doing homework which is BS work, and I always had high scores on my midterms and finals which I think matter way more.
Why does busywork play such a big role? It should be exempted. If I can get a 95+ on midterm and final I obviously understand the material. Why did I need to spend hours of my day doing BS exercises when I already understood the material? Nonsense.
So I get a B at best overall because I get a F on homework? What a joke. I really think school is a waste of time when it’s done that way. Learn stuff, move on. I wish that when I have kids I can afford to send them somewhere they can actually grow and thrive, not just sit down and do a bunch of homework problems.
(Just my $0.02)
[+] [-] rayiner|3 years ago|reply
Maybe that’s all true. But it’s quite a remarkable claim that would turn the world as we know it upside down.
[+] [-] karpierz|3 years ago|reply
If you structured society such that only the top 0.5% of households by income could send their kids to college, you'd find that a lot of your successful entrepreneurs, judges, scientists, etc. went to college and had a high income growing up.
But I don't know that proposing something else, like standardized testing, as an alternative to this would be quite a remarkable claim. You're just saying that the scoring function produces a lot of false negatives.
[+] [-] jonnybgood|3 years ago|reply
[+] [-] derbOac|3 years ago|reply
[+] [-] masterof0|3 years ago|reply
[+] [-] cbruns|3 years ago|reply
[+] [-] faeriechangling|3 years ago|reply
What people are getting up in arms about is the validity of testing as a whole regarding real life outcomes, and this is an effective argument as it is damn hard to grade people by any metric that doesn't get gamified to death and resulting in people wasting a ton of time on playing the game.
[+] [-] paraknight|3 years ago|reply
Using the same methods to measure wildly diverse people is a tragedy that causes individuals who could otherwise contribute to society in very meaningful ways to have fewer doors open to them.
[+] [-] nickelpro|3 years ago|reply
A student who fails a Calculus exam is nearly guaranteed to be poor at Calculus.
To say exams have no predictive power is to ignore the obvious.
[+] [-] club_tropical|3 years ago|reply
[+] [-] Aunche|3 years ago|reply
Of course, the far left will claim that "real world outcomes" are also systemically biased when they don't align with their priors.
[+] [-] klabb3|3 years ago|reply
Very well put. The whole question of "do test scores predict college graduation" is easily conflated with "do test scores predict success/ability". In reality, college is a big set of tests. It would be interesting to compare with other metrics.
[+] [-] PaulHoule|3 years ago|reply
[+] [-] derbOac|3 years ago|reply
[+] [-] insane_dreamer|3 years ago|reply
[+] [-] ouid|3 years ago|reply
[+] [-] athorax|3 years ago|reply
[+] [-] WBrentWilliams|3 years ago|reply
I view "test optional" as a signaling exercise. In brief, how closely can you read the instructions and infer what you are not being told? The key is "test optional". If you sit for the test, you get more than just admissions, as merit-based financial awards are unlocked. The University Admissions does not require a test, but many of the donor contracts that fund scholarships do require those tests.
[+] [-] tqi|3 years ago|reply
Sorry, what does that mean - who is the "you" in that sentence, and what are they inferring?
It sounds like what you mean is that that your school went test optional (ostensibly to help level the playing field), but those who are lucky or privileged enough to have the secret decoder ring can decipher the instructions and find the key to optimizing their chances of admission. If so (as someone who grew up with immigrant parents who were educated but definitely would not have had the awareness to navigate this), that seems really fucked up...
[+] [-] felideon|3 years ago|reply
Unless the issue is students just don't want to take the test to begin with.
[+] [-] pwpw|3 years ago|reply
It’s a difficult problem to solve, but it’s unfair if a student gets easily distracted or anxious when taking a test around others in a time constrained setting. Those types of external factors generally shouldn’t be represented in a student’s test results yet often are. If tests are to be a good indicator for all students, I think accommodations have a ways to go.
If tests are indeed a generally good indicator of a student’s success and are continued to be used in such a manner, I feel it’s important that the experience is fair for all types of test takers. Many US tests such as the CPA exam do a great job of handling accommodations. However, many others (like the PSAT) have a lot of room for improvement.
[+] [-] gumby|3 years ago|reply
I have long wondered how these interact / can support people who are a couple of sigma out from what we have decided is the “norm”. I can see how additional test time can help someone with anxiety or distraction issues, which can help someone with other strengths shine.
But how can this work outside the academy? A trial lawyer has only so much time to work on the case before it goes to trial or before some response to a filing is required. An engineer building a rocket still has to get something designed before assembly. Arent such time accommodations are hard to implement in the “real world”?
There are nice counter examples to my question. For example dyslexic people struggle with all sorts of cases, yet technology can help many of them (e.g. new kinds of fonts), and not just current students but people already in the work force.
I can also see the school accommodation helping someone who is separately working on their anxiety issues — though mental health cost support is quite poor, at least in the US.
But at their root: do these accommodations help or do they provide an unrealistic hope to the student?
NB: I want everyone to have a fair shake, and am a fan of ADA accommodation and affirmative action and related diversity efforts. I ask this question within that context.
[+] [-] derbOac|3 years ago|reply
Those density gradient plots in the linked blog post are interesting and useful to think about, but they're kinda hiding the fact that the vast majority of the data going into them is based on test-takers with a certain standard background, characteristics, and so forth. That in turn shapes the contours of what is error in those plots and analyses. The problem isn't necessarily that the tests are useless per se, it's that it's hard to interpret them in a way that accommodates people in nonstandard situations, or even accommodates the idiosyncracies of differences between people in general, differences that wouldn't matter in the real world.
The errors of using ACT-only, GPA-only, and so forth in the post is pretty interesting, but the author is missing the fact that that table has never really been the point of contention. The point of contention is whether that table's patterns apply equally across divisions of gender, race, SES background, age, disability status, and so forth and so on. They raise the point that the improvement in error from switching from ACT to GPA is comparable to (or better than) adding ACT to GPA, but isn't the real question whether doing so increases certain types of "predictable error variance", in the sense that you could predict the residual from things like SES, race, and so forth and so on?
I think GPA is seen as more acceptable than ACT because it's exchangeable in an important sense with the criterion being predicted. That is, if you want to know college GPA, maybe secondary school GPA is a little flawed, but at least it's ostensibly similar in terms of what it actually is. I think people have a sense that, say, you aren't using standardized tests as the criterion for college graduation, so why use it for prediction? Why have college and college GPA at all? Why not just let people take standardized tests and skip the whole degree program thing? There's reasonable arguments for doing that, but also reasonable arguments for not doing it, and many of those are the same arguments for and against using the test for admissions.
[+] [-] lukas099|3 years ago|reply
On the other hand, on the GRE, the writing portion was last and I could not bring myself to care. I scored in the 50th percentile, which for someone who regularly scores in the 99th percentile on standardized tests (including ones with writing portions), is pretty bad.
[+] [-] vivegi|3 years ago|reply
Sounds a lot like the precogs in Philip K. Dick's Minority Report.
Several decades ago, I had a high-school History teacher. We had an elite private high-school in the same city as I lived (India has both private and public schools). They had an extremely rigorous selection process, including infamous interviews with parents for admission to kindergarten. Our history teacher once remarked, "They take horses and send them out as horses. We take horses, asses and everything else and send them out as horses".
The SAT/ACT/GPA filter seems to be focused on selecting horses and proudly declaring all horses graduated.
[+] [-] club_tropical|3 years ago|reply
Correct. Although there are huge benefits of a gate-kept horse-only environment. Competing against each other, the horses become more excellent. This is why the parents are paying the big bucks and putting up with the process. It is about gaining access to a horse-only environment.
> We take horses, asses and everything else and send them out as horses.
This is just envy at the easier job of the private school teachers, not indicative of their actual transformational capabilities of the public school teacher. The best that these teachers can do (which is no small feat) is to manage the classroom such that the donkeys and the horses do not fight and the horses can stay motivated despite having fewer peers.
[+] [-] jklm|3 years ago|reply
Ended up going to what I thought was my safety school and rejected/waitlisted at every reach school I applied to. I sometimes wish colleges mandated name/ethnicity-blind application reviews - not to sound ungrateful, but I’m still incredulous to this day that the best I could do was a public state school (still top 50 admittedly).
It ended up being a great experience in that it was a forcing function and made me realize most of these rules and expectations around admissions were meaningless. But I probably could have saved years in high school exploring things that mattered rather than optimizing for a college application process that didn’t ultimately end up feeling very fair.
[+] [-] scarface74|3 years ago|reply
I graduated with the highest SAT scores in my school and the second highest in my city back in the early 90s. I’m not bragging, it was a relatively small town in the south with not great school systems and a total graduating class of 1500 across all of the schools.
What are the chances that happen because I was so smart or because I had a mother who was not only a high school math teacher but also spent years volunteering to teach SAT prep classes?
Second story: my step son didn’t have the ACT scores to get into the college he wanted to go to. Four months later and with the help of a private tutor who was a teacher (not in his school) that we paid $100/hour for 12 hours, he suddenly raised his scores more than enough. Did he gain aptitude in 12 weeks or did he learn test taking techniques?
[+] [-] hcurtiss|3 years ago|reply
[+] [-] lelanthran|3 years ago|reply
It sounds like you are saying that good parenting shouldn't have any effect of kids success in life.
Look, we know that poor parenting results in kids with poor outlooks, but penalising good parenting by taking away the effect of good parenting is probably not a good way to go about helping those kids with poor parenting.
[+] [-] rajin444|3 years ago|reply
And nobody is denying that test scores can be raised. That’s likely a fixed amount so just amounts to constant noise. I think you’re as likely to find a way to consistently raise a 600 SAT to 1600 as you are trying to train any couch potato into an Olympic sprinter.
[+] [-] cr4nberry|3 years ago|reply
[+] [-] pocketsand|3 years ago|reply
It's true that people who are otherwise well off are more likely to have test prep, and those people's test scores are often several standard deviations higher than low-income students from bad schools without test prep. But the counterfactual of (same person) +/- test prep = about 40 SAT points.
[+] [-] KevinBenSmith|3 years ago|reply
[+] [-] jeffhwang|3 years ago|reply
[+] [-] xbar|3 years ago|reply
"2. It might be that test scores don’t predict college graduation rates[...] is provably false."
[+] [-] MarcoSanto|3 years ago|reply
[+] [-] awkward|3 years ago|reply
Big oof. This is an example of terminal data science poisoning. The data in the paper already proved the point of the article! Using it to create some chart gore where you reproduce the same plot in uncanny valley levels of resolution doesn't add to the argument. This is using very sophisticated techniques to achieve what you could by simply blurring an image of the first chart.
[+] [-] forgotmypw17|3 years ago|reply
[+] [-] tagami|3 years ago|reply
[+] [-] dirtyid|3 years ago|reply
[+] [-] readthenotes1|3 years ago|reply
My older brother switched from a B student in a private high school to the public high school and slept his ways to A's