[nucrow]

First off, congrats! Cool paper! Q: did you measure the correlation between CodeAid use and test scores? When educational youtube videos took off, people liked them a lot, but turns out people weren't really retaining very much information. You'd think this wouldn't be the case for CodeAid, but it's important to check

[majeedkazemi]

Thanks!

> did you measure the correlation between CodeAid use and test scores?

Unfortunately, we couldn't measure such correlations due to many external factors that can impact students' performance on test scores.

However, our previous research specifically compared students learning python coding for the first time with and without LLM code generators. Austin and I wrote a summary on it here: https://austinhenley.com/blog/learningwithai.html

We found that students who performed higher on our Scratch programming pre-tests (before starting the Python lessons), performed significantly better if they had access to the code generator. However, the system that we used in that study was very different, it showed the exact code solution to the students' query. Here with CodeAid we were trying to avoid producing direct code solutions!

[nucrow]

> Unfortunately, we couldn't measure such correlations due to many external factors that can impact students' performance on test scores.

Fair. Guess we'll have to roll it out to increase that sample size :)

> We found that students who performed higher on our Scratch programming pre-tests (before starting the Python lessons), performed significantly better if they had access to the code generator.

Interesting!

From the paper > Additionally, students in the Codex group were more eager and excited to continue learning about programming, and felt much less stressed and discouraged during the training

This by itself could reap great benefits years down the line. Me and my wife have felt the same way at work!