I knew this reminded me of something, and it turns out it was one of his older posts which is also worth a read: http://blog.regehr.org/archives/970 (Finding Undefined Behavior Bugs by Finding Dead Code)
The PDF it links is 404, but the Wayback Machine caught it in a few different revisions (the date of Regehr's post would correspond more closely with the earlier revision):
It's an interesting method. Does anyone know if these kinds of torture tests get collected into a common library to help any future compilers or if they just result in a paper and bug reports to current ones?
The generated tests themselves are hardly worth the bytes they occupy. The value is in the compact program that can generate as many of them as desired very quickly, with varying options to focus on varying aspects of compilation (with or without bit-fields, with or without deep pointer nesting, …)
We forced Xuejun Yang (who turned Randprog, the prototype that came before Csmith into Csmith) to fix more bugs than were necessary for his PhD or than you can expect a research prototype to have bugs fixed (I am one of the developers of Frama-C, pitting Frama-C “against” Csmith was my hobby for a summer, and we found and reported as many bugs in Csmith as we found bugs in Frama-C). The sentence “over the last couple of years I’ve slacked off on reporting compiler bugs” near the post's conclusion is telling. You can expect the same story to unroll for EMI. Researchers are not rewarded for maintaining software ad vitam æternam, even if the software is useful, but hopefully they have or will soon release the generator as open-source, and then if you find it too useful not to use further, you can fix or work around bugs as you find them.
As an example, I think that there is still one bug in Csmith that we work around by ignoring programs that have the symptoms that usually indicate that bug (we still use Csmith to test Frama-C after we have finished a major feature that could introduce the sort of bug it can detect).
Looks like there is significant overlap between metamorphic testing an property based testing. Can anyone come up with examples that are clearly one but not the other?
[+] [-] petercooper|11 years ago|reply
[+] [-] pasbesoin|11 years ago|reply
The PDF it links is 404, but the Wayback Machine caught it in a few different revisions (the date of Regehr's post would correspond more closely with the earlier revision):
http://web.archive.org/web/20130810232543/http://pdos.csail....
http://web.archive.org/web/20131214080638/http://pdos.csail....
[+] [-] pedrocr|11 years ago|reply
[+] [-] pascal_cuoq|11 years ago|reply
We forced Xuejun Yang (who turned Randprog, the prototype that came before Csmith into Csmith) to fix more bugs than were necessary for his PhD or than you can expect a research prototype to have bugs fixed (I am one of the developers of Frama-C, pitting Frama-C “against” Csmith was my hobby for a summer, and we found and reported as many bugs in Csmith as we found bugs in Frama-C). The sentence “over the last couple of years I’ve slacked off on reporting compiler bugs” near the post's conclusion is telling. You can expect the same story to unroll for EMI. Researchers are not rewarded for maintaining software ad vitam æternam, even if the software is useful, but hopefully they have or will soon release the generator as open-source, and then if you find it too useful not to use further, you can fix or work around bugs as you find them.
As an example, I think that there is still one bug in Csmith that we work around by ignoring programs that have the symptoms that usually indicate that bug (we still use Csmith to test Frama-C after we have finished a major feature that could introduce the sort of bug it can detect).
See also http://blog.regehr.org/archives/1058 on the same blog for additional thoughts on the fate of academic software.
[+] [-] TheLoneWolfling|11 years ago|reply
Another idea: taking bug reports for one compiler for a language and using that as fuzzing input for another compiler for the same language.
[+] [-] michaelfeathers|11 years ago|reply
[+] [-] davorak|11 years ago|reply
http://en.wikipedia.org/wiki/Metamorphic_testing