This can easily be bypassed by the application itself if it checks that the DLLs or .so files loaded in memory are the approved and check-summed ones and only then progresses to doing anything. This is a non-issue. Applications that are worried about their IP already do this today. Many encrypt even their DLLs and only load then as necessary based on when they feel they are not under a debugger or a malicious DLL or .so is not loaded.
datenwolf|10 years ago
Regarding OpenGL this is a No-Go.
On Linux (and Solaris and the *BSDs) The actual OpenGL driver resides in the libGL.so, due to the lack of a standardized ICD hooking mechanism (as it exists on Windows). Hence the libGL.so on your system depends on the installed driver and version. Also on Linux people expect to be able compiling and installing their libGL.so themself. On Windows the OpenGL ICD resides in a DLL that gets loaded into the program by the graphics driver the moment a OpenGL context is created. That ICD again depends on the driver vendor and version. So checksumming is not possible as well.
To make matters worse the proprietary drivers of NVidia and ATI/AMD, if they detect program with known issues, yet broad audience (think about every AAA game ever) will actually patch parts of the program text in the memory image to silently fix bugs in that program. If you wondered why every big game release is usually accompanied by a driver update release from NVidia and ATI/AMD, well, that's why.
But even if DLL/.so checksumming were applicable, you could still ptrace into the program binary and patch the PLT/GOT entries for the `glDraw…` jumping to a little bit of dumper code (added with ptrace again) that extracts the data and then trampolines into the actual `glDraw…` function called.