Hello, dear community.
I'm a rookie programmer and the problem i ran into seems too deep for me, so i need help.
I have an issue with OpenGL API interception in Win7 x64 with newest Catalyst (10.11) drivers. My program integrates into OpenGL based CAD application and intercept calls to opengl32.dll for a purpose to inject own graphics in render context during frame rendering. The way it works is quite simple: the CAD application loads my dll and when it calls my startup function i patch an Import Address Table (IAT) of the particular module in the process to intercept some OpenGL function calls from this module and render my own graphics. This is possibly the one and only method to integrate into this specific application, because i cannot rely on other non render-specific function calls.
This simple trick worked fine until i ran into a problem with newest Catalyst drivers in Win7. It seems that in terms of optimization, when CAD application start up, someone (loader, or ati driver) is patching it, so after loading into memory it is no more calling opengl32 API thru IAT. In those places of code where it should do that, now 'call ptr' commands have new arguments (i compared asm codes of both loaded and unloaded images of same module). This argument leads to an element of new table, placed somewhere between other loaded modules. This is address table of internal functions inside atio6axx module. So now, each opengl call leads to ati driver directly, skipping opengl32 interleaf. This is truly an optimization issue because every call to opengl32 in my own 'unpatched' module after few instructions jumps to corresponded atio6axx function anyway. It's even more obvious when taking into account a fact that when i run CAD application from debugger, the loaded modules are unpatched, in other words: every call to OpenGL API goes thru IAT, as it supposed to do.
The thing that surprises me most, is when i attach to the process with debugger, that specific ati address table resets to the addresses of original opengl32 functions. This costed me few weeks of understanding why hooking directly opengl32 module with hooking libraries wont work if debugger hasn't attached to the process yet, until i started to inspect process memory dumps.
So my questions are:
1. Is it vendor specific optimization, or should i look inside Windows7 DWM and WDDM specs to deal with it?
2. Is there a way to turn that optimization off? Switching Catalyst A.I. options didn't help.
3. Is there a way to locate this specific address table of ATI functions and find out which function represents which opengl call?
4. Is there another methods to intercept OpenGL calls, or to find out it was supposed to be called?
I would appreciate any your suggestions or help. Thank you in advance.