Tested on 2 machines:
(1) XP32, Radeon 3850, Catalyst 10.8
(2) Vista64, Mobility X1800, Legacy Catalyst 10.2
The problem is the same on both PCs: When calling ADL_Display_CustomizedMode_Validate, I'm getting a crash. When calling ADL_Display_CustomizedMode_Add, I'm always getting ADL_ERR_INVALID_PARAM, no matter which mode values I use. I've tracked the _Add problem down a bit. Basically "atiadlxx.Send" is called (which I suppose contacts driver land) and returns with -1.
What am I trying to do? On both machines only a 1080p24 with 24.000Hz is offered, but no 1080p23 mode with 23.976Hz. So I'm trying to add a customized mode with 23Hz. Maybe this is not what the CustomizedMode APIs are for? But still, even when trying to duplicate what kind of custom modes the Catalyst Control Center creates, the problems are still the same.
Of course I've called ADL_Main_Control_Create properly, I'm not using any threading in my simple test program, stack size is big enough, and I've provided a proper memory allocation callback.
All other APIs appear to work fine, including CustomizedModeList_Get.
Edit: I've hooked into the atiadlxx.Send API, to see how the CCC calls ADL_Display_CustomizedMode_Add, but all "Send" is ever called for by the Catalyst Control Center seems to be ADL_Adapter_CrossdisplayAdapterRole_Caps. Seems that the CCC contacts the driver directly for most other stuff, instead of using the ADL APIs? Of course that would explain why the CCC can successfully create custom modes, while it seems to be impossible by using the ADL. Has anybody ever actually used ADL_Display_CustomizedMode_Add yet?