I have a Radeon HD 4670. I installed Redhad Enterprise Workstation (RHEL) 6.1 and the latest HD 4XXX series linux propriority driver. I realized that hardware overlay is not used by the media players (mplayer, totem, vlc...). I forced Mplayer to different play with video output methods (xv, sdl, vaapi...). However, none of them used hardware overlay even if the hardware acceleration was enabled (<%1 cpu usage). Note that if the screen is captured when the hardware overlay is used for video, the video frame is not visible on the captured screen image. Instead, a plain color (usually dark green) area is displayed on the movie frame on the captured screen image.
I would like to know if the hardware overlay feature is not supported anymore on the never AMD Radeon cards (HD 4XXX or never). If it is not, then does the linux propriority driver supports hardware overlays?
AFAIK we have generally moved away from using the overlay for video playback, for a few reasons. The main ones are :
a) beginning with the R5xx family the video processing block was largely removed from the overlay pipe and the "render" portion of the video processing (scaling, colour space conversion, de-interlace, filtering etc..) was performed on the shader core instead
b) around the same time the use of compositors started to ramp up; video processed by the overlay pipe can't be picked up by the compositor so rendering video via the shader core (aka textured video) became the norm
I don't think the Linux Catalyst driver uses overlays for video these days either (I know the open source stack does not) but not 100% sure.
To further complicate things, overlay hardware is also sometimes used for a completely unrelated function - floating an OpenGL overlay (typically used for menus etc..) on top of normal application data. OpenGL overlay appears to be less common these days and eventually I expect it will get replaced with a shader-based compositing operation as well (if it hasn't already happened) but AFAIK the demand has not gone away completely yet.