I believe (correct me if I am wrong) that you are referencing the OpenGL Overlay feature that allowed to create layered context.
If so, this has been deprecated and is no longer widely supported but new technique can achieve the same result with better performance: using FBO and final composition.
I would suggest that you render all your GUI and overlay rendering in a offscreen rendertarget (using Frame Buffer Object is the best for this), with an alpha channel (make sure it is being cleared with alpha value at 0).
Then render all your GUI/overlay graphic in that FBO.
When done, render a screen aligned quad binding the texture associated with that FBO on top of your existing video (which I assume to be in the backbuffer at that time).
Just make sure blending state has been enable when you render that quad.
You can even control the global alpha/coloring with texture mode MODULATE if needed or simply use REPLACE.
I hope this helps.
The hardware overlay feature I mentioned is a hardware capability that allows to draw video (or graphics) as a separate layer (just like hardware cursor) other than the GUI layer (where OpenGL drawings and graphics windows are drawn).
I'm not sure if hardware overlay method is depracated since most video players still use hardware overlay. One can easily test if a video player uses overlay just by taking a screen snapshot. In the snapshot, if you see a plain area with the same color on the video window rather than the captured frame, then video player uses the hardware overlay to display the video. Otherwise, textures or the frame buffer are used. The video players use a simple overlay transparancy technique called as color keying, which uses a specific color or color range (i.e dark green) to display the video.
In our application we are using an Mplayer backend to play the streaming video. Mplayer can display the movie on the overlay layer with hardware acceleration.
We are looking for a higher layer solution to display the video using with Mplayer since it already achieves hardware acceleration (less cpu usage around 1-2%).
We cannot use the color keying technique to draw transparent symbols since our symbology have transparent gradient bars and semi transparent layers.
If a graphics card supports per pixel alpha blending on its overlay unit, then we can draw the accelerated video in a Mplayer window and draw another window including alpha gradient symbols over the video window. By switching to the per pixel alpha mode by setting the proper register value (I don't know how on new cards), video and graphics symbology windows overlapping each other can be blended and transparency can be obtained.
I'm can perform this solution on M9 and Radeon 9200 cards. If this feature is still available on the new cards that will be the most feasible solution I believe.