We are developing a video overlay driver for an embedded platform using ATI M9 chip. When we use an RGB overlay surface, we observed that some high frequency patterns (simply 1 pixel wide white-black columns) at 1.0 (linear) gamma value are displayed with green shades. When we set the overlay gamma register for 2.2 or higher the green layers turn to magenta. A gamma value between 1.0 and 2.2 seems better. However, there are still green or magenta layers. We did not observe the same problem for the YUV422 overlay surfaces. It seems that the color space transformation for YUV(YUV->RGB) somehow prevents this issue. I tried to change the gamma values written to OV0_GAMMA_XXX registers. However, there was no visible change. Changing OV0_GAMMA_SEL bits of OV0_SCALE_CNTL register have affect on the display. Since changing OV0_GAMMA_XXX have no affect for the given OV0_GAMMA_SEL, OV0_GAMMA_SEL bits seems to control hardcoded gamma values.
Is there any suggestion to program the gamma values for the RGB overlay surfaces?
I realized that the issue was due to an incorrect register update.
I solve the issue when both OV0_P1_H_INC and OV0_P23_H_INC are set to the same h. inc. value for rgba format (unlike yuv).