I have had this monitor for about 5 years, give or take. It's been forever since I've revisited this issue, but I have never been able to put my monitor in 165hz. I have searched countless forums for answer's, but they all say to check your display and change it in the advanced settings. However, my settings don't even include any other choices... overclocked or not.
I have tried every display possible, sometimes I can gain 0.004hz and get up to 60.004hz when I go to the smallest resolution. I don't know why I didn't try sooner, but if anyone has any idea, please help me out. Also, I only have 1 monitor so there is no chance it is not the correct monitor. And yes I have used the center on the bottom right to overclock, change the settings, and reboot ( it is successful every time, but does not change the framerate). I also have a GTX 980 (a little old), but should be able to do this easily.
I have been reading the manual for your monitor..
You will need to use the display port cable to get 144hz, HDMI only goes to 60Hz..
From what i can see, you should have a disc for the monitor with the monitor driver on it.. It does say you will need the driver installed..
Also from the manual, you may have to use the OSD on the monitor to set to DP and overclock to a higher refresh rate..
Personally i would leave it at 144Hz and not OC..
AOC AG271QG 27 inch monitor | AOC Monitors | AOC Monitors