I have 3 monitors hooked up as well, use two adapters, and do not have this issue. I have had vexing problems which I found to be due to defective adapters. What I've done to isolate the issue is to swap cables around, try new cables, and try new adapters. While completely possible it is an issue with your card, I highly recommend taking the trouble to swap to different/new cables and adapters to see if that resolves your problem. Pay some attention to the age of the adapters and cables. I've found that older adapters, particularly HDMI-to-DVI, seem to have problems with newer versions of the HDMI standards. I've also found that HDMI-to-DIsplay Port adapters are not all equal. It will be somewhat arduous trying different cables and adapters; however, I suspect that is the path most likely to result in success.
Hope this helps, and good luck.