Hi,
Actually, GL_OUT_OF_MEMORY can be generated at any time. Checking the spec, v.4.3, p.16, it says:
Several error generation conditions are implicit in the description of every GL command.
- ...
- ...
- If memory is exhausted as a side effect of the execution of a command, an OUT_OF_MEMORY error may be generated.
The transfer you are attempting is very large, even on a machine with 8G of memory and a high end graphics card. I suspect that you are squeaking in just under a limit on NVIDIA and just over on AMD. It is likely that some internal allocation has failed in the driver and it can't allocate some temporary buffer to make the transfer. We'll see if there's something we can do in the driver to break such large transfers up.
In the meantime, you could try calling glReadPixels multiple times to transfer regions of the framebuffer. This could get you running again without needing to sacrifice framebuffer resolution.
Cheers,