Yes, this will work on discrete graphics cards. Keep in mind that any memory you access will go over the PCIe bus which will be limited to 3-4 GB/s. This will work much better on Fusion systems (APUs).
Theoretically, there isn't a limit to the amount of memory that can be pinned. However, when the OS pins memory, it removes it from the regular pagable pool and cannot swap it to disk (this is what pinning means). If you ask for too much, the OS will refuse to do it and the call will fail (the GL driver will return generate a GL_OUT_OF_MEMORY error). It is very likely that you'll hit this limit long before you run out of address space on the GPU, although in practice we do impose a moderate limit on the amount of pinned memory so as to not impact system stability and performance.