In theory, we should be able to support very large buffer sizes - well in excess of a gigabyte. However, there could well be some internal problem that either causes the memory allocation to fail (such as address space fragmentation or operating system state), or may cause us to fail to upload the data correctly.
As a workaround, I would suggest that you break the data-set up. If you use two buffers and separate vertex attributes rather than interleaved attributes, with one buffer for 'xyz' and one for 'rgbflags', you might get it working. Otherwise, you will need to break your dataset up further.
A few questions:
- What operating system are you running?
- Does it appear that the buffer is allocated but that the data is incorrect, or that the buffer is not allocated at all?
- Do you get a GL_OUT_OF_MEMORY error or anything like that?
- How many other buffers does your application create and how big are they?