I am pondering how I am going to invert a matrix that isn't going to fit in GPU memory. The matrix size is about 23000 currently but is expected to grow significantly over time.

I tried looking a matrix block decomposition, but accuracy suffers greatly (Matlab simulation).

Is this perhaps one problem not suitable for the GPU due to memory limitations?

From what I remember in linear algebra, matrix inversion is just about the worst thing you can do from a numerical stability standpoint. What is the condition number of your matrix? Blocked LU with partial pivoting is fairly stable and is commonly used to solve linear systems. Furthermore, it's one of the best things you can do on a GPU as you can acheive a significant fraction of the GPU's peak performance (since most of your time is spent in matrix multiplications and symmetric rank-k updates). People have even looked at out-of-core solvers (though, I can't find the paper at the moment) running on multiple-GPUs. So, in short, no, this problem is the epitome of what should be done using a GPU.