The assert means that the cost associated with an edge between two clusters is too high. I believe if you increase the number of target clusters, this error should go away. If you are unable to make the error go away, you may need to use one of the alternative functions to optimize your mesh.
This problem occured, when I tried optimizing a bit incorrect mesh. It would be not a parameter problem sometimes. Mesh should be well formed, and in this case it would be a very complex primitive (1,000,000 works fine).