Basically we switched the viewport renderer to use an OpenGL interop, and thus give more interactivity by keeping the rendered image native on the graphics card. See this video for a before/after.