I got it to "work" now, the output is still wrong, but it seems to at least be visible now. I have one question though, what does AMF_REPEAT mean and why is it returned? Does it mean that it needs more data to encode? What do I do if I get AMF_REPEAT, but SubmitInput returns AMF_INPUT_FULL?
To Issue 1:
Turns out that the HPitch and VPitch of the AMF Surface is slightly larger than the actual needed area. This caused distortions and crashes in my original code. Solution: copy line by line and calculate the offset yourself.
To Issue 2:
Solved itself after Issue 1 was resolved.