Because it works with your TV I don't believe you're doing anything wrong. Although it's ideal to use native connections and cables your setup is fairly normal.
The only thing I can think of is you have a flaky active adapter. Can you replace it? Can you exchange it for a DVI-HDMI active adapter and use native HDMI to HDMI for one of the monitors?
Sorry, that's all I got.
ASUS TUF X570 wifi, AMD Ryzen 5 5600G, ASRock 6700XT Challenger, Antec Truepower new 700w. 32GB