The RAM could be the issue. Mixing two kits can cause frame rate drops. Ryzen depends on frequencies of 3200mhz to 3600mhz to really perform well. Some would argue you can go over those frequencies but I personally do not have the money needed to get a crazy high frequency kit.
I would use the one set that supports higher frequencies. Also, you will want to ensure that you are installing the RAM in the correct slots in order to run dual channel.
If it were me, I would install one set in slot A2 and B2, look in the bios to see what the XMP profile will set the frequency to, power down and repeat for the other set. Then I would use the set that has the higher frequency profile.
Installing RAM in the correct order is also important. It determines if you are going to run in dual or single channel. There is a very large performance uplift just by running in dual channel as opposed to single channel.
I hope this helps. The layout of that board is from left to right B1 B2 A1 A2. The correct installation would be B2 and A2, so the farthest slot from the CPU and the second closest to the CPU.
Ryzen 7 7800X3D, ASUS ROG STRIX X670E-F GAMING WIF, G.SKILL TRIDENT Z NEO 2x16GB DDR5 DRAM 6000MT/s CL30, AMD Radeon RX 7900 XTX, LIAN LI EDGE 1300, Corsair MP600 PRO NH 4TB