Are the speeds you're watching peak or average?
These Ryzen CPU's constantly change their frequency and voltage (1000x a second!) to maximize performance up to their thermal or power limits. But different workloads affect the CPU thermals and power draws differently, so unless you are seeing the speed differences by testing with the exact same constant load in the exact same conditions (even room temperature matters!), don't worry about it.