It is an odd problem. I can't see how the AMD Graphics drivers will slow down your computer downloading speed. Normally your downloading speed is determined by your Network Driver plus the type of Internet connection you have.
How many people in your household is using the same Internet Router/Modem as you? What type of Internet Connection do you have (WiFi, LAN, Ethernet, or Cable)? Is your Desktop Network Driver the latest?
Trust me it was the last thing I expected as well.
To answer your questions, 4, WiFi, yes. It's not a problem with the router or a lack of bandwidth due to other machines. Other machines in the house get the roughly 40 down, and this one does as well provided the AMD Display Drivers are disabled.