a couple of good articles that may help:
This is a good basic video explaining things:
Ok, great, this helps quite a bit. Now when overclocking the GPU core clock speed, do I always have to increase the voltage as well? This is something I really want to avoid since I only have a 500 watt power supply.
Well when it comes to overclocking it really depends on how stable your system is at default settings. There is no, one size fits all approach. I don't water cool either but I do have some decent overclocks on my cpu. I don't typically OC my memory ever, other that the XMP profile, which is kind of an official supported overclock. I tend to like to not increase voltage to OC. I push the speeds as far as I can while staying stable. I also keep a cpu a long time and have been known to go ahead and up the voltage and speed even more in the final couple years of use. Just know that all OC'ing will reduce the overall life of a CPU but done lightly will probably not cause an issue during it's lifetime with you. Not knowing all your specifics, a 500 watt PS is a pretty low power PS in this day. If it is of high quality, say gold+ and meets your needs may be fine. For instance though if it is say a minimum, then you OC components, it may quickly become the non-stable part of your equation.
Well here are my PC specs:
Windows 10/64 bit
RX 470 4 gb
16 gb ram
My motherboard is kind of old. It's an HP PC from about 2013. I'm not sure if this is a factor at all. In any case, I'm planning on buying a new PC next year anyway because its starting to show its age in games like Hitmam 2. It struggles to maintain 60 fps even though I'm overclocking, dipping into the 40's and even 30's range on occasion. This is even on certain parts where it did a much better job maintaining 60 frames in Hitman 2016.
Not sure if your telling my you specs at this point for a reason. Is there a question at this point or were you just giving the specs for reference?