The HD 5770 is a PCIe GPU card as mentioned by mstfbsrn980. In fact, I believe all HD 3xxx and up are PCIe GPU cards.
From what I can see is that the original XPS 8100 came with a 350 watt PSU. If this is true then you need to find a GPU card that needs a less powerful PSU to run in your computer.
Check out this website that shows all the various GPU Cards and the minimum PSU wattage need for the computer to run with the GPU installed: PSU REQUIREMENTS - RealHardTechX
There are a least two fairly modern AMD GPU cards that can run on a 350 watt PSU:
Both of the AMD GPU cards highlighted doesn't need any extra GPU Power connection cables from the PSU.
If you want to install a more powerful and modern GPU Card you will need to replace the PSU with a much higher wattage PSU with an efficiency range of one of the metals PSU (Platinum, Gold, Silver, or Bronze).
Plus you need to be sure it will fit inside your Dell XPS 8100 computer case.
EDIT: Found from Dell Support the Studio XPS 8100 PDF File that shows your motherboard has a PCI, PCIex1 and PCIex16 slots on the motherboard with a 350 Watt PSU. I have attached it to this post for your to use when you decide which GPU card you are planning to purchase in the future as a reference.