Please can you tell us the specification of the whole rig if you can and not only the cpu.
Which is you budget, how much you can spend?
Which resolution you would like to play, which resolution is your actual screen.
Which games you play?
To answer directly to your question, yes you should be able to install a GTX1030, i would maybe advice you to get a GTX1050 at least.
Regarding the AMD dual graphic, it work using both the AMD gpu inside the cpu and an AMD discrete gpu on the pci-express, in a crossfire fashion.
But dunno if the last AMD RX550/560 support this kind of dual graphic.
In any case either a GTX1030/RX550, GTX1050/RX560 are a good choice if you are on low budget but still want to play decently at 720p/1080p.
You can also check old second hand gpu like a RX270/370, RX280/RX380 or even a HD7970 are good candidates for decent 720p/1080p gaming.
No you can't use an Nvidia GPU and and AMD GPU together in dual graphics. If you use a GT 1030, you'll only be using the A8-7600 as the CPU and not the IGP. GPUs which officially support dual graphics are actually displayed in the side of the box your A8-7600 came in. Also, I think you can use an r7 250 in dual graphics with the A8-7600, I'm not sure tho. Any GPU newer than that most likely isn't gonna support dual graphics. As far as the best graphics card for gaming on the A8 goes, I'd recommend a GTX 1050, not the gtx 1050ti or anything higher as your cpu will bottleneck it. Anything lower than a 1050 wouldn't really give you good performance in modern games at 1080p. An RX 460 would be fine too but the 1050 is a much better card at similar price.
first of all you should get a 1050ti or rx 460 (or rx 560) or maybe rx 550 - way better performance and:
DualGraphics still have the problem of micro-stutter - i am very sensitive to that and it really bothers me -> better test yourself before using it