I did more test by using my Windows Overclock test partition.
Overclocking and increasing the vcore from the AMD Master tool work fine, I can achieve a stable overclock to 4Ghz at 1.26ishV (not bad for a R7 1700).
When I do set any vcore offset in the BIOS, back on Windows in hwinfo64, I get reading of a fix 1.6Ghz CPU clock and a fix 1.0V vcore.
Can it be possible that the Ryzen have a default fail safe mode when the board send him invalid parameter?
I think my CPU default voltage is too low for that el cheapo board (better binning because of segfault RMA = lower operating voltage). and the BIOS vcore offset ask something impossible to the CPU.
Now that I was able to OC the CPU properly on Windows, I can conclude that it's, again, another beautiful Board/BIOS issue.
It looks like a known 15.5 multi bug which is quite common across different mobos. Your speed is exactly 1.55GHz if you check it.
New chips have lower VID and it seems exhibit this problem more often than old chips with higher VID.
Most likely BIOS issue - asus fixed it in latests BIOSes for C6H (partially - works in windows, does not work in linux). But p-state OC works and additionaly downclocks in idle.
Yes, it's 1.55Ghz.
Someone made a custom bios for my board that supposedly add p-states OC. I will look into that, thanks.