Catalyst 19.9.3 + Radeon RX 570.
As the title said. It' simpossible to force AA and AF via Catalyst Settings for Vulkan applications.
In OpenGL mode AA and AF work.
You can try with Quake 3 Kenny Edition (self build with MSVC 2019):
GitHub - kennyalive/Quake-III-Arena-Kenny-Edition: The cradle of Quake III Arena