gyeong9m, most of your terms I do not recognize, suggesting that you are running Linux. I cannot help with Linux but some here can but they will require some information. Please tell us your OS and HW specifications. Here are mine as an example:
ASRock Fatal1ty X399 Pro Gaming, Threadripper 1950X, 2xSamsung SSD 960 EVO RAID, 1TB &
500 GB WD Black, G.SKILL [Flare X (for AMD)] F4-3200C14Q-32GFX, Windows 10 x64 Pro,
Enermx Platimax 850, Enermx Liqtech TR4 CPU Cooler, Radeon RX580, BIOS 2.0
Here is the policy for my SSD RAID0 NVMe and therefore PCIe:
I have run with write-cache buffer flushing both enabled and disabled with very measurable results. This may not answer your question, but I suspect Linux people will need more information. Enjoy, John.
Sorry for providing not enough information.
MTRR determines cache attribute for a physical memory space, and PTE/PAT determines cache attribute of a virtual memory space.
Cache attribute of a virtual memory space should be same with that of the physical memory space.
So we set both of MTRR and PTE/PAT by write-back cache as follow.
First, here is our physical address of target device and default MTRR state:
$ lspci -v -s 0f:00
0f:00.0 Memory controller: Xilinx Corporation Device 7022
Subsystem: Xilinx Corporation Device 0007
Flags: fast devsel
Memory at 80000000 (64-bit, prefetchable) [disabled] [size=1G]
$ cat /proc/mtrr
reg00: base=0x000000000 ( 0MB), size= 2048MB, count=1: write-back
reg01: base=0x07f000000 ( 2032MB), size= 16MB, count=1: uncachable
After running our kernel code, a new MTRR entry for our target device is inserted:
$ cat /proc/mtrr
reg00: base=0x000000000 ( 0MB), size= 2048MB, count=1: write-back
reg01: base=0x07f000000 ( 2032MB), size= 16MB, count=1: uncachable
reg02: base=0x080000000 ( 2048MB), size= 1024MB, count=0: write-back
Then, we allocate a virtual memory space to the target device and set PTE/PAT for the memory space.
Here is our kernel code:
int dummyKernelModule_init(void) { // main function of the kernel module
...
BAR_virt = pcim_iomap(pci_xilinx_dev, 0, len); // allocate virtual memory space for the target device
printk(KERN_INFO "dummyKernelModule : BAR_virt %p - %x\n", BAR_virt, len); // print address of the virtual memory space
dummyKernelModule_print_pte(BAR_virt); // print default pte of the virtual memory sapce
dummyKernelModule_PAT_cache_enable(BAR_virt); // set or clear PAT, PCD, PWT bit for pte of the virtual memory space
dummyKernelModule_print_pte(BAR_virt); // print changed pte of the virtual memory space
...
}
static inline void dummyKernelModule_PAT_cache_enable(unsigned long ptr)
{
...
dummyKernelModule_clear_flag(ptr, _PAGE_PWT); // Enable write back == Clear Write Through flag
dummyKernelModule_clear_flag(ptr, _PAGE_PCD); // Enable cache == Clear Cache Disable flag
...
}
And here is the results of dmesg after running the kernel code (blue is our comment):
[ 391.276949] dummyKernelModule : BAR_virt 00000000886e1eed - 3e8
[ 391.276953] dummyKernelModule : pte at 00000000886e1eed is 80000173. // Default PTE
[ 391.277644] dummyKernelModule : pte at 00000000886e1eed is 80000163. // Changed PTE
As shown the dmesg log, the virtual memory space is properly mapped to the our target device.
(high-order bit of PTE indicates the physical address and it is same with physical address of our target device)
And low-order bit of PTE indicates PTE attribute as follow:
(This is from AMD64 Architecture Programmer's Manual Volume 2 System Programming 7.8.2 PAT Indexing, https://developer.amd.com/wordpress/media/2012/10/24593_APM_v21.pdf):
• PAT (page attribute table)—The PAT bit is bit 7 in 4-Kbyte PTEs; it is bit 12 in 2-Mbyte and 4-Mbyte PDEs.
Page-table entries that don’t have a PAT bit (PML4 entries, for example) assume PAT = 0.
• PCD (page cache disable)—The PCD bit is bit 4 in all page-table entries.
• PWT (page writethrough)—The PWT bit is bit 3 in all page-table entries.
As shown the dmesg log, the default PAT bit is 0, PCD bit is 1, PWT bit is 0. So PCD should be cleared to enable cache.
After our kernel code clear PCD flag, the PTE is properly changed as shown the dmesg (PAT, PCD, PWT = 0, 0, 0).
As shown Table 7-9, our PTE/PAT setting is WB (Write-Back) cache.
When I get response from AMD, I'll post it.
Thank you.
