EDIT: I found the reason for the page fault, and it was not a caching problem. It turned out that I tried to use the "global" flag for PD and PDP entries, where it is a reserved field and must be 0.
This problem is driving me crazy for a few days already. My 64 bit kernel code runs perfectly fine on Bochs, QEMU and VirtualBox - but it crashes on AMD SimNow!. So here's what I'm doing:
I have set up a PML4T at physical address 0x112000, so I have CR3 = 0x112000. I use a recursive mapping, so the last entry of the PML4T also points to 0x112000. (Actually it contains 0x112063 - present, r/w, dirty and accessed bits are set.) This means that the PML4T is mapped to 0xfffffffffffff000, the last page of virtual memory.
Now I would like to map a page from my kernel to virtual address 0xffff800000000000 - right at the beginning of the upper half of virtual memory. In order to do that, I need to create some lower-level paging structures for that address. So my kernel does the following (you may skip this - it's just to explain how I end up in the situation I'll describe in the next paragraph):
First, it notices that there is no page table for virtual address 0xffff800000000000. So it requests a free page from the page allocator, and gets one. This happens to be 0x0ff0f000 - somewhere close to the top of my 256MB RAM. It tries to map this page to 0xffffffc000000000 - this is where the page table for 0xffff800000000000 should appear by the recursive mapping I use. It finds that there is no page table for that address either, so it requests another free page, gets 0x0ff0e000 from the and tries to map it to 0xffffffffe0000000 - again, this address results from the recursive mapping, and it denotes both the page table for 0xffffffc000000000 and the page directory for 0xffff800000000000. Once again it turns out that there is no page table for 0xffffffffe0000000, so it requests yet another page, gets 0x0ff0d000, and tries to map it to 0xfffffffffff00000. Now finally, this mapping succeeds. The kernel adds an entry to virtual address 0xfffffffffffff800, which is mapped to physical 0x112800 - the middle of the PML4T. So finally all the other mappings should succeed and I end up with the following (this is what SimNow!'s debugger tells me as well):
[code]Virtual Physical Content
0xfffffffffffff000 => 0x00112000: 0x0000000000112063
0xfffffffffffff800 => 0x00112800: 0x000000000ff0d165
0xfffffffffff00000 => 0x0ff0d000: 0x000000000ff0e165
0xffffffffe0000000 => 0x0ff0e000: 0x000000000ff0f105[/code]
So, if you go through the page mappings, you should conclude that virtual 0xffffffc000000000 is mapped to physical 0x0ff0f000, right? And setting the first entry at 0xffffffc000000000 to some address should map this address to 0xffff800000000000? Well, on Bochs, QEMU and VirtualBox this is exactly what happens, and the mappings are exactly the same as here. (Except that the physical addresses are a bit different.) However, on SimNow! I get a page fault as soon as I try to write something to 0xffffffc000000000 - this is also the address I get reported in CR2. If I ask SimNow!'s debugger for a memory dump at virtual 0xffffffc000000000, it tells me that this page is not mapped.
So, if anybody has a clue what's going wrong, please let me know. I thought about a caching / TLB issue, so maybe I'm doing something wrong when it comes to TLB invalidation? However, I don't know how this can happen. I never access 0xffffffc000000000 before, so there should be no TLB entry. And even if there were a TLB entry, it should be cleared, because I do an "invlpg [rdx]" with rdx = 0xffffffc000000000 between the memory writes to 0xffffffffe0000000 and 0xffffffc000000000 - so that should clear the TLB entry for that address, right?
Now I'm really at wit's end. Does anybody have a suggestion what I should check next? Is there a way to dump the cache contents on AMD SimNow!? I searched the web, but I could not find any.
The paging code can be found in [url=http://xenos.svn.sourceforge.net/viewvc/xenos/trunk/kernel/arch/x86/X86Pager.cpp?view=markup]X86Pager.cpp[/url] and [url=http://xenos.svn.sourceforge.net/viewvc/xenos/trunk/kernel/arch/x86/X86Pager.h?view=markup]X86Pager.h[/url].