AnsweredAssumed Answered

Questions about the address translation mechanism of KFD and IOMMU

Question asked by gic4107 on Apr 26, 2015
Latest reply on May 8, 2015 by gic4107

Hi all,


I'm current working on a research project  to virtualize an HSA compliant system.  In particular, I am attempting to allow multiple guest OSes to share the resources in an HSA system. The implementation is based on KVM. My work has been blocked by one problem, and I was hoping any of you may help me out.


For this project, I need to enable IOMMU's two-stage (guest virtual -> guest physical -> machine physical) address translation. When I turned on IOMMU's two-stage address translation (I set the page table for stage 1&2 properly) and run an HSA sample program on the guest OS. However, IO_PAGE_FAULTs showed up in log and the faulting address is the same as physical address of cik_mqd created in init_mqd during KFD_IOC_CREATE_QUEUE ioctl. With some additional tracing study, surveys and discussion with AMD engineers, I found that there are two HW components to carry out address translation for GPU: GPUVM and IOMMU.


Here are my questions:

1. Is cik_mqd translated by GPUVM?


I made this guess since gart_mqd_addr is assigned to pm4_packet in the pm_create_map_queue.


2. Will GPUVM go through two-stage address translation when IOMMU is enable to do two-stage address translation?


I assume GPUVM translates cik_mqd since mqd_gart_addr is set in the pm_create_map_queue, and it ends up translating to physical address of cik_mqd. With two-stage translation enabled, GPUVM takes the physical address of cik_mqd as input of the 2nd-stage. This causes translation fault because it is already a physical address. I guess this is why I got the IO_PAGE_FAULT.


3. Can I set GPUVM to do one-stage address translation while IOMMU is doing two-stage address translation?


4. Can cik_mqd be translated by IOMMU rather than GPUVM?


Could you please give me some advices or and a pointer that I should follow to continue my work? Your help or advise will be deeply appreciated.