2 Replies Latest reply on May 8, 2015 11:28 AM by gic4107

    Questions about the address translation mechanism of KFD and IOMMU

    gic4107

      Hi all,

       

      I'm current working on a research project  to virtualize an HSA compliant system.  In particular, I am attempting to allow multiple guest OSes to share the resources in an HSA system. The implementation is based on KVM. My work has been blocked by one problem, and I was hoping any of you may help me out.

       

      For this project, I need to enable IOMMU's two-stage (guest virtual -> guest physical -> machine physical) address translation. When I turned on IOMMU's two-stage address translation (I set the page table for stage 1&2 properly) and run an HSA sample program on the guest OS. However, IO_PAGE_FAULTs showed up in log and the faulting address is the same as physical address of cik_mqd created in init_mqd during KFD_IOC_CREATE_QUEUE ioctl. With some additional tracing study, surveys and discussion with AMD engineers, I found that there are two HW components to carry out address translation for GPU: GPUVM and IOMMU.

       

      Here are my questions:

      1. Is cik_mqd translated by GPUVM?

       

      I made this guess since gart_mqd_addr is assigned to pm4_packet in the pm_create_map_queue.

       

      2. Will GPUVM go through two-stage address translation when IOMMU is enable to do two-stage address translation?

       

      I assume GPUVM translates cik_mqd since mqd_gart_addr is set in the pm_create_map_queue, and it ends up translating to physical address of cik_mqd. With two-stage translation enabled, GPUVM takes the physical address of cik_mqd as input of the 2nd-stage. This causes translation fault because it is already a physical address. I guess this is why I got the IO_PAGE_FAULT.

       

      3. Can I set GPUVM to do one-stage address translation while IOMMU is doing two-stage address translation?

       

      4. Can cik_mqd be translated by IOMMU rather than GPUVM?

       

      Could you please give me some advices or and a pointer that I should follow to continue my work? Your help or advise will be deeply appreciated.

       

      Thanks!

        • Re: Questions about the address translation mechanism of KFD and IOMMU
          bridgman

          >>1. Is cik_mqd translated by GPUVM?

           

          Yes, all kernel memory allocations are accessed via GPUVM rather than via ATC/IOMMUv2.

           

          >>2. Will GPUVM go through two-stage address translation when IOMMU is enable to do two-stage address translation?

           

          Don't know but will find out. Behaviour might also be different between APU and dGPU here - my guess is that on APUs GPUVM accesses would not get translated by 2nd stage IOMMU but that's just a guess at the moment.

           

          >>3. Can I set GPUVM to do one-stage address translation while IOMMU is doing two-stage address translation?

           

          That is actually how I expected it to work, so file this under "don't know but will find out" as well.

           

          >>4. Can cik_mqd be translated by IOMMU rather than GPUVM?

           

          Let's go with "no" for now. IIRC we are only using IOMMUv2 for access to userspace memory at the moment to make sure we don't open security holes. Will focus on Q2 & Q3 for now.