3 Replies Latest reply on Feb 20, 2018 6:53 AM by bmargos

    S7150 with Xenserver 7.2 Pass-through ok MxGPU with error

    bmargos

      We installed Xenserver 7.2 with S7150.

      Works fine when Pass-through whole GPU Tonga XT GL. (Performance nearly like a GTX780).

       

      But with MxGPU we get:

      "Failed","Starting VM 'Windows 10.1'

      Internal error: Subprocess exited with unexpected code 1; stdout = [  ]; stderr = [ modprobe: ERROR: could not insert 'gim': Operation not permitted ]

      Time: 00:00:02","xen-graf17","Nov 17, 2017 1:45 PM"

       

      Intel GVT-g virtual GPU works with this VM.

        • Re: S7150 with Xenserver 7.2 Pass-through ok MxGPU with error
          martijnk79

          Exact same issue, XenServer 7.2 latest updates. When I go to shell and do modprobe gim I get

           

          <code>

          [root@xenserver01 log]# modprobe gim

          modprobe: ERROR: could not insert 'gim': Operation not permitted

          </code>

           

          <code>

          [  734.300134] gim info:(gim_init:199) *******AMD GIM init

          [  734.300137] gim info:(gim_init:202) GPU IOV MODULE - version 1.0.0-k

          [  734.300138] gim info:(gim_init:204) Copyright (c) 2014-2016 AMD Corporation.

          [  734.300152] gim info:(parse_config_file:289) AMD GIM fb_option = 0

          [  734.300154] gim info:(parse_config_file:289) AMD GIM sched_option = 0

          [  734.300155] gim info:(parse_config_file:289) AMD GIM vf_num = 0

          [  734.300156] gim info:(parse_config_file:289) AMD GIM pf_fb = 0

          [  734.300177] gim info:(parse_config_file:289) AMD GIM vf_fb = 0

          [  734.300178] gim info:(parse_config_file:289) AMD GIM sched_interval = 0

          [  734.300180] gim info:(parse_config_file:289) AMD GIM fb_clear = 1

          [  734.300181] gim info:(parse_config_file:289) AMD GIM hang_detect_timeout = 100

          [  734.300183] gim info:(parse_config_file:289) AMD GIM max_quanta = 1000

          [  734.300184] gim info:(parse_config_file:289) AMD GIM self_switch = 500

          [  734.300202] gim info:(parse_config_file:289) AMD GIM exclusive = 500

          [  734.300203] gim info:(parse_config_file:289) AMD GIM fair_scheduling = 0

          [  734.300205] gim info:(parse_config_file:289) AMD GIM debug_level = 3

          [  734.300206] gim info:(parse_config_file:289) AMD GIM clear_fb_on_flr = 0

          [  734.300208] gim info:(init_config:434) INIT CONFIG

          [  734.300639] gim error:(BdfToAdapter:184) WARNING: bdf 0x400 cannot be found on any adapter

          [  734.300650] gim info:(alloc_adapter:394) allocate adapter for PF 0x0400

          [  734.300651] gim info:(alloc_adapter:397) Found free adapter at index 0

          [  734.300658] gim info:(SetNewAdapter:941) curr allocated at ffffffffa0904ee0

          [  734.300659] gim info:(SetNewAdapter:947) Can't disable ATS --> Not enabled in the first place

          [  734.300661] gim info:(SetNewAdapter:958) SRIOV is supported

          [  734.300662] gim info:(SetNewAdapter:966) found PCI bridge device

          [  734.300664] gim info:(SetNewAdapter:969) found: 00:3.2

          [  734.300734] gim info:(SetNewAdapter:990) mmio_base = ffffc90041880000

          [  734.300957] gim info:(SetNewAdapter:992) doorbell = ffffc90044400000

          [  734.320618] gim info:(SetNewAdapter:994) pf.fb_va = ffffc90160000000

          [  734.320632] gim info:(sriov_is_ari_enabled:158) PCI_SRIOV_CAP = 0x00000002

          [  734.320633] gim info:(sriov_is_ari_enabled:168) PCI_SRIOV_CTRL = 0x00000010

          [  734.320634] gim info:(sriov_is_ari_enabled:172) PCI_SRIOV_CTRL_ARI is set --> ARI is supported

          [  734.320636] gim info:(program_ari_mode:797) Read bif_strap8 = 0x00000004

          [  734.320637] gim info:(program_ari_mode:803) program_ari_mode - Set ARI_Mode = PF_BUS

          [  734.320638] gim info:(program_ari_mode:818) Write bif_strap8 = 0x00000004

          [  734.320639] gim info:(gim_read_rom_from_reg:483) Reading VBios from ROM

          [  734.320787] gim info:(gim_read_VBIOS:543) VBIOS starts:  0x55, 0xaa

          [  734.320788] gim info:(gim_read_VBIOS:546) VBios size is 0x10000

          [  734.320792] gim info:(gim_read_VBIOS:555) pVBIOS allocated at ffff880854300000

          [  734.320793] gim info:(gim_read_rom_from_reg:483) Reading VBios from ROM

          [  735.729979] gim info:(gim_read_VBIOS:565) BIOS Version Major 0xF Minor 0x31

          [  735.730012] gim info:(gim_read_VBIOS:578) Valid video BIOS image, size = 0x10000, check sum is 0x537f00

          [  735.730013] gim info:(gim_read_VBIOS:579) Read in full Vbios image of size = 0x80000

          [  735.730015] gim info:(SetNewAdapter:1097) Scheduler Time interval is per-vf from XL config file

          [  735.730016] gim info:(EnableSriov:398) Enable SRIOV

          [  735.730017] gim info:(EnableSriov:399) Enable SRIOV vfs count = 16

          [  735.730090] pciback 0000:04:00.0: not enough MMIO resources for SR-IOV

          [  735.730099] gim error:(EnableSriov:410) Fail to enable sriov, status = fffffff4

          [  735.730104] gim error:(SetNewAdapter:1108) Failed to properly enable SRIOV

          [  735.730130] gim error:(gim_probe:131) Failed to create new adapter

          [  735.730133] gim error:(gim_init:216) gim_init failed

          </code>

          • Re: S7150 with Xenserver 7.2 Pass-through ok MxGPU with error
            martijnk79

            Hi,

             

            I found the solution you need to enable SR-IOV in the system bios of your XenServer (run latest bios update first). I tested and this works for me.


            See this article: Implementing SR-IOV on XenServer | Citrix Blogs

             

            Hope this helps.

            • Re: S7150 with Xenserver 7.2 Pass-through ok MxGPU with error
              bmargos

              Server-Hardware (DEll R730) works fine with up to 16 VMs per Card.