10 Replies Latest reply on Nov 27, 2017 5:22 AM by sbi

    Supermicro X9DRi-F + AMD FirePro S7150 VMs fail to start with MxGPU

    sbi

      Hi everyone,

       

      I'm trying to setup AMD FirePro S7150 MxGPU on Supermicro X9DRi-F MB with ESXi 6.5.

      I've managed to enable next features from the AMD MxGPU requirements list on my X9DRi-F MB:

       

      • Intel VT-d;
      • SR-IOV Support;
      • Above 4G Decoding (without this option system boot up stuck on "System initializing... 91");

      Also I failed to find next one:

      • ARI Forwarding;
      • MMIO High Size;
      • MMIOH Base;
      • SVM capability;
      • IOMMU capability.
      After AMD drivers setup I've managed to configure some VMs with SR-IOVed devices:

       

      Inline image 1

      Inline image 1

      Unfortunately VM fails to start with the next error message: "Module 'DevicePowerOn' power on failed."

      Here are some LOGs:

       

      VM's vmware.log

       

      2017-04-19T20:27:34.263Z| vmx| I125: PCIPassthru: 0000:83:02.3 : Device BAR 0 requested 64-bit memory address that exceeded MPN type (61708076843008).

       

      2017-04-19T20:27:34.263Z| vmx| I125: PCIPassthru: 0000:83:02.3 : Device BAR 2 requested 64-bit memory address that exceeded MPN type (61709834256384).

      2017-04-19T20:27:34.263Z| vmx| I125: Module 'DevicePowerOn' power on failed.

      2017-04-19T20:27:34.263Z| vmx| I125: VMX_PowerOn: ModuleTable_PowerOn = 0

       

      vmkernel.log during VM start

       

      2017-04-19T20:27:34.263Z cpu20:69873)VSCSI: 6532: handle 8193(vscsi0:0):Destroying Device for world 69874 (pendCom 0)

       

      2017-04-19T20:27:34.390Z cpu11:65634)WARNING: IOMMU: 2219: Unable to unset device 0000:83:02.3 since it hasn't been set.

       

       

      cat /var/log/vmkernel.log | grep amdgpuv

       

      VMB: 323:    name: /amdgpuv.v00

      VMB: 323:    name: /amdgpuv_.v00

      2017-04-19T20:38:50.732Z cpu0:65536)VisorFSTar: 1954: amdgpuv.v00 for 0x70228 bytes

      2017-04-19T20:38:50.734Z cpu0:65536)VisorFSTar: 1954: amdgpuv_.v00 for 0x507c6 bytes

      2017-04-19T20:38:53.610Z cpu4:65893)ALERT: Failed to verify signatures of the following vib(s): [amdgpuv amdgpuv-cim]. All tardisks validated

      2017-04-19T20:39:10.207Z cpu22:66086)Loading module amdgpuv ...

      2017-04-19T20:39:10.211Z cpu22:66086)Elf: 2043: module amdgpuv has license VMware

      2017-04-19T20:39:10.217Z cpu22:66086)Device: 191: Registered driver 'amdgpuv' from 24

      2017-04-19T20:39:10.217Z cpu22:66086)amdgpuv_log: amdgpuv_device_list_init:41: [amdgpuv]: amdgpuv device list initialized successfully

      2017-04-19T20:39:10.217Z cpu22:66086)amdgpuv_log: init_module:455: [amdgpuv]: amdgpuv module loaded successfully

      2017-04-19T20:39:10.217Z cpu22:66086)amdgpuv_log: init_module:457: [amdgpuv]: AMD SRIOV GPU-V Device Driver 0.9.1 BUILD: Dec 23 2016

      2017-04-19T20:39:10.217Z cpu22:66086)Mod: 4968: Initialization of amdgpuv succeeded with module ID 24.

      2017-04-19T20:39:10.217Z cpu22:66086)amdgpuv loaded successfully.

      2017-04-19T20:39:10.222Z cpu15:66023)amdgpuv_log: amdgpuv_attachDevice:678: [amdgpuv]: amdgpuv_attachDevice: attachDevice...

      2017-04-19T20:39:10.222Z cpu15:66023)amdgpuv_log: amdgpuv_attachDevice:686: [amdgpuv]: attacheDevice of amdgpuv driver invoked, moduleID = 24, heapID = 0x43060eead000, num = 0

      2017-04-19T20:39:10.222Z cpu15:66023)amdgpuv_log: amdgpuv_attachDevice:697: [amdgpuv]: vmkDev = 0x14b343052f2ab48e

      2017-04-19T20:39:10.222Z cpu15:66023)amdgpuv_log: amdgpuv_attachDevice:706: [amdgpuv]: get pf pci device handle successfully

      2017-04-19T20:39:10.222Z cpu15:66023)amdgpuv_log: amdgpuv_pci_match_device:68: [amdgpuv]: found the supported asic, deviceID = 0x6929

      2017-04-19T20:39:10.222Z cpu15:66023)amdgpuv_log: amdgpuv_manage_pci_resource:138: [amdgpuv]: reg_idx=5, doorbell_idx=2

      2017-04-19T20:39:10.222Z cpu15:66023)amdgpuv_log: amdgpuv_manage_pci_resource:185: [amdgpuv]: map mmio reg successfully, pDev->pIOReg = 0x41001d180000

      2017-04-19T20:39:10.222Z cpu15:66023)amdgpuv_log: amdgpuv_manage_pci_resource:186: [amdgpuv]: pDev->regAperSize = 0x40000

      2017-04-19T20:39:10.222Z cpu15:66023)amdgpuv_log: amdgpuv_manage_pci_resource:198: [amdgpuv]: map doorbell successfully, pDev->pDoorbell = 0x410015400000

      2017-04-19T20:39:10.222Z cpu15:66023)amdgpuv_log: amdgpuv_manage_pci_resource:199: [amdgpuv]: pDev->doorbellAperSize = 0x200000

      2017-04-19T20:39:10.224Z cpu15:66023)amdgpuv_log: amdgpuv_manage_pci_resource:210: [amdgpuv]: map FB successfully, pDev->pFramebuffer = 0x438087c00000

      2017-04-19T20:39:10.224Z cpu15:66023)amdgpuv_log: amdgpuv_manage_pci_resource:211: [amdgpuv]: pDev->fbAperSize = 0x10000000

      2017-04-19T20:39:10.224Z cpu15:66023)amdgpuv_log: KCL_Ih_Ctx_Init:63: [amdgpuv]: kcl ih ctx initialized successfully

      2017-04-19T20:39:10.224Z cpu15:66023)amdgpuv_log: amdgpuv_attachDevice:740: [amdgpuv]: register interrupt successfully

      2017-04-19T20:39:10.224Z cpu15:66023)amdgpuv_log: amdgpuv_read_VBIOS:313: [amdgpuv]: VBIOS starts:  0x55, 0xaa

      2017-04-19T20:39:10.357Z cpu15:66023)amdgpuv_log: amdgpuv_read_VBIOS:337: [amdgpuv]: BIOS Version Major 0xF Minor 0x31

      2017-04-19T20:39:10.357Z cpu15:66023)amdgpuv_log: amdgpuv_read_VBIOS:347: [amdgpuv]: Valid video BIOS image, size = 0x10000, check sum is 0x553a00

      2017-04-19T20:39:10.357Z cpu15:66023)amdgpuv_log: ATOM_PostVBIOS:189: [amdgpuv]: VBIOS post: init ASIC

      2017-04-19T20:39:10.680Z cpu15:66023)amdgpuv_log: ATOM_PostVBIOS:187: [amdgpuv]: VBIOS post: load FW

      2017-04-19T20:39:13.060Z cpu15:66023)amdgpuv_log: amdgpuv_post_VBIOS:404: [amdgpuv]: Post VBIOS successfully!

      2017-04-19T20:39:13.061Z cpu15:66023)amdgpuv_log: amdgpuv_set_thermal_cntl_enable:358: [amdgpuv]: Thermal Control Enable

      2017-04-19T20:39:13.061Z cpu15:66023)amdgpuv_log: KCL_PCI_Find_Ext_Capability:324: [amdgpuv]: found the Capability 0xb @pos 0x100

      2017-04-19T20:39:13.061Z cpu15:66023)amdgpuv_log: KCL_PCI_Find_Ext_Capability:324: [amdgpuv]: found the Capability 0xb @pos 0x400

      2017-04-19T20:39:13.061Z cpu15:66023)amdgpuv_log: amdgpuv_pci_gpuiov_init:65: [amdgpuv]: gpuiov->pos = 0x400

      2017-04-19T20:39:13.061Z cpu15:66023)amdgpuv_log: amdgpuv_pci_gpuiov_init:81: [amdgpuv]: Total FB size @pos = 0x424 is 8190 MB

      2017-04-19T20:39:13.061Z cpu15:66023)amdgpuv_log: amdgpuv_pci_gpuiov_init:101: [amdgpuv]: initialize gpu iov capability successfully

      2017-04-19T20:39:13.061Z cpu15:66023)amdgpuv_log: parse_conf_str:57: [amdgpuv]: Parse Configuration:16,B131

      2017-04-19T20:39:13.061Z cpu15:66023)amdgpuv_log: get_fb_partition:192: [amdgpuv]: maximal framebuffer size: 8176 MB

      2017-04-19T20:39:13.061Z cpu15:66023)amdgpuv_log: get_fb_partition:196: [amdgpuv]: minimal pf framebuffer size: 256 MB

      2017-04-19T20:39:13.061Z cpu15:66023)amdgpuv_log: get_fb_partition:201: [amdgpuv]: pf framebuffer size: 256 MB

      2017-04-19T20:39:13.061Z cpu15:66023)amdgpuv_log: get_fb_partition:205: [amdgpuv]: total available framebuffer size for VFs: 7904 MB

      2017-04-19T20:39:13.061Z cpu15:66023)amdgpuv_log: get_vf_mem_ap_size:224: [amdgpuv]: the STRAP_BIF_VF_MEM_AP_SIZE limits the VF FB to 128 MB

      2017-04-19T20:39:13.061Z cpu15:66023)amdgpuv_log: validate_vf_params:161: [amdgpuv]: idx=0,bus=131

      2017-04-19T20:39:13.061Z cpu15:66023)amdgpuv_log: valid_fb_size:108: [amdgpuv]: Set the FB size as upper limit 7904 by default

      2017-04-19T20:39:13.061Z cpu15:66023)amdgpuv_log: valid_sched_intv:137: [amdgpuv]: Invalid scheduler interval 0 from config

      2017-04-19T20:39:13.061Z cpu15:66023)amdgpuv_log: valid_sched_intv:143: [amdgpuv]: Calculate the scheduler interval as 5000 us by default

      2017-04-19T20:39:13.061Z cpu15:66023)amdgpuv_log: calibrate_vf_config:260: [amdgpuv]: set the default FB to 480 MB

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_enableVFs:133: [amdgpuv]: 16 VFs are enabled successfully

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_set_gpuiov_total_fb_consumed:214: [amdgpuv]: the amount of FB reserved for virtualizaion @pos = 0x426, is 7926 MB

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_set_gpuiov_context:137: [amdgpuv]: set gpuiov ctx @pos = 0x420, data = 0x100001

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_set_gpuiov_vf_frame_buffer:242: [amdgpuv]: set vf frame buffer @pos = 0x42c, size = 480, offset = 0x110

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_set_gpuiov_vf_frame_buffer:242: [amdgpuv]: set vf frame buffer @pos = 0x430, size = 480, offset = 0x2f0

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_set_gpuiov_vf_frame_buffer:242: [amdgpuv]: set vf frame buffer @pos = 0x434, size = 480, offset = 0x4d0

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_set_gpuiov_vf_frame_buffer:242: [amdgpuv]: set vf frame buffer @pos = 0x438, size = 480, offset = 0x6b0

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_set_gpuiov_vf_frame_buffer:242: [amdgpuv]: set vf frame buffer @pos = 0x43c, size = 480, offset = 0x890

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_set_gpuiov_vf_frame_buffer:242: [amdgpuv]: set vf frame buffer @pos = 0x440, size = 480, offset = 0xa70

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_set_gpuiov_vf_frame_buffer:242: [amdgpuv]: set vf frame buffer @pos = 0x444, size = 480, offset = 0xc50

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_set_gpuiov_vf_frame_buffer:242: [amdgpuv]: set vf frame buffer @pos = 0x448, size = 480, offset = 0xe30

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_set_gpuiov_vf_frame_buffer:242: [amdgpuv]: set vf frame buffer @pos = 0x44c, size = 480, offset = 0x1010

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_set_gpuiov_vf_frame_buffer:242: [amdgpuv]: set vf frame buffer @pos = 0x450, size = 480, offset = 0x11f0

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_set_gpuiov_vf_frame_buffer:242: [amdgpuv]: set vf frame buffer @pos = 0x454, size = 480, offset = 0x13d0

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_set_gpuiov_vf_frame_buffer:242: [amdgpuv]: set vf frame buffer @pos = 0x458, size = 480, offset = 0x15b0

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_set_gpuiov_vf_frame_buffer:242: [amdgpuv]: set vf frame buffer @pos = 0x45c, size = 480, offset = 0x1790

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_set_gpuiov_vf_frame_buffer:242: [amdgpuv]: set vf frame buffer @pos = 0x460, size = 480, offset = 0x1970

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_set_gpuiov_vf_frame_buffer:242: [amdgpuv]: set vf frame buffer @pos = 0x464, size = 480, offset = 0x1b50

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_set_gpuiov_vf_frame_buffer:242: [amdgpuv]: set vf frame buffer @pos = 0x468, size = 480, offset = 0x1d30

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: KCL_PCI_Find_Ext_Capability:324: [amdgpuv]: found the Capability 0x10 @pos 0x330

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_query_sriov_configuration:108: [amdgpuv]: sriov->numVFs = 16

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_query_sriov_configuration:109: [amdgpuv]: sriov->vfOffset = 0x10, sriov->vfStride = 1

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_query_sriov_configuration:110: [amdgpuv]: sriov->vfDeviceID = 0x692f

      2017-04-19T20:39:14.947Z cpu15:66023)WARNING: amdgpuv_log: amdgpuv_attachDevice:866: [amdgpuv]:Success attachDevice Done

      2017-04-19T20:39:14.947Z cpu15:66023)Device: 326: Found driver amdgpuv for device 0x14b343052f2ab48e

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_startDevice:1068: [amdgpuv]: amdgpuv_startDevice: startDevice...

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_init_device:168: [amdgpuv]: pDev->vmkDev = 0x14b343052f2ab48e

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_init_device:169: [amdgpuv]: pDev->pfPciDev = 0x43008805c030

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_init_device:170: [amdgpuv]: pDev->pIOReg = 0x41001d180000

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_init_device:171: [amdgpuv]: pDev->doorbell = 0x410015400000

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_ih_iv_ring_disable:340: [amdgpuv]: disable iv ring successfully

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_alloc_iv_ring:87: [amdgpuv]: ih->ivRingNumEntries = 256

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_alloc_iv_ring:90: [amdgpuv]: ih->ivRingSizeInBytes = 4096

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_alloc_iv_ring:94: [amdgpuv]: ih->ivRingAllocSizeInBytes = 4100

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_alloc_iv_ring:96: [amdgpuv]: iv ring page_cnt = 2

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: KCL_MEM_AllocPage:160: [amdgpuv]: Allocated one physical page, page_MA = 0x2083133000

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: KCL_MEM_AllocPage:160: [amdgpuv]: Allocated one physical page, page_MA = 0x2080f4a000

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_alloc_iv_ring:116: [amdgpuv]: ih->ivRing_MA.QuadPart = 0x2083133000

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_alloc_iv_ring:120: [amdgpuv]: ih->ivRing_VA = 0x41001c140000

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_alloc_iv_ring:122: [amdgpuv]: ih->ivRingWptrWB = 0x41001c141000

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_alloc_iv_ring:124: [amdgpuv]: ih->ivRingWptrWB_MA.QuadPart = 0x2080f4a000

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_alloc_iv_ring:135: [amdgpuv]: update rptr via doorbell

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_ih_iv_ring_init:261: [amdgpuv]: ih->rptrDoorbell = 0x4100154007a0

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_ih_iv_ring_init:262: [amdgpuv]: ih->rptrDoorbellOffset = 0x1e8

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_ih_iv_ring_hw_init:187: [amdgpuv]: the physical address of ring buffer: 0x20831330

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_ih_iv_ring_setupRPTR:399: [amdgpuv]: write mmBIF_DOORBELL_APER_EN: 0x1

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_ih_iv_ring_enable:309: [amdgpuv]: ih->ivRingWptr_Reg = 0x0

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_ih_iv_ring_enable:311: [amdgpuv]: ih->ivRingWptr = 0

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_ih_iv_ring_enable:313: [amdgpuv]: ih->ivRingRptr_Reg = 0x0

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_ih_iv_ring_enable:315: [amdgpuv]: ih->ivRingRptr = 0

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_ih_iv_ring_enable:317: [amdgpuv]: *(ih->rptrDoorbell) = 0x0

      2017-04-19T20:39:14.947Z cpu15:66023)amdgpuv_log: amdgpuv_ih_iv_ring_init:267: [amdgpuv]: init iv ring successfully

      2017-04-19T20:39:14.948Z cpu15:66023)CpuSched: 692: user latency of 66087 tq:amdgpuv_sched_timer 0 changed by 66023 vmkdevmgr -6

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_sched_init:842: [amdgpuv]: create timer queue for scheduler successfully

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_sched_init:860: [amdgpuv]: Scheduler uses 5000 us VF switching interval

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_sched_init:871: [amdgpuv]: Scheduler uses 3000 (ticks) timeout, 200 (ticks) to start hang detection, 100 (ticks) to judge GPU hang

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_sched_init:875: [amdgpuv]: Scheduler uses schema: fixed percentage

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_sched_init:884: [amdgpuv]: Scheduler uses switch method all-in-one

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_ih_irq_source_enable:520: [amdgpuv]: irq sourceID 0x89 get enabled

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_ih_irq_source_enable:520: [amdgpuv]: irq sourceID 0x88 get enabled

      2017-04-19T20:39:14.948Z cpu15:66023)WARNING: amdgpuv_log: amdgpuv_startDevice:1086: [amdgpuv]:Success startDevice Done

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_scanDevice:891: [amdgpuv]: amdgpuv_scanDevice: scanDevice...

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:366: [amdgpuv]: Get vf pci device handle @vf_idx = 0, 0x430088073c90

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:377: [amdgpuv]: register VF successfully, vf_idx = 0

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:366: [amdgpuv]: Get vf pci device handle @vf_idx = 1, 0x4300880740f0

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:377: [amdgpuv]: register VF successfully, vf_idx = 1

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:366: [amdgpuv]: Get vf pci device handle @vf_idx = 2, 0x4300880746a0

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:377: [amdgpuv]: register VF successfully, vf_idx = 2

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:366: [amdgpuv]: Get vf pci device handle @vf_idx = 3, 0x430088074c50

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:377: [amdgpuv]: register VF successfully, vf_idx = 3

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:366: [amdgpuv]: Get vf pci device handle @vf_idx = 4, 0x430088075200

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:377: [amdgpuv]: register VF successfully, vf_idx = 4

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:366: [amdgpuv]: Get vf pci device handle @vf_idx = 5, 0x4300880757b0

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:377: [amdgpuv]: register VF successfully, vf_idx = 5

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:366: [amdgpuv]: Get vf pci device handle @vf_idx = 6, 0x430088075d60

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:377: [amdgpuv]: register VF successfully, vf_idx = 6

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:366: [amdgpuv]: Get vf pci device handle @vf_idx = 7, 0x430088076310

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:377: [amdgpuv]: register VF successfully, vf_idx = 7

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:366: [amdgpuv]: Get vf pci device handle @vf_idx = 8, 0x4300880768c0

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:377: [amdgpuv]: register VF successfully, vf_idx = 8

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:366: [amdgpuv]: Get vf pci device handle @vf_idx = 9, 0x430088076e70

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:377: [amdgpuv]: register VF successfully, vf_idx = 9

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:366: [amdgpuv]: Get vf pci device handle @vf_idx = 10, 0x430088077420

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:377: [amdgpuv]: register VF successfully, vf_idx = 10

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:366: [amdgpuv]: Get vf pci device handle @vf_idx = 11, 0x4300880779d0

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:377: [amdgpuv]: register VF successfully, vf_idx = 11

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:366: [amdgpuv]: Get vf pci device handle @vf_idx = 12, 0x430088077f80

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:377: [amdgpuv]: register VF successfully, vf_idx = 12

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:366: [amdgpuv]: Get vf pci device handle @vf_idx = 13, 0x430088078530

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:377: [amdgpuv]: register VF successfully, vf_idx = 13

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:366: [amdgpuv]: Get vf pci device handle @vf_idx = 14, 0x430088078ae0

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:377: [amdgpuv]: register VF successfully, vf_idx = 14

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:366: [amdgpuv]: Get vf pci device handle @vf_idx = 15, 0x430088079090

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_registerVFs:377: [amdgpuv]: register VF successfully, vf_idx = 15

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_scanDevice:951: [amdgpuv]: Register logical character device successfully

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_init_idh_queue:198: [amdgpuv]: Initialize a new idh queue

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_init_idh_queue:198: [amdgpuv]: Initialize a new idh queue

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_init_idh_queue:198: [amdgpuv]: Initialize a new idh queue

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_init_idh_queue:198: [amdgpuv]: Initialize a new idh queue

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_init_idh_queue:198: [amdgpuv]: Initialize a new idh queue

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_init_idh_queue:198: [amdgpuv]: Initialize a new idh queue

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_init_idh_queue:198: [amdgpuv]: Initialize a new idh queue

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_init_idh_queue:198: [amdgpuv]: Initialize a new idh queue

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_init_idh_queue:198: [amdgpuv]: Initialize a new idh queue

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_init_idh_queue:198: [amdgpuv]: Initialize a new idh queue

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_init_idh_queue:198: [amdgpuv]: Initialize a new idh queue

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_init_idh_queue:198: [amdgpuv]: Initialize a new idh queue

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_init_idh_queue:198: [amdgpuv]: Initialize a new idh queue

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_init_idh_queue:198: [amdgpuv]: Initialize a new idh queue

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_init_idh_queue:198: [amdgpuv]: Initialize a new idh queue

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_init_idh_queue:198: [amdgpuv]: Initialize a new idh queue

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_idh_init:1894: [amdgpuv]: pending signal thread untill it get invoked by interrupt

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: KCL_Thread_Start:149: [amdgpuv]: amdgpuv kernel thread 66088 start successfully

      2017-04-19T20:39:14.948Z cpu8:66088)amdgpuv_log: amdgpuv_signal_scheduler:1620: [amdgpuv]: signal thread get invoked

      2017-04-19T20:39:14.948Z cpu8:66088)amdgpuv_log: amdgpuv_signal_scheduler:1656: [amdgpuv]: get an IDH_PENDING event

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_bind_vf:246: [amdgpuv]: add a node to vf list, list->node_cnt = 1

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_bind_vf:246: [amdgpuv]: add a node to vf list, list->node_cnt = 2

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_bind_vf:246: [amdgpuv]: add a node to vf list, list->node_cnt = 3

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_bind_vf:246: [amdgpuv]: add a node to vf list, list->node_cnt = 4

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_bind_vf:246: [amdgpuv]: add a node to vf list, list->node_cnt = 5

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_bind_vf:246: [amdgpuv]: add a node to vf list, list->node_cnt = 6

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_bind_vf:246: [amdgpuv]: add a node to vf list, list->node_cnt = 7

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_bind_vf:246: [amdgpuv]: add a node to vf list, list->node_cnt = 8

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_bind_vf:246: [amdgpuv]: add a node to vf list, list->node_cnt = 9

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_bind_vf:246: [amdgpuv]: add a node to vf list, list->node_cnt = 10

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_bind_vf:246: [amdgpuv]: add a node to vf list, list->node_cnt = 11

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_bind_vf:246: [amdgpuv]: add a node to vf list, list->node_cnt = 12

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_bind_vf:246: [amdgpuv]: add a node to vf list, list->node_cnt = 13

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_bind_vf:246: [amdgpuv]: add a node to vf list, list->node_cnt = 14

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_bind_vf:246: [amdgpuv]: add a node to vf list, list->node_cnt = 15

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: amdgpuv_bind_vf:246: [amdgpuv]: add a node to vf list, list->node_cnt = 16

      2017-04-19T20:39:14.948Z cpu15:66023)amdgpuv_log: init_vf:928: [amdgpuv]: INIT_GPU done on PF0

      2017-04-19T20:39:14.951Z cpu15:66023)amdgpuv_log: SearchMcfg:473: [amdgpuv]: MCFG is found at system BIOS 60000000

      2017-04-19T20:39:14.951Z cpu15:66023)amdgpuv_log: SearchP2PBridge:578: [amdgpuv]: Search P2P bridge for bus 83

      2017-04-19T20:39:14.957Z cpu15:66023)amdgpuv_log: TraverseBus:515: [amdgpuv]: P2P bridge depth 0: vender 8086 device e08 BDF 80:03.0 CFG 68018000

      2017-04-19T20:39:14.957Z cpu15:66023)amdgpuv_log: SearchP2PBridge:599: [amdgpuv]: P2P bridge found, dpeth 1

      2017-04-19T20:39:14.957Z cpu15:66023)amdgpuv_log: SearchP2PBridge:633: [amdgpuv]: downstream P2P cfg 68018000

      2017-04-19T20:39:14.957Z cpu15:66023)amdgpuv_log: SearchP2PBridge:638: [amdgpuv]: mask unrecoverable error at P2P cfg 68018000

      2017-04-19T20:39:14.957Z cpu15:66023)amdgpuv_log: amdgpuv_mask_unrecoverable_errors:866: [amdgpuv]: cfg sapce mapping 68018000

      2017-04-19T20:39:14.957Z cpu15:66023)amdgpuv_log: amdgpuv_mask_unrecoverable_errors:881: [amdgpuv]: ext cap pos 100

      2017-04-19T20:39:14.957Z cpu15:66023)amdgpuv_log: amdgpuv_mask_unrecoverable_errors:881: [amdgpuv]: ext cap pos 110

      2017-04-19T20:39:14.957Z cpu15:66023)amdgpuv_log: amdgpuv_mask_unrecoverable_errors:881: [amdgpuv]: ext cap pos 148

      2017-04-19T20:39:14.957Z cpu15:66023)amdgpuv_log: amdgpuv_mask_unrecoverable_errors:886: [amdgpuv]: Advance error reporting found: 148

      2017-04-19T20:39:14.957Z cpu15:66023)amdgpuv_log: SearchP2PBridge:644: [amdgpuv]: mask unrecoverable error at PF cfg 68300000

      2017-04-19T20:39:14.957Z cpu15:66023)amdgpuv_log: amdgpuv_mask_unrecoverable_errors:866: [amdgpuv]: cfg sapce mapping 68300000

      2017-04-19T20:39:14.957Z cpu15:66023)amdgpuv_log: amdgpuv_mask_unrecoverable_errors:881: [amdgpuv]: ext cap pos 100

      2017-04-19T20:39:14.957Z cpu15:66023)amdgpuv_log: amdgpuv_mask_unrecoverable_errors:881: [amdgpuv]: ext cap pos 150

      2017-04-19T20:39:14.957Z cpu15:66023)amdgpuv_log: amdgpuv_mask_unrecoverable_errors:886: [amdgpuv]: Advance error reporting found: 150

      2017-04-19T20:39:14.957Z cpu15:66023)amdgpuv_log: amdgpuv_check_sclk_dpm:470: [amdgpuv]: SCLK DPM ARG = 0x0,ID=0x162, SCLK DPM disabled

      2017-04-19T20:39:14.957Z cpu15:66023)WARNING: amdgpuv_log: amdgpuv_scanDevice:965: [amdgpuv]:Success scanDevice Done