5 Replies Latest reply on May 19, 2012 10:28 AM by yurtesen

    OpenCL crash multiple devices after multiple kernel enqueues.

    yurtesen

      I am trying to run a multi-device program on HD 5870 and 7970 devices.

       

      I tried it with single context multiple devices/queues, and multiple context a device/queue per context. Also both on Ubuntu 11 with kernel 3.0.x and Fedora 16 with kernel 3.3.x

       

      The program divides the problem into small pieces, if global size was 1million, it divides it to 50k pieces and runs 0-50k, 50k-100k ... so instead of a single kernel enqueue between 1 and 1million, I do enqueue offset 0 size 50k, offset 50k, size 50k and so on... After several kernel enqueues, this error appears in dmesg and program freezes and I have to reboot the box...

       

      The single context version works correctly on Nvidia with 2x Tesla cards.

       

      Multiple context version does run on Nvidia platform but kernels just do not execute for some reason. (there might be a bug in my multi-context version, I am trying to debug it yet). But I guess I shouldnt have to make a multi-context program to use 2 GPGPUs...

       

      Any ideas on how can I debug this further? (I use latest 12.4 catalyst and 2.6 app sdk)

       

      [ 5545.593322] [fglrx] ASIC hang happened

      [ 5545.593331] Pid: 5164, comm: reference Tainted: P           O 3.3.2-6.fc16.x86_64 #1

      [ 5545.593337] Call Trace:

      [ 5545.593420]  [<ffffffffa010bf9e>] KCL_DEBUG_OsDump+0xe/0x10 [fglrx]

      [ 5545.593475]  [<ffffffffa01195cc>] firegl_hardwareHangRecovery+0x1c/0x50 [fglrx]

      [ 5545.593584]  [<ffffffffa01b51d9>] ? _ZN4Asic9WaitUntil15ResetASICIfHungEv+0x9/0x10 [fglrx]

      [ 5545.593691]  [<ffffffffa01b517c>] ? _ZN4Asic9WaitUntil15WaitForCompleteEv+0x9c/0xf0 [fglrx]

      [ 5545.593798]  [<ffffffffa01aef41>] ? _ZN4Asic19PM4ElapsedTimeStampEj14_LARGE_INTEGER12_QS_CP_RING_+0x141/0x160 [fglrx]

      [ 5545.593860]  [<ffffffffa01372a2>] ? firegl_trace+0x72/0x1e0 [fglrx]

      [ 5545.593921]  [<ffffffffa01372a2>] ? firegl_trace+0x72/0x1e0 [fglrx]

      [ 5545.593983]  [<ffffffffa01372a2>] ? firegl_trace+0x72/0x1e0 [fglrx]

      [ 5545.594129]  [<ffffffffa01a5ce3>] ? _ZN15QS_PRIVATE_CORE27multiVpuPM4ElapsedTimeStampEj14_LARGE_INTEGER12_QS_CP_RING_+0x33/0x50 [fglrx]

      [ 5545.594240]  [<ffffffffa019e4c4>] ? _Z19uQSTimeStampRetiredmjj14_LARGE_INTEGER+0x74/0x80 [fglrx]

      [ 5545.594348]  [<ffffffffa019a34d>] ? _Z8uCWDDEQCmjjPvjS_+0x54d/0x10c0 [fglrx]

      [ 5545.594367]  [<ffffffff8107fcee>] ? down+0x2e/0x50

      [ 5545.594438]  [<ffffffffa0139baf>] ? firegl_cmmqs_CWDDE_32+0x36f/0x480 [fglrx]

      [ 5545.594507]  [<ffffffffa013831e>] ? firegl_cmmqs_CWDDE32+0x6e/0x100 [fglrx]

      [ 5545.594523]  [<ffffffff812692c8>] ? security_capable+0x18/0x20

      [ 5545.594592]  [<ffffffffa01382b0>] ? firegl_cmmqs_createdriver+0x170/0x170 [fglrx]

      [ 5545.594652]  [<ffffffffa01152ed>] ? firegl_ioctl+0x1ed/0x250 [fglrx]

      [ 5545.594668]  [<ffffffff815f7c60>] ? do_page_fault+0x200/0x4f0

      [ 5545.594720]  [<ffffffffa01059ae>] ? ip_firegl_unlocked_ioctl+0xe/0x20 [fglrx]

      [ 5545.594735]  [<ffffffff81193228>] ? do_vfs_ioctl+0x98/0x550

      [ 5545.594750]  [<ffffffff81193771>] ? sys_ioctl+0x91/0xa0

      [ 5545.594764]  [<ffffffff815fc1e9>] ? system_call_fastpath+0x16/0x1b

      [ 5545.594778] pubdev:0xffffffffa03b3320, num of device:2 , name:fglrx, major 8, minor 96.

      [ 5545.594789] device 0 : 0xffff8804361ec000 .

      [ 5545.594798] Asic ID:0x6898, revision:0x2, MMIOReg:0xffffc90012c00000.

      [ 5545.594808] FB phys addr: 0xc0000000, MC :0xf00000000, Total FB size :0x80000000.

      [ 5545.594819] gart table MC:0xf0f8fd000, Physical:0xcf8fd000, size:0x402000.

      [ 5545.594829] mc_node :FB, total 1 zones

      [ 5545.594837]     MC start:0xf00000000, Physical:0xc0000000, size:0xfd00000.

      [ 5545.594848]     Mapped heap -- Offset:0x0, size:0xf8fd000, reference count:21, mapping count:0,

      [ 5545.594859]     Mapped heap -- Offset:0x0, size:0x1000000, reference count:1, mapping count:0,

      [ 5545.594869]     Mapped heap -- Offset:0xf8fd000, size:0x403000, reference count:1, mapping count:0,

      [ 5545.594880] mc_node :INV_FB, total 1 zones

      [ 5545.594891]     MC start:0xf0fd00000, Physical:0xcfd00000, size:0x70300000.

      [ 5545.594901]     Mapped heap -- Offset:0x702f4000, size:0xc000, reference count:1, mapping count:0,

      [ 5545.594912] mc_node :GART_USWC, total 3 zones

      [ 5545.594920]     MC start:0x40100000, Physical:0x0, size:0x50000000.

      [ 5545.594931]     Mapped heap -- Offset:0x0, size:0x2000000, reference count:17, mapping count:0,

      [ 5545.594941] mc_node :GART_CACHEABLE, total 3 zones

      [ 5545.594950]     MC start:0x10400000, Physical:0x0, size:0x2fd00000.

      [ 5545.594960]     Mapped heap -- Offset:0x1400000, size:0x100000, reference count:2, mapping count:0,

      [ 5545.594972]     Mapped heap -- Offset:0x2600000, size:0x900000, reference count:2, mapping count:0,

      [ 5545.594982]     Mapped heap -- Offset:0xb00000, size:0x900000, reference count:2, mapping count:0,

      [ 5545.594994]     Mapped heap -- Offset:0x200000, size:0x900000, reference count:3, mapping count:0,

      [ 5545.595014]     Mapped heap -- Offset:0x0, size:0x200000, reference count:4, mapping count:0,

      [ 5545.595026]     Mapped heap -- Offset:0xef000, size:0x11000, reference count:1, mapping count:0,

      [ 5545.595038] GRBM : 0xa0635028, SRBM : 0x200000c0 .

      [ 5545.595049] CP_RB_BASE : 0x401000, CP_RB_RPTR : 0x9a70 , CP_RB_WPTR :0x9a70.

      [ 5545.595061] CP_IB1_BUFSZ:0x0, CP_IB1_BASE_HI:0x0, CP_IB1_BASE_LO:0x4023b000.

      [ 5545.595071] last submit IB buffer -- MC :0x4023b000,phys:0x422fb3000.

      [ 5545.595086] device 1 : 0xffff880436134000 .

      [ 5545.595095] Asic ID:0x6798, revision:0x5, MMIOReg:0xffffc90012c80000.

      [ 5545.595104] FB phys addr: 0xb0000000, MC :0xf400000000, Total FB size :0xc0000000.

      [ 5545.595114] gart table MC:0xf40f8fd000, Physical:0xbf8fd000, size:0x402000.

      [ 5545.595124] mc_node :FB, total 1 zones

      [ 5545.595133]     MC start:0xf400000000, Physical:0xb0000000, size:0xfd00000.

      [ 5545.595143]     Mapped heap -- Offset:0x0, size:0xf8fd000, reference count:21, mapping count:0,

      [ 5545.595154]     Mapped heap -- Offset:0x0, size:0x1000000, reference count:1, mapping count:0,

      [ 5545.595164]     Mapped heap -- Offset:0xf8fd000, size:0x403000, reference count:1, mapping count:0,

      [ 5545.595174] mc_node :INV_FB, total 1 zones

      [ 5545.595183]     MC start:0xf40fd00000, Physical:0xbfd00000, size:0xb0300000.

      [ 5545.595193]     Mapped heap -- Offset:0xb02f4000, size:0xc000, reference count:1, mapping count:0,

      [ 5545.595203] mc_node :GART_USWC, total 3 zones

      [ 5545.595212]     MC start:0xffa0100000, Physical:0x0, size:0x50000000.

      [ 5545.595222]     Mapped heap -- Offset:0x0, size:0x2000000, reference count:16, mapping count:0,

      [ 5545.595232] mc_node :GART_CACHEABLE, total 3 zones

      [ 5545.595241]     MC start:0xff70400000, Physical:0x0, size:0x2fd00000.

      [ 5545.595251]     Mapped heap -- Offset:0x2600000, size:0x900000, reference count:2, mapping count:0,

      [ 5545.595262]     Mapped heap -- Offset:0x1d00000, size:0x900000, reference count:3, mapping count:0,

      [ 5545.595273]     Mapped heap -- Offset:0xb00000, size:0x900000, reference count:2, mapping count:0,

      [ 5545.595284]     Mapped heap -- Offset:0x200000, size:0x900000, reference count:3, mapping count:0,

      [ 5545.595294]     Mapped heap -- Offset:0x0, size:0x200000, reference count:4, mapping count:0,

      [ 5545.595304]     Mapped heap -- Offset:0xef000, size:0x11000, reference count:1, mapping count:0,

      [ 5545.595316] GRBM : 0xa04c7028, SRBM : 0x200000c0 .

      [ 5545.595327] CP_RB_BASE : 0xffa01000, CP_RB_RPTR : 0x5180 , CP_RB_WPTR :0x5180.

      [ 5545.595338] CP_IB1_BUFSZ:0x0, CP_IB1_BASE_HI:0xff, CP_IB1_BASE_LO:0xa054e000.

      [ 5545.595348] last submit IB buffer -- MC :0xffa054e000,phys:0x420590000.

      [ 5545.595363] Dump the trace queue.

      [ 5545.595371] End of dump

      [ 5579.462541] [fglrx] ASIC hang happened

      [ 5579.462550] Pid: 5163, comm: reference Tainted: P           O 3.3.2-6.fc16.x86_64 #1

      [ 5579.462556] Call Trace:

      [ 5579.462639]  [<ffffffffa010bf9e>] KCL_DEBUG_OsDump+0xe/0x10 [fglrx]

      [ 5579.462694]  [<ffffffffa01195cc>] firegl_hardwareHangRecovery+0x1c/0x50 [fglrx]

      [ 5579.462803]  [<ffffffffa01b51d9>] ? _ZN4Asic9WaitUntil15ResetASICIfHungEv+0x9/0x10 [fglrx]

      [ 5579.462910]  [<ffffffffa01b517c>] ? _ZN4Asic9WaitUntil15WaitForCompleteEv+0x9c/0xf0 [fglrx]

      [ 5579.463016]  [<ffffffffa01aef41>] ? _ZN4Asic19PM4ElapsedTimeStampEj14_LARGE_INTEGER12_QS_CP_RING_+0x141/0x160 [fglrx]

      [ 5579.463131]  [<ffffffffa01372a2>] ? firegl_trace+0x72/0x1e0 [fglrx]

      [ 5579.463200]  [<ffffffffa01372a2>] ? firegl_trace+0x72/0x1e0 [fglrx]

      [ 5579.463271]  [<ffffffffa01372a2>] ? firegl_trace+0x72/0x1e0 [fglrx]

      [ 5579.463391]  [<ffffffffa01a5ce3>] ? _ZN15QS_PRIVATE_CORE27multiVpuPM4ElapsedTimeStampEj14_LARGE_INTEGER12_QS_CP_RING_+0x33/0x50 [fglrx]

      [ 5579.463501]  [<ffffffffa019e4c4>] ? _Z19uQSTimeStampRetiredmjj14_LARGE_INTEGER+0x74/0x80 [fglrx]

      [ 5579.463609]  [<ffffffffa019a34d>] ? _Z8uCWDDEQCmjjPvjS_+0x54d/0x10c0 [fglrx]

      [ 5579.463627]  [<ffffffff8107fcee>] ? down+0x2e/0x50

      [ 5579.463696]  [<ffffffffa0139baf>] ? firegl_cmmqs_CWDDE_32+0x36f/0x480 [fglrx]

      [ 5579.463766]  [<ffffffffa013831e>] ? firegl_cmmqs_CWDDE32+0x6e/0x100 [fglrx]

      [ 5579.463782]  [<ffffffff812692c8>] ? security_capable+0x18/0x20

      [ 5579.463850]  [<ffffffffa01382b0>] ? firegl_cmmqs_createdriver+0x170/0x170 [fglrx]

      [ 5579.463910]  [<ffffffffa01152ed>] ? firegl_ioctl+0x1ed/0x250 [fglrx]

      [ 5579.463925]  [<ffffffff815f7c60>] ? do_page_fault+0x200/0x4f0

      [ 5579.463976]  [<ffffffffa01059ae>] ? ip_firegl_unlocked_ioctl+0xe/0x20 [fglrx]

      [ 5579.463992]  [<ffffffff81193228>] ? do_vfs_ioctl+0x98/0x550

      [ 5579.464012]  [<ffffffff81193771>] ? sys_ioctl+0x91/0xa0

      [ 5579.464020]  [<ffffffff815fc1e9>] ? system_call_fastpath+0x16/0x1b

      [ 5579.464032] pubdev:0xffffffffa03b3320, num of device:2 , name:fglrx, major 8, minor 96.

      [ 5579.464043] device 0 : 0xffff8804361ec000 .

      [ 5579.464052] Asic ID:0x6898, revision:0x2, MMIOReg:0xffffc90012c00000.

      [ 5579.464061] FB phys addr: 0xc0000000, MC :0xf00000000, Total FB size :0x80000000.

      [ 5579.464072] gart table MC:0xf0f8fd000, Physical:0xcf8fd000, size:0x402000.

      [ 5579.464083] mc_node :FB, total 1 zones

      [ 5579.464092]     MC start:0xf00000000, Physical:0xc0000000, size:0xfd00000.

      [ 5579.464102]     Mapped heap -- Offset:0x0, size:0xf8fd000, reference count:21, mapping count:0,

      [ 5579.464112]     Mapped heap -- Offset:0x0, size:0x1000000, reference count:1, mapping count:0,

      [ 5579.464123]     Mapped heap -- Offset:0xf8fd000, size:0x403000, reference count:1, mapping count:0,

      [ 5579.464134] mc_node :INV_FB, total 1 zones

      [ 5579.464143]     MC start:0xf0fd00000, Physical:0xcfd00000, size:0x70300000.

      [ 5579.464154]     Mapped heap -- Offset:0x702f4000, size:0xc000, reference count:1, mapping count:0,

      [ 5579.464164] mc_node :GART_USWC, total 3 zones

      [ 5579.464173]     MC start:0x40100000, Physical:0x0, size:0x50000000.

      [ 5579.464183]     Mapped heap -- Offset:0x0, size:0x2000000, reference count:17, mapping count:0,

      [ 5579.464194] mc_node :GART_CACHEABLE, total 3 zones

      [ 5579.464202]     MC start:0x10400000, Physical:0x0, size:0x2fd00000.

      [ 5579.464212]     Mapped heap -- Offset:0x1400000, size:0x100000, reference count:2, mapping count:0,

      [ 5579.464224]     Mapped heap -- Offset:0x2600000, size:0x900000, reference count:2, mapping count:0,

      [ 5579.464234]     Mapped heap -- Offset:0xb00000, size:0x900000, reference count:2, mapping count:0,

      [ 5579.464247]     Mapped heap -- Offset:0x200000, size:0x900000, reference count:3, mapping count:0,

      [ 5579.464258]     Mapped heap -- Offset:0x0, size:0x200000, reference count:4, mapping count:0,

      [ 5579.464269]     Mapped heap -- Offset:0xef000, size:0x11000, reference count:1, mapping count:0,

      [ 5579.464281] GRBM : 0xe0735828, SRBM : 0x200000c0 .

      [ 5579.464291] CP_RB_BASE : 0x401000, CP_RB_RPTR : 0x9ab0 , CP_RB_WPTR :0x9ab0.

      [ 5579.464302] CP_IB1_BUFSZ:0x0, CP_IB1_BASE_HI:0x0, CP_IB1_BASE_LO:0x4023d000.

      [ 5579.464313] last submit IB buffer -- MC :0x4023d000,phys:0x422fb5000.

      [ 5579.464329] device 1 : 0xffff880436134000 .

      [ 5579.464339] Asic ID:0x6798, revision:0x5, MMIOReg:0xffffc90012c80000.

      [ 5579.464348] FB phys addr: 0xb0000000, MC :0xf400000000, Total FB size :0xc0000000.

      [ 5579.464359] gart table MC:0xf40f8fd000, Physical:0xbf8fd000, size:0x402000.

      [ 5579.464369] mc_node :FB, total 1 zones

      [ 5579.464377]     MC start:0xf400000000, Physical:0xb0000000, size:0xfd00000.

      [ 5579.464388]     Mapped heap -- Offset:0x0, size:0xf8fd000, reference count:21, mapping count:0,

      [ 5579.464399]     Mapped heap -- Offset:0x0, size:0x1000000, reference count:1, mapping count:0,

      [ 5579.464410]     Mapped heap -- Offset:0xf8fd000, size:0x403000, reference count:1, mapping count:0,

      [ 5579.464421] mc_node :INV_FB, total 1 zones

      [ 5579.464430]     MC start:0xf40fd00000, Physical:0xbfd00000, size:0xb0300000.

      [ 5579.464441]     Mapped heap -- Offset:0xb02f4000, size:0xc000, reference count:1, mapping count:0,

      [ 5579.464452] mc_node :GART_USWC, total 3 zones

      [ 5579.464460]     MC start:0xffa0100000, Physical:0x0, size:0x50000000.

      [ 5579.464470]     Mapped heap -- Offset:0x0, size:0x2000000, reference count:16, mapping count:0,

      [ 5579.464480] mc_node :GART_CACHEABLE, total 3 zones

      [ 5579.464489]     MC start:0xff70400000, Physical:0x0, size:0x2fd00000.

      [ 5579.464499]     Mapped heap -- Offset:0x2600000, size:0x900000, reference count:2, mapping count:0,

      [ 5579.464510]     Mapped heap -- Offset:0x1d00000, size:0x900000, reference count:3, mapping count:0,

      [ 5579.464521]     Mapped heap -- Offset:0xb00000, size:0x900000, reference count:2, mapping count:0,

      [ 5579.464532]     Mapped heap -- Offset:0x200000, size:0x900000, reference count:3, mapping count:0,

      [ 5579.464543]     Mapped heap -- Offset:0x0, size:0x200000, reference count:4, mapping count:0,

      [ 5579.464554]     Mapped heap -- Offset:0xef000, size:0x11000, reference count:1, mapping count:0,

      [ 5579.464565] GRBM : 0xa0407028, SRBM : 0x200000c0 .

      [ 5579.464576] CP_RB_BASE : 0xffa01000, CP_RB_RPTR : 0x5180 , CP_RB_WPTR :0x5180.

      [ 5579.464587] CP_IB1_BUFSZ:0x0, CP_IB1_BASE_HI:0xff, CP_IB1_BASE_LO:0xa054e000.

      [ 5579.464597] last submit IB buffer -- MC :0xffa054e000,phys:0x420590000.

      [ 5579.464611] Dump the trace queue.

      [ 5579.464619] End of dump

       

        • Re: OpenCL crash multiple devices after multiple kernel enqueues.
          yurtesen

          I made changes in my code so I can now select Cypress or Tahiti when running the program. It functions fine on Cypress with several kernel enqueues. On Tahiti it does crash after few enqueues... so what can you suggest? this looks like a bug?

          • Re: OpenCL crash multiple devices after multiple kernel enqueues.
            yurtesen

            APP SDK 2.7 still crashes...

             

            [ 3180.358015] [fglrx] ASIC hang happened

            [ 3180.358024] Pid: 6388, comm: reference Tainted: P           O 3.3.5-2.fc16.x86_64 #1

            [ 3180.358035] Call Trace:

            [ 3180.358119]  [<ffffffffa023ff9e>] KCL_DEBUG_OsDump+0xe/0x10 [fglrx]

            [ 3180.358174]  [<ffffffffa024d5cc>] firegl_hardwareHangRecovery+0x1c/0x50 [fglrx]

            [ 3180.358284]  [<ffffffffa02e91d9>] ? _ZN4Asic9WaitUntil15ResetASICIfHungEv+0x9/0x10 [fglrx]

            [ 3180.358392]  [<ffffffffa02e917c>] ? _ZN4Asic9WaitUntil15WaitForCompleteEv+0x9c/0xf0 [fglrx]

            [ 3180.358499]  [<ffffffffa02e2f41>] ? _ZN4Asic19PM4ElapsedTimeStampEj14_LARGE_INTEGER12_QS_CP_RING_+0x141/0x160 [fglrx]

            [ 3180.358563]  [<ffffffffa026b2a2>] ? firegl_trace+0x72/0x1e0 [fglrx]

            [ 3180.358624]  [<ffffffffa026b2a2>] ? firegl_trace+0x72/0x1e0 [fglrx]

            [ 3180.358687]  [<ffffffffa026b2a2>] ? firegl_trace+0x72/0x1e0 [fglrx]

            [ 3180.358805]  [<ffffffffa02d9ce3>] ? _ZN15QS_PRIVATE_CORE27multiVpuPM4ElapsedTimeStampEj14_LARGE_INTEGER12_QS_CP_RING_+0x33/0x50 [fglrx]

            [ 3180.358909]  [<ffffffffa02d24c4>] ? _Z19uQSTimeStampRetiredmjj14_LARGE_INTEGER+0x74/0x80 [fglrx]

            [ 3180.359002]  [<ffffffffa02ce34d>] ? _Z8uCWDDEQCmjjPvjS_+0x54d/0x10c0 [fglrx]

            [ 3180.359045]  [<ffffffff8107fcce>] ? down+0x2e/0x50

            [ 3180.359108]  [<ffffffffa026dbaf>] ? firegl_cmmqs_CWDDE_32+0x36f/0x480 [fglrx]

            [ 3180.359178]  [<ffffffffa026c31e>] ? firegl_cmmqs_CWDDE32+0x6e/0x100 [fglrx]

            [ 3180.359195]  [<ffffffff81268f88>] ? security_capable+0x18/0x20

            [ 3180.359265]  [<ffffffffa026c2b0>] ? firegl_cmmqs_createdriver+0x170/0x170 [fglrx]

            [ 3180.359324]  [<ffffffffa02492ed>] ? firegl_ioctl+0x1ed/0x250 [fglrx]

            [ 3180.359341]  [<ffffffff815f7d60>] ? do_page_fault+0x200/0x4f0

            [ 3180.359394]  [<ffffffffa02399ae>] ? ip_firegl_unlocked_ioctl+0xe/0x20 [fglrx]

            [ 3180.359409]  [<ffffffff81192fb8>] ? do_vfs_ioctl+0x98/0x550

            [ 3180.359423]  [<ffffffff81193501>] ? sys_ioctl+0x91/0xa0

            [ 3180.359437]  [<ffffffff815fc2e9>] ? system_call_fastpath+0x16/0x1b

            [ 3180.359450] pubdev:0xffffffffa04e7320, num of device:2 , name:fglrx, major 8, minor 96.

            [ 3180.359461] device 0 : 0xffff880436b20000 .

            [ 3180.359471] Asic ID:0x6898, revision:0x2, MMIOReg:0xffffc90012c00000.

            [ 3180.359482] FB phys addr: 0xc0000000, MC :0xf00000000, Total FB size :0x80000000.

            [ 3180.359492] gart table MC:0xf0f8fd000, Physical:0xcf8fd000, size:0x402000.

            [ 3180.359503] mc_node :FB, total 1 zones

            [ 3180.359511]     MC start:0xf00000000, Physical:0xc0000000, size:0xfd00000.

            [ 3180.359522]     Mapped heap -- Offset:0x0, size:0xf8fd000, reference count:22, mapping count:0,

            [ 3180.359533]     Mapped heap -- Offset:0x0, size:0x1000000, reference count:1, mapping count:0,

            [ 3180.359544]     Mapped heap -- Offset:0xf8fd000, size:0x403000, reference count:1, mapping count:0,

            [ 3180.359553] mc_node :INV_FB, total 1 zones

            [ 3180.359562]     MC start:0xf0fd00000, Physical:0xcfd00000, size:0x70300000.

            [ 3180.359572]     Mapped heap -- Offset:0x702f4000, size:0xc000, reference count:1, mapping count:0,

            [ 3180.359583] mc_node :GART_USWC, total 3 zones

            [ 3180.359592]     MC start:0x40100000, Physical:0x0, size:0x50000000.

            [ 3180.359602]     Mapped heap -- Offset:0x0, size:0x2000000, reference count:17, mapping count:0,

            [ 3180.359613] mc_node :GART_CACHEABLE, total 3 zones

            [ 3180.359622]     MC start:0x10400000, Physical:0x0, size:0x2fd00000.

            [ 3180.359632]     Mapped heap -- Offset:0x2f00000, size:0x1100000, reference count:1, mapping count:0,

            [ 3180.359643]     Mapped heap -- Offset:0x1500000, size:0x100000, reference count:2, mapping count:0,

            [ 3180.359654]     Mapped heap -- Offset:0x1400000, size:0x100000, reference count:1, mapping count:0,

            [ 3180.359666]     Mapped heap -- Offset:0x2600000, size:0x900000, reference count:2, mapping count:0,

            [ 3180.359678]     Mapped heap -- Offset:0xb00000, size:0x900000, reference count:3, mapping count:0,

            [ 3180.359689]     Mapped heap -- Offset:0x200000, size:0x900000, reference count:2, mapping count:0,

            [ 3180.359700]     Mapped heap -- Offset:0x0, size:0x200000, reference count:6, mapping count:0,

            [ 3180.359711]     Mapped heap -- Offset:0xef000, size:0x11000, reference count:1, mapping count:0,

            [ 3180.359723] GRBM : 0xa0635028, SRBM : 0x200000c0 .

            [ 3180.359734] CP_RB_BASE : 0x401000, CP_RB_RPTR : 0xdfa0 , CP_RB_WPTR :0xdfa0.

            [ 3180.359746] CP_IB1_BUFSZ:0x0, CP_IB1_BASE_HI:0x0, CP_IB1_BASE_LO:0x4034a000.

            [ 3180.359756] last submit IB buffer -- MC :0x4034a000,phys:0x4220b6000.

            [ 3180.359771] device 1 : 0xffff880436cd0000 .

            [ 3180.359780] Asic ID:0x6798, revision:0x5, MMIOReg:0xffffc90012c80000.

            [ 3180.359790] FB phys addr: 0xb0000000, MC :0xf400000000, Total FB size :0xc0000000.

            [ 3180.359801] gart table MC:0xf40f8fd000, Physical:0xbf8fd000, size:0x402000.

            [ 3180.359810] mc_node :FB, total 1 zones

            [ 3180.359819]     MC start:0xf400000000, Physical:0xb0000000, size:0xfd00000.

            [ 3180.359829]     Mapped heap -- Offset:0x0, size:0xf8fd000, reference count:21, mapping count:0,

            [ 3180.359840]     Mapped heap -- Offset:0x0, size:0x1000000, reference count:1, mapping count:0,

            [ 3180.359851]     Mapped heap -- Offset:0xf8fd000, size:0x403000, reference count:1, mapping count:0,

            [ 3180.359862] mc_node :INV_FB, total 1 zones

            [ 3180.359871]     MC start:0xf40fd00000, Physical:0xbfd00000, size:0xb0300000.

            [ 3180.359881]     Mapped heap -- Offset:0xb02f4000, size:0xc000, reference count:1, mapping count:0,

            [ 3180.359891] mc_node :GART_USWC, total 3 zones

            [ 3180.359900]     MC start:0xffa0100000, Physical:0x0, size:0x50000000.

            [ 3180.359910]     Mapped heap -- Offset:0x0, size:0x2000000, reference count:16, mapping count:0,

            [ 3180.359920] mc_node :GART_CACHEABLE, total 3 zones

            [ 3180.359929]     MC start:0xff70400000, Physical:0x0, size:0x2fd00000.

            [ 3180.359939]     Mapped heap -- Offset:0x2f00000, size:0x1100000, reference count:1, mapping count:0,

            [ 3180.359949]     Mapped heap -- Offset:0x2600000, size:0x900000, reference count:2, mapping count:0,

            [ 3180.359961]     Mapped heap -- Offset:0x1d00000, size:0x900000, reference count:3, mapping count:0,

            [ 3180.359972]     Mapped heap -- Offset:0xb00000, size:0x900000, reference count:2, mapping count:0,

            [ 3180.359983]     Mapped heap -- Offset:0x200000, size:0x900000, reference count:3, mapping count:0,

            [ 3180.359995]     Mapped heap -- Offset:0x0, size:0x200000, reference count:4, mapping count:0,

            [ 3180.360010]     Mapped heap -- Offset:0xef000, size:0x11000, reference count:1, mapping count:0,

            [ 3180.360017] GRBM : 0xa04c7028, SRBM : 0x20000fc0 .

            [ 3180.360023] CP_RB_BASE : 0xffa01000, CP_RB_RPTR : 0xedd0 , CP_RB_WPTR :0xedd0.

            [ 3180.360030] CP_IB1_BUFSZ:0x0, CP_IB1_BASE_HI:0xff, CP_IB1_BASE_LO:0xa083b000.

            [ 3180.360035] last submit IB buffer -- MC :0xffa083b000,phys:0x41e0f8000.

            [ 3180.360051] Dump the trace queue.

            [ 3180.360058] End of dump

            [ 3197.866847] [fglrx] ASIC hang happened

            [ 3197.866852] Pid: 6387, comm: reference Tainted: P           O 3.3.5-2.fc16.x86_64 #1

            [ 3197.866854] Call Trace:

            [ 3197.866905]  [<ffffffffa023ff9e>] KCL_DEBUG_OsDump+0xe/0x10 [fglrx]

            [ 3197.866931]  [<ffffffffa024d5cc>] firegl_hardwareHangRecovery+0x1c/0x50 [fglrx]

            [ 3197.866979]  [<ffffffffa02e91d9>] ? _ZN4Asic9WaitUntil15ResetASICIfHungEv+0x9/0x10 [fglrx]

            [ 3197.867027]  [<ffffffffa02e917c>] ? _ZN4Asic9WaitUntil15WaitForCompleteEv+0x9c/0xf0 [fglrx]

            [ 3197.867086]  [<ffffffffa02e2f41>] ? _ZN4Asic19PM4ElapsedTimeStampEj14_LARGE_INTEGER12_QS_CP_RING_+0x141/0x160 [fglrx]

            [ 3197.867118]  [<ffffffffa026b2a2>] ? firegl_trace+0x72/0x1e0 [fglrx]

            [ 3197.867148]  [<ffffffffa026b2a2>] ? firegl_trace+0x72/0x1e0 [fglrx]

            [ 3197.867180]  [<ffffffffa026b2a2>] ? firegl_trace+0x72/0x1e0 [fglrx]

            [ 3197.867237]  [<ffffffffa02d9ce3>] ? _ZN15QS_PRIVATE_CORE27multiVpuPM4ElapsedTimeStampEj14_LARGE_INTEGER12_QS_CP_RING_+0x33/0x50 [fglrx]

            [ 3197.867286]  [<ffffffffa02d24c4>] ? _Z19uQSTimeStampRetiredmjj14_LARGE_INTEGER+0x74/0x80 [fglrx]

            [ 3197.867333]  [<ffffffffa02ce34d>] ? _Z8uCWDDEQCmjjPvjS_+0x54d/0x10c0 [fglrx]

            [ 3197.867342]  [<ffffffff8107fcce>] ? down+0x2e/0x50

            [ 3197.867373]  [<ffffffffa026dbaf>] ? firegl_cmmqs_CWDDE_32+0x36f/0x480 [fglrx]

            [ 3197.867404]  [<ffffffffa026c31e>] ? firegl_cmmqs_CWDDE32+0x6e/0x100 [fglrx]

            [ 3197.867412]  [<ffffffff81268f88>] ? security_capable+0x18/0x20

            [ 3197.867443]  [<ffffffffa026c2b0>] ? firegl_cmmqs_createdriver+0x170/0x170 [fglrx]

            [ 3197.867470]  [<ffffffffa02492ed>] ? firegl_ioctl+0x1ed/0x250 [fglrx]

            [ 3197.867477]  [<ffffffff815f7d60>] ? do_page_fault+0x200/0x4f0

            [ 3197.867501]  [<ffffffffa02399ae>] ? ip_firegl_unlocked_ioctl+0xe/0x20 [fglrx]

            [ 3197.867508]  [<ffffffff81192fb8>] ? do_vfs_ioctl+0x98/0x550

            [ 3197.867515]  [<ffffffff81193501>] ? sys_ioctl+0x91/0xa0

            [ 3197.867522]  [<ffffffff815fc2e9>] ? system_call_fastpath+0x16/0x1b

            [ 3197.867527] pubdev:0xffffffffa04e7320, num of device:2 , name:fglrx, major 8, minor 96.

            [ 3197.867533] device 0 : 0xffff880436b20000 .

            [ 3197.867537] Asic ID:0x6898, revision:0x2, MMIOReg:0xffffc90012c00000.

            [ 3197.867541] FB phys addr: 0xc0000000, MC :0xf00000000, Total FB size :0x80000000.

            [ 3197.867546] gart table MC:0xf0f8fd000, Physical:0xcf8fd000, size:0x402000.

            [ 3197.867551] mc_node :FB, total 1 zones

            [ 3197.867556]     MC start:0xf00000000, Physical:0xc0000000, size:0xfd00000.

            [ 3197.867561]     Mapped heap -- Offset:0x0, size:0xf8fd000, reference count:22, mapping count:0,

            [ 3197.867566]     Mapped heap -- Offset:0x0, size:0x1000000, reference count:1, mapping count:0,

            [ 3197.867571]     Mapped heap -- Offset:0xf8fd000, size:0x403000, reference count:1, mapping count:0,

            [ 3197.867575] mc_node :INV_FB, total 1 zones

            [ 3197.867579]     MC start:0xf0fd00000, Physical:0xcfd00000, size:0x70300000.

            [ 3197.867583]     Mapped heap -- Offset:0x702f4000, size:0xc000, reference count:1, mapping count:0,

            [ 3197.867588] mc_node :GART_USWC, total 3 zones

            [ 3197.867592]     MC start:0x40100000, Physical:0x0, size:0x50000000.

            [ 3197.867596]     Mapped heap -- Offset:0x0, size:0x2000000, reference count:17, mapping count:0,

            [ 3197.867601] mc_node :GART_CACHEABLE, total 3 zones

            [ 3197.867605]     MC start:0x10400000, Physical:0x0, size:0x2fd00000.

            [ 3197.867609]     Mapped heap -- Offset:0x2f00000, size:0x1100000, reference count:1, mapping count:0,

            [ 3197.867614]     Mapped heap -- Offset:0x1500000, size:0x100000, reference count:2, mapping count:0,

            [ 3197.867618]     Mapped heap -- Offset:0x1400000, size:0x100000, reference count:1, mapping count:0,

            [ 3197.867624]     Mapped heap -- Offset:0x2600000, size:0x900000, reference count:2, mapping count:0,

            [ 3197.867628]     Mapped heap -- Offset:0xb00000, size:0x900000, reference count:3, mapping count:0,

            [ 3197.867633]     Mapped heap -- Offset:0x200000, size:0x900000, reference count:2, mapping count:0,

            [ 3197.867638]     Mapped heap -- Offset:0x0, size:0x200000, reference count:6, mapping count:0,

            [ 3197.867643]     Mapped heap -- Offset:0xef000, size:0x11000, reference count:1, mapping count:0,

            [ 3197.867649] GRBM : 0xa0637828, SRBM : 0x20000cc0 .

            [ 3197.867654] CP_RB_BASE : 0x401000, CP_RB_RPTR : 0xdfe0 , CP_RB_WPTR :0xdfe0.

            [ 3197.867660] CP_IB1_BUFSZ:0x0, CP_IB1_BASE_HI:0x0, CP_IB1_BASE_LO:0x4034c000.

            [ 3197.867664] last submit IB buffer -- MC :0x4034c000,phys:0x4220b8000.

            [ 3197.867671] device 1 : 0xffff880436cd0000 .

            [ 3197.867675] Asic ID:0x6798, revision:0x5, MMIOReg:0xffffc90012c80000.

            [ 3197.867680] FB phys addr: 0xb0000000, MC :0xf400000000, Total FB size :0xc0000000.

            [ 3197.867684] gart table MC:0xf40f8fd000, Physical:0xbf8fd000, size:0x402000.

            [ 3197.867689] mc_node :FB, total 1 zones

            [ 3197.867692]     MC start:0xf400000000, Physical:0xb0000000, size:0xfd00000.

            [ 3197.867697]     Mapped heap -- Offset:0x0, size:0xf8fd000, reference count:21, mapping count:0,

            [ 3197.867702]     Mapped heap -- Offset:0x0, size:0x1000000, reference count:1, mapping count:0,

            [ 3197.867707]     Mapped heap -- Offset:0xf8fd000, size:0x403000, reference count:1, mapping count:0,

            [ 3197.867712] mc_node :INV_FB, total 1 zones

            [ 3197.867715]     MC start:0xf40fd00000, Physical:0xbfd00000, size:0xb0300000.

            [ 3197.867720]     Mapped heap -- Offset:0xb02f4000, size:0xc000, reference count:1, mapping count:0,

            [ 3197.867724] mc_node :GART_USWC, total 3 zones

            [ 3197.867728]     MC start:0xffa0100000, Physical:0x0, size:0x50000000.

            [ 3197.867732]     Mapped heap -- Offset:0x0, size:0x2000000, reference count:16, mapping count:0,

            [ 3197.867737] mc_node :GART_CACHEABLE, total 3 zones

            [ 3197.867740]     MC start:0xff70400000, Physical:0x0, size:0x2fd00000.

            [ 3197.867745]     Mapped heap -- Offset:0x2f00000, size:0x1100000, reference count:1, mapping count:0,

            [ 3197.867750]     Mapped heap -- Offset:0x2600000, size:0x900000, reference count:2, mapping count:0,

            [ 3197.867755]     Mapped heap -- Offset:0x1d00000, size:0x900000, reference count:3, mapping count:0,

            [ 3197.867760]     Mapped heap -- Offset:0xb00000, size:0x900000, reference count:2, mapping count:0,

            [ 3197.867764]     Mapped heap -- Offset:0x200000, size:0x900000, reference count:3, mapping count:0,

            [ 3197.867769]     Mapped heap -- Offset:0x0, size:0x200000, reference count:4, mapping count:0,

            [ 3197.867774]     Mapped heap -- Offset:0xef000, size:0x11000, reference count:1, mapping count:0,

            [ 3197.867780] GRBM : 0xa0407028, SRBM : 0x200000c0 .

            [ 3197.867786] CP_RB_BASE : 0xffa01000, CP_RB_RPTR : 0xedd0 , CP_RB_WPTR :0xedd0.

            [ 3197.867792] CP_IB1_BUFSZ:0x0, CP_IB1_BASE_HI:0xff, CP_IB1_BASE_LO:0xa083b000.

            [ 3197.867796] last submit IB buffer -- MC :0xffa083b000,phys:0x41e0f8000.

            [ 3197.867803] Dump the trace queue.

            [ 3197.867806] End of dump