AnsweredAssumed Answered

Inconsistent number of wavefronts reported by sprofile

Question asked by gbilotta on Jan 31, 2012
Latest reply on Feb 3, 2012 by gbilotta

I'm analyzing the impact of workgroup size in the performance of a kernel, in a rather naive and brute-force way: I upload the kernel parameters once, and then execute the kernels repeatedly with different workgroup size configurations, starting from the kernel-preferred workgroup size multiple and doubling each time. For each size, I also try different workgroup shapes (from size x 1 to 1 x size, again halving/doubling for each test), and each configuration is tried multiple times to also find 'jitter' in the execution.

I noticed that even with the same exact configuration, the profiler sometimes reports a _different_ number of wavefronts, ALU instructions, fetch instructions, write instructions etc. This doesn't seem to be correlated with performance, in the sense that the runtime outliers in the repeated tests are not necessarily the ones with the different number of wavefronts etc. An extract is in the table that follows. The work size is always 3712*3172, needing exactly 215296 wavefronts. The actual WGS for the NULL configuration is unknown (is there a way to tell? the only thing I could get is that it uses 256-sized workgroups, from the occupancy table). Notice how the 64x1 configuration has two repetition with too many wavefronts, lower ALUinsts etc than the third 64x1 repetition, similarly for the last 32x2 vs the first two 32x2, and see how the 12ms 16x4 is identical to the 16x4.

WorkGroupSizeTimeWavefrontsALUInstsFetchInstsWriteInstsALUBusyALUFetchRatioALUPacking
NULL3.06689278594.0043.114.201.3967.5810.2669.42
NULL3.05967278594.0043.114.201.3967.7210.2669.42
NULL3.07189278594.0043.114.201.3967.7810.2669.42
64x13.32733256322.0042.934.191.4057.7910.2469.50
64x13.32289256322.0042.934.191.4057.8610.2469.50
64x13.31767215296.0050.734.901.4762.7210.3669.50
32x23.31378215296.0048.854.701.3961.1510.3969.21
32x23.29900215296.0048.854.701.3960.9910.3969.21
32x23.30444256322.0041.354.031.3356.0110.2769.21
16x43.30778215296.0047.534.561.3359.4310.4268.99
16x43.28522215296.0047.534.561.3359.4310.4268.99
16x412.92222215296.0047.534.561.3359.3610.4268.99

 

It is my understanding that the overlong kernels are probably caused by the X server preempting the GPU (Radeon HD 6970) (by the way, when will OpenCL be available on AMD cards _without_ an X server?), but what can be the cause for the odd wavefront computations? Is it a bug in the profiler (2.4.1317 for amd64 running in Ubuntu), OpenCL (AMD APP 2.6) or the video driver (fglrx 8.91.4, from fglrx-updates on Ubuntu)?

Outcomes