cancel
Showing results for 
Search instead for 
Did you mean: 

OpenCL

Highlighted

OpenCL compiler bug

I've been working on adding OpenCL support to our code generator (GitHub - genn-team/genn: GeNN is a GPU-enhanced Neuronal Network simulation environment based on cod... ) and the generated code is now working on NVIDIA, Intel and ARM devices but we've been having ongoing issues getting this to work on AMD GPUs. With new enough Adrenaline drivers and a relatively modern GCN GPU, all now seems good on Windows but, in order to do some more rigorous testing in-house, we have now bought a Radeon 5700 XT for one of our Linux machines. Now, using the AMD GPU PRO 20.30 drivers we're seeing similar broken behavior. 

I have reduced a simple case to the attached minimal reproduction which, on NVIDIA and Intel hardware, prints out:

1,1,0,0

However, on the 5700 XT, it prints out:

0,0,0,0

Similarly to our previous issue, if you add any printf to any kernel, it works correctly. Additionally, interspersing commandQueue.flush() calls between kernel launch has the same effect however, from my understanding of the OpenCL spec, this should not be necessary.

Tags (1)
0 Kudos
Reply
27 Replies
Highlighted
Staff
Staff

Re: OpenCL compiler bug

Thank you for reporting the above issue and providing the reproducible test-case. We will look in this and get back to you. Please attach the clinfo out.

Thanks.

0 Kudos
Reply
Highlighted

Re: OpenCL compiler bug

Thanks for your rapid response. clInfo output for the AMD device is as follows:

Platform Name AMD Accelerated Parallel Processing
Platform Vendor Advanced Micro Devices, Inc.
Platform Version OpenCL 2.1 AMD-APP (3143.9)
Platform Profile FULL_PROFILE
Platform Extensions cl_khr_icd cl_amd_event_callback cl_amd_offline_devices
Platform Host timer resolution 1ns
Platform Extensions function suffix AMD

Platform Name AMD Accelerated Parallel Processing
Number of devices 1
Device Name gfx1010
Device Vendor Advanced Micro Devices, Inc.
Device Vendor ID 0x1002
Device Version OpenCL 2.0 AMD-APP (3143.9)
Driver Version 3143.9 (PAL,LC)
Device OpenCL C Version OpenCL C 2.0
Device Type GPU
Device Board Name (AMD) AMD Radeon RX 5700 XT
Device Topology (AMD) PCI-E, 8f:00.0
Device Profile FULL_PROFILE
Device Available Yes
Compiler Available Yes
Linker Available Yes
Max compute units 20
SIMD per compute unit (AMD) 4
SIMD width (AMD) 32
SIMD instruction width (AMD) 1
Max clock frequency 2100MHz
Graphics IP (AMD) 10.10
Device Partition (core)
Max number of sub-devices 20
Supported partition types None
Max work item dimensions 3
Max work item sizes 1024x1024x1024
Max work group size 256
Preferred work group size (AMD) 256
Max work group size (AMD) 1024
Preferred work group size multiple 32
Wavefront width (AMD) 32
Preferred / native vector sizes
char 4 / 4
short 2 / 2
int 1 / 1
long 1 / 1
half 1 / 1 (cl_khr_fp16)
float 1 / 1
double 1 / 1 (cl_khr_fp64)
Half-precision Floating-point support (cl_khr_fp16)
Denormals No
Infinity and NANs No
Round to nearest No
Round to zero No
Round to infinity No
IEEE754-2008 fused multiply-add No
Support is emulated in software No
Single-precision Floating-point support (core)
Denormals Yes
Infinity and NANs Yes
Round to nearest Yes
Round to zero Yes
Round to infinity Yes
IEEE754-2008 fused multiply-add Yes
Support is emulated in software No
Correctly-rounded divide and sqrt operations Yes
Double-precision Floating-point support (cl_khr_fp64)
Denormals Yes
Infinity and NANs Yes
Round to nearest Yes
Round to zero Yes
Round to infinity Yes
IEEE754-2008 fused multiply-add Yes
Support is emulated in software No
Address bits 64, Little-Endian
Global memory size 8573157376 (7.984GiB)
Global free memory (AMD) 8306688 (7.922GiB)
Global memory channels (AMD) 8
Global memory banks per channel (AMD) 4
Global memory bank width (AMD) 256 bytes
Error Correction support No
Max memory allocation 7287183769 (6.787GiB)
Unified memory for Host and Device No
Shared Virtual Memory (SVM) capabilities (core)
Coarse-grained buffer sharing Yes
Fine-grained buffer sharing Yes
Fine-grained system sharing No
Atomics No
Minimum alignment for any data type 128 bytes
Alignment of base address 2048 bits (256 bytes)
Preferred alignment for atomics
SVM 0 bytes
Global 0 bytes
Local 0 bytes
Max size for global variable 6558465280 (6.108GiB)
Preferred total size of global vars 8573157376 (7.984GiB)
Global Memory cache type Read/Write
Global Memory cache size 16384 (16KiB)
Global Memory cache line size 64 bytes
Image support Yes
Max number of samplers per kernel 16
Max size for 1D images from buffer 134217728 pixels
Max 1D or 2D image array size 2048 images
Base address alignment for 2D image buffers 256 bytes
Pitch alignment for 2D image buffers 256 pixels
Max 2D image size 16384x16384 pixels
Max 3D image size 2048x2048x2048 pixels
Max number of read image args 128
Max number of write image args 64
Max number of read/write image args 64
Max number of pipe args 16
Max active pipe reservations 16
Max pipe packet size 2992216473 (2.787GiB)
Local memory type Local
Local memory size 65536 (64KiB)
Local memory syze per CU (AMD) 65536 (64KiB)
Local memory banks (AMD) 32
Max number of constant args 8
Max constant buffer size 7287183769 (6.787GiB)
Preferred constant buffer size (AMD) 16384 (16KiB)
Max size of kernel argument 1024
Queue properties (on host)
Out-of-order execution No
Profiling Yes
Queue properties (on device)
Out-of-order execution Yes
Profiling Yes
Preferred size 262144 (256KiB)
Max size 8388608 (8MiB)
Max queues on device 1
Max events on device 1024
Prefer user sync for interop Yes
Number of P2P devices (AMD) 0
P2P devices (AMD) <printDeviceInfo:144: get number of CL_DEVICE_P2P_DEVICES_AMD : error -30>
Profiling timer resolution 1ns
Profiling timer offset since Epoch (AMD) 1600456210444955455ns (Fri Sep 18 20:10:10 2020)
Execution capabilities
Run OpenCL kernels Yes
Run native kernels No
Thread trace supported (AMD) Yes
Number of async queues (AMD) 4
Max real-time compute queues (AMD) 1
Max real-time compute units (AMD) 0
printf() buffer size 4194304 (4MiB)
Built-in kernels
Device Extensions cl_khr_fp64 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_3d_image_writes cl_khr_byte_addressable_store cl_khr_fp16 cl_khr_gl_sharing cl_khr_gl_depth_images cl_amd_device_attribute_query cl_amd_media_ops cl_amd_media_ops2 cl_khr_image2d_from_buffer cl_khr_subgroups cl_khr_gl_event cl_khr_depth_images cl_khr_mipmap_image cl_khr_mipmap_image_writes cl_amd_copy_buffer_p2p

0 Kudos
Reply
Highlighted
Staff
Staff

Re: OpenCL compiler bug

I was able to reproduce the issue on Windows with the latest driver. So, I'm not sure if this is related to the other issue you mentioned (https://community.amd.com/thread/254777 ). Did you try it on Windows? 

I ran the code with CodeXL and found some interesting findings. It doesn't seem a compilation issue. From the CodeXL profiler report, it looks like the buffer reading is happening before all the kernels finish their execution. As a result, we are getting zeros values. This seems to be a sync issue here.

Here are the related code section:

CHECK_OPENCL_ERRORS(updatePresynapticKernel.setArg(0, d_mergedPresynapticUpdateGroup0));
CHECK_OPENCL_ERRORS(commandQueue.enqueueNDRangeKernel(updatePresynapticKernel, cl::NullRange, globalWorkSize, localWorkSize));

CHECK_OPENCL_ERRORS(updateNeuronsKernel.setArg(0, d_mergedNeuronUpdateGroup1));
CHECK_OPENCL_ERRORS(commandQueue.enqueueNDRangeKernel(updateNeuronsKernel, cl::NullRange, globalWorkSize, localWorkSize));

// Copy output back to host
CHECK_OPENCL_ERRORS(commandQueue.enqueueReadBuffer(d_xPost, CL_TRUE, 0, 4 * sizeof(float), xPost));

The way output buffer "d_xPost" is indirectly accessed in the kernel, this buffer seems to be independent of two other buffers which are passed to the kernels i.e. "d_mergedPresynapticUpdateGroup0" and "d_mergedNeuronUpdateGroup1".  As a result,  enqueueReadBuffer also seems to be independent of these two kernels, so, it can be independently executed without affecting the program consistency. I guess, that is the case here.

From the CodeXL profiler report, it seems that all the commands were submitted to the queue in the same order as mentioned in the program, but on the device, enqueueReadBuffer was actually executed before the above two kernels completed. However, both the kernels were executed in order. I think, the below description in the AMD OpenCL optimization guide might be helpful to explain this behavior.

It is best to use non-blocking commands to allow multiple commands to be queued before the command queue is flushed to the GPU. This sends larger batches of commands, which amortizes the cost of preparing and submitting work to the GPU. Use event tracking to specify the dependence between operations. It is recommended to queue operations that do not depend of the results of previous copy and map operations. This can help keep the GPU busy with kernel execution and DMA transfers. Command execution begins as soon as there are commands in the queue for execution.

If commandQueue.flush() is added before the buffer reading, it looks like all these above commands are following the program order. 

Also, when enqueueReadBuffer(d_mergedNeuronUpdateGroup1 ,..) was added before reading the output buffer "d_xPost" without any flush() call, the output was as expected.

Regarding the printf behavior, it seems that some implicit synchronizations are added when kernels contain printf statements, so we are getting the expected output.

Could you please try the CodeXL profiler to see if you have similar findings?

Thanks.

Highlighted

Re: OpenCL compiler bug

Thank you so much for your detailed investigation of this issue. You're right that this does indeed occur on Windows and that part of the problem was down to the read occuring too early (which is very counter-intuitive on an in-order queue but still). I entirely failing to get CodeXL or Radeon Compute Profiler to work but I added profiling events into the queue and you can see that, without any additional flushes, the readXPostEventStart happens before any kernels (long numbers are times of profiling events and everything is sorted by time):

762101045076768(mapXPostEventStart)
762101045111522(mapSpkCntPreEventStart)
762101045303778(fillXPostEventStart)
762101045307418(fillXPostEventEnd)
762101045307778(fillInSynEventStart)
762101045308098(fillInSynEventEnd)
762101045312138(buildNeuronKernelEventStart)
762101045314218(buildNeuronKernelEventEnd)
762101045317818(buildPresynapticKernelEventStart)
762101045319858(buildPresynapticKernelEventEnd)
762101045794578(writeSpkCntPreEventStart)
762101045800058(writeSpkCntPreEventEnd)
762101045924489(readXPostEventStart)
762101045924729(updatePresynapticEventStart)
762101045927369(readXPostEventEnd)
762101045930089(updatePresynapticEventEnd)
762101045930449(updateNeuronsEventStart)
762101045930769(updateNeuronsEventEnd)

However, when I add a single flush event before reading (uncomment FLUSH_BEFORE_READ), the events are ordered correctly:

767847480970088(mapXPostEventStart)
767847481011538(mapSpkCntPreEventStart)
767847481308052(fillXPostEventStart)
767847481310852(fillXPostEventEnd)
767847481311212(fillInSynEventStart)
767847481311532(fillInSynEventEnd)
767847481315532(buildNeuronKernelEventStart)
767847481317012(buildNeuronKernelEventEnd)
767847481320372(buildPresynapticKernelEventStart)
767847481322012(buildPresynapticKernelEventEnd)
767847481509852(writeSpkCntPreEventStart)
767847481515172(writeSpkCntPreEventEnd)
767847481619399(updatePresynapticEventStart)
767847481624799(updatePresynapticEventEnd)
767847481625159(updateNeuronsEventStart)
767847481625479(updateNeuronsEventEnd)
767847481733447(readXPostEventStart)
767847481736487(readXPostEventEnd)

BUT the outut is still incorrect (0,0,0,0). Only when I add another flush between the two kernel launches (uncomment FLUSH_BEFORE_READ and FLUSH_BETWEEN_KERNELS) do I get the correct output (1,1,0,0):

767788672578162(mapXPostEventStart)
767788672615890(mapSpkCntPreEventStart)
767788672900801(fillXPostEventStart)
767788672905881(fillXPostEventEnd)
767788672906201(fillInSynEventStart)
767788672906561(fillInSynEventEnd)
767788672910561(buildNeuronKernelEventStart)
767788672912081(buildNeuronKernelEventEnd)
767788672915481(buildPresynapticKernelEventStart)
767788672917121(buildPresynapticKernelEventEnd)
767788673111441(writeSpkCntPreEventStart)
767788673116801(writeSpkCntPreEventEnd)
767788673223561(updatePresynapticEventStart)
767788673230921(updatePresynapticEventEnd)
767788673251235(updateNeuronsEventStart)
767788673253355(updateNeuronsEventEnd)
767788673358035(readXPostEventStart)
767788673361035(readXPostEventEnd)

Furthermore, attempting to replace the flush before the read with a wait on the update neurons kernel (uncomment WAIT_BEFORE_READ) seems to have no effect whatsoever on the ordering which is really bizarre:

825588782273194(mapXPostEventStart)
825588782305039(mapSpkCntPreEventStart)
825588782459757(fillXPostEventStart)
825588782462837(fillXPostEventEnd)
825588782463197(fillInSynEventStart)
825588782463517(fillInSynEventEnd)
825588782795957(buildNeuronKernelEventStart)
825588782797517(buildNeuronKernelEventEnd)
825588782801117(buildPresynapticKernelEventStart)
825588782802677(buildPresynapticKernelEventEnd)
825588782987157(writeSpkCntPreEventStart)
825588782992477(writeSpkCntPreEventEnd)
825588783142546(readXPostEventStart)
825588783144946(updatePresynapticEventStart)
825588783146266(readXPostEventEnd)
825588783150186(updatePresynapticEventEnd)
825588783150546(updateNeuronsEventStart)
825588783150866(updateNeuronsEventEnd)

The code with these additions is attached.

Thanks again for all your help

Jamie

Highlighted
Staff
Staff

Re: OpenCL compiler bug

Thank you for sharing the above information.

However, when I add a single flush event before reading (uncomment FLUSH_BEFORE_READ), the events are ordered correctly...

...
BUT the outut is still incorrect (0,0,0,0). Only when I add another flush between the two kernel launches (uncomment FLUSH_BEFORE_READ and FLUSH_BETWEEN_KERNELS) do I get the correct output (1,1,0,0).

In my case, when I added a single flush before the reading, I got the expected result. I tried it on Windows with some other card, so I'm not sure if that might be reason for this different  observations. I'll try the new attached code with CodeXL and check the execution order. Also, if needed, I will share these findings with the OpenCL team for their feedback.

Thanks.

0 Kudos
Reply
Highlighted

Re: OpenCL compiler bug

Interesting - I forgot to add that we also reproduced the same behaviour on Windows with an Radeon RX 580 and 20.5.1 drivers

0 Kudos
Reply
Highlighted

Re: OpenCL compiler bug

Also, similarly to using events, using clEnqueueBarrierWithWaitList before reading has no effect on the ordering which seems to directly go against description in the spec that it "waits for all commands previously enqueued in command_queue to complete before it completes" (https://www.khronos.org/registry/OpenCL/specs/opencl-1.2.pdf )

0 Kudos
Reply
Highlighted
Staff
Staff

Re: OpenCL compiler bug

Thank you for sharing this interesting findings. Earlier I didn't check the code with the event objects, but I was under the impression that it should work if some explicit synchronization is added to order those commands. However, as you said above, you are still observing the issue with event object/barrier. In that case, I'll check with the OpenCL team for their feedback on this.

Thanks.

0 Kudos
Reply
Highlighted

Re: OpenCL compiler bug

Indeed - while I can understand the potential optimisation benefits of scheduling buffer reads earlier than they are enqueued, ignoring synchronisation barriers and events seems like a bug to me.

0 Kudos
Reply