OpenCL

[develancer]

[To moderator:] I didn't know whether I should post it here, or in the Support Forums → Graphics section, so please delete the one that does not fit.

AMD OpenCL compiler used for RX 5700 XT with ROCM 4.0.1 sometimes generates invalid code for integer multiplication.

The self-contained example is here: https://github.com/develancer/amd-opencl-test-case

My clinfo:

  Platform Name:				 AMD Accelerated Parallel Processing
Number of devices:				 1
  Device Type:					 CL_DEVICE_TYPE_GPU
  Vendor ID:					 1002h
  Board name:					 Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT]
  Device Topology:				 PCI[ B#11, D#0, F#0 ]
  Max compute units:				 20
  Max work items dimensions:			 3
    Max work items[0]:				 1024
    Max work items[1]:				 1024
    Max work items[2]:				 1024
  Max work group size:				 256
  Preferred vector width char:			 4
  Preferred vector width short:			 2
  Preferred vector width int:			 1
  Preferred vector width long:			 1
  Preferred vector width float:			 1
  Preferred vector width double:		 1
  Native vector width char:			 4
  Native vector width short:			 2
  Native vector width int:			 1
  Native vector width long:			 1
  Native vector width float:			 1
  Native vector width double:			 1
  Max clock frequency:				 2200Mhz
  Address bits:					 64
  Max memory allocation:			 7287183768
  Image support:				 Yes
  Max number of images read arguments:		 128
  Max number of images write arguments:		 8
  Max image 2D width:				 16384
  Max image 2D height:				 16384
  Max image 3D width:				 16384
  Max image 3D height:				 16384
  Max image 3D depth:				 8192
  Max samplers within kernel:			 29471
  Max size of kernel argument:			 1024
  Alignment (bits) of base address:		 1024
  Minimum alignment (bytes) for any datatype:	 128
  Single precision floating point capability
    Denorms:					 Yes
    Quiet NaNs:					 Yes
    Round to nearest even:			 Yes
    Round to zero:				 Yes
    Round to +ve and infinity:			 Yes
    IEEE754-2008 fused multiply-add:		 Yes
  Cache type:					 Read/Write
  Cache line size:				 64
  Cache size:					 16384
  Global memory size:				 8573157376
  Constant buffer size:				 7287183768
  Max number of constant args:			 8
  Local memory type:				 Scratchpad
  Local memory size:				 65536
  Max pipe arguments:				 16
  Max pipe active reservations:			 16
  Max pipe packet size:				 2992216472
  Max global variable size:			 7287183768
  Max global variable preferred total size:	 8573157376
  Max read/write image args:			 64
  Max on device events:				 1024
  Queue on device max size:			 8388608
  Max on device queues:				 1
  Queue on device preferred size:		 262144
  SVM capabilities:				 
    Coarse grain buffer:			 Yes
    Fine grain buffer:				 Yes
    Fine grain system:				 No
    Atomics:					 No
  Preferred platform atomic alignment:		 0
  Preferred global atomic alignment:		 0
  Preferred local atomic alignment:		 0
  Kernel Preferred work group size multiple:	 32
  Error correction support:			 0
  Unified memory for Host and Device:		 0
  Profiling timer resolution:			 1
  Device endianess:				 Little
  Available:					 Yes
  Compiler available:				 Yes
  Execution capabilities:				 
    Execute OpenCL kernels:			 Yes
    Execute native function:			 No
  Queue on Host properties:				 
    Out-of-Order:				 No
    Profiling :					 Yes
  Queue on Device properties:				 
    Out-of-Order:				 Yes
    Profiling :					 Yes
  Platform ID:					 0x7f65a6148cf0
  Name:						 gfx1010
  Vendor:					 Advanced Micro Devices, Inc.
  Device OpenCL C version:			 OpenCL C 2.0 
  Driver version:				 3212.0 (HSA1.1,LC)
  Profile:					 FULL_PROFILE
  Version:					 OpenCL 2.0 
  Extensions:					 cl_khr_fp64 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_3d_image_writes cl_khr_byte_addressable_store cl_khr_fp16 cl_khr_gl_sharing cl_amd_device_attribute_query cl_amd_media_ops cl_amd_media_ops2 cl_khr_image2d_from_buffer cl_khr_subgroups cl_khr_depth_images cl_amd_copy_buffer_p2p cl_amd_assembly_program

[dipak]

Hi @develancer ,

I've whitelisted you and moved the post to "AMD OpenCL" forum.

Thanks.

[dipak]

Could you please build the kernel without optimization (i.e. set optimization flag " -cl-opt-disable" or "-O0" to clBuildProgram) to see if it produces correct result?

Thanks.

[develancer]

• -cl-opt-disable: correct
• -O0: correct
• -O1: correct
• -O2: incorrect

[dipak]

Thank you for the information. It seems to be a compiler optimization issue. I'll report it to the OpenCL team.

P.S. For ROCm related support, you may post your query or issue at its GitHub site here: https://github.com/RadeonOpenCompute/ROCm/issues

Thanks.

[develancer]

@dipak  I don't think it's ROCm-related, as I've had exactly same issues with pure AMDGPU-PRO drivers.

Can you link me somehow to this internal bug report?

I also have another issue in my main program resulting in errors also in -O0 mode. I'll try to extract it and submit.

[develancer]

@dipakI have just updated the test case and now it produces incorrect results also with all optimizations disabled!

The previous version of the test case is now on branch “previous”.

[dipak]

---

@develancer wrote:

I don't think it's ROCm-related, as I've had exactly same issues with pure AMDGPU-PRO drivers.

It mainly depends on the compiler and, I think, in this case the compiler toolchain is same for both the drivers. 

By the way, I just ran the latest code on Win10 with a Vega card and I observed correct result for both cases i.e. "with and without optimization". 

---

@develancer wrote:

Can you link me somehow to this internal bug report?

---

The bug tracking system is for internal use. We will let you know if there is any update on this issue.

Thanks.

[develancer]

@dipakOne last question:

Should I also report this bug on the ROCm's github? Would it help in fixing it?

PS. (unrelated)
Who should I write to if I don't receive my forum verification e-mail despite trying multiple times?

[dipak]

Spoiler

Should I also report this bug on the ROCm's github? Would it help in fixing it?

Please feel free to report it to the Github site. 

Spoiler

Who should I write to if I don't receive my forum verification e-mail despite trying multiple times?

Please send me a PM regarding this. I'll forward it to the concerned team.

Thanks.

[develancer]

@dipakOne more question, since you mentioned that you had run this code on Win10 with Vega card.

It looks like the current Radeon Pro drivers don't include the SDK. Therefore, I installed Microsoft Visual Studio 2019 enterprise and downloaded OCL_SDK_Light from github. I managed to compile the host code, but the call to clBuildProgram crashes.

What is the recommended (or, at least, working) way to compile OpenCL code on Win10 and recent GPU cards?

[dipak]

Yes,  on Windows, you need the OpenCL  SDK which provides the libraries and headers required for OpenCL development. I also used the OCL_SDK_Light available here: https://github.com/GPUOpen-LibrariesAndSDKs/OCL-SDK/releases

Regarding the crash on Radeon Pro, could you please try some other simple kernels to see if the crash happens for any kernel or it's specific to this kernel only? 

Also please share the clinfo output and more information about the setup.

Thanks.

[develancer]

@dipak  Sure. clinfo attached below.

The problem is specific to this kernel. The interesting part is that on Windows it only compiles with -O0, but then it does not seem to run correctly (the output is all zeros).

It does compile with default -O2 only if I comment lines 653-655 in file weird.cl. It seems that referencing r->y somehow breaks the compiler. Poor register management? The best part is that this commenting only helps with -O2, and affects neither -O1 nor -O0.

Number of platforms:				 1
  Platform Profile:				 FULL_PROFILE
  Platform Version:				 OpenCL 2.1 AMD-APP (3075.13)
  Platform Name:				 AMD Accelerated Parallel Processing
  Platform Vendor:				 Advanced Micro Devices, Inc.
  Platform Extensions:				 cl_khr_icd cl_khr_d3d10_sharing cl_khr_d3d11_sharing cl_khr_dx9_media_sharing cl_amd_event_callback cl_amd_offline_devices 


  Platform Name:				 AMD Accelerated Parallel Processing
Number of devices:				 1
  Device Type:					 CL_DEVICE_TYPE_GPU
  Vendor ID:					 1002h
  Board name:					 AMD Radeon RX 5700 XT
  Device Topology:				 PCI[ B#11, D#0, F#0 ]
  Max compute units:				 20
  Max work items dimensions:			 3
    Max work items[0]:				 1024
    Max work items[1]:				 1024
    Max work items[2]:				 1024
  Max work group size:				 256
  Preferred vector width char:			 4
  Preferred vector width short:			 2
  Preferred vector width int:			 1
  Preferred vector width long:			 1
  Preferred vector width float:			 1
  Preferred vector width double:		 1
  Native vector width char:			 4
  Native vector width short:			 2
  Native vector width int:			 1
  Native vector width long:			 1
  Native vector width float:			 1
  Native vector width double:			 1
  Max clock frequency:				 1935Mhz
  Address bits:					 64
  Max memory allocation:			 7059013632
  Image support:				 Yes
  Max number of images read arguments:		 128
  Max number of images write arguments:		 64
  Max image 2D width:				 16384
  Max image 2D height:				 16384
  Max image 3D width:				 2048
  Max image 3D height:				 2048
  Max image 3D depth:				 2048
  Max samplers within kernel:			 16
  Max size of kernel argument:			 1024
  Alignment (bits) of base address:		 2048
  Minimum alignment (bytes) for any datatype:	 128
  Single precision floating point capability
    Denorms:					 Yes
    Quiet NaNs:					 Yes
    Round to nearest even:			 Yes
    Round to zero:				 Yes
    Round to +ve and infinity:			 Yes
    IEEE754-2008 fused multiply-add:		 Yes
  Cache type:					 Read/Write
  Cache line size:				 64
  Cache size:					 16384
  Global memory size:				 8573157376
  Constant buffer size:				 7059013632
  Max number of constant args:			 8
  Local memory type:				 Scratchpad
  Local memory size:				 65536
  Max pipe arguments:				 16
  Max pipe active reservations:			 16
  Max pipe packet size:				 2764046336
  Max global variable size:			 6353112064
  Max global variable preferred total size:	 8573157376
  Max read/write image args:			 64
  Max on device events:				 1024
  Queue on device max size:			 8388608
  Max on device queues:				 1
  Queue on device preferred size:		 262144
  SVM capabilities:				 
    Coarse grain buffer:			 Yes
    Fine grain buffer:				 Yes
    Fine grain system:				 No
    Atomics:					 No
  Preferred platform atomic alignment:		 0
  Preferred global atomic alignment:		 0
  Preferred local atomic alignment:		 0
  Kernel Preferred work group size multiple:	 32
  Error correction support:			 0
  Unified memory for Host and Device:		 0
  Profiling timer resolution:			 1
  Device endianess:				 Little
  Available:					 Yes
  Compiler available:				 Yes
  Execution capabilities:				 
    Execute OpenCL kernels:			 Yes
    Execute native function:			 No
  Queue on Host properties:				 
    Out-of-Order:				 No
    Profiling :					 Yes
  Queue on Device properties:				 
    Out-of-Order:				 Yes
    Profiling :					 Yes
  Platform ID:					 00007FF870083FD0
  Name:						 gfx1010
  Vendor:					 Advanced Micro Devices, Inc.
  Device OpenCL C version:			 OpenCL C 2.0 
  Driver version:				 3075.13 (PAL,LC)
  Profile:					 FULL_PROFILE
  Version:					 OpenCL 2.0 AMD-APP (3075.13)
  Extensions:					 cl_khr_fp64 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_3d_image_writes cl_khr_byte_addressable_store cl_khr_fp16 cl_khr_gl_sharing cl_khr_gl_depth_images cl_amd_device_attribute_query cl_amd_media_ops cl_amd_media_ops2 cl_khr_d3d10_sharing cl_khr_d3d11_sharing cl_khr_dx9_media_sharing cl_khr_image2d_from_buffer cl_khr_subgroups cl_khr_gl_event cl_khr_depth_images cl_khr_mipmap_image cl_khr_mipmap_image_writes cl_amd_liquid_flash cl_amd_copy_buffer_p2p cl_amd_planar_yuv

[dipak]

Hi @develancer,

Thank you for sharing the above information.

As you said, it's specific to this kernel so it could be related to the compiler issue which you reported earlier. I already reported that issue to the OpenCL team. 

Please note, on my Windows setup where it worked fine, the compiler chain was HSAIL, whereas in your case it's the LC compiler. So I suspect the issue could be specific to the LC compiler chain used for Navi.

Thanks.

[develancer]

Hi @dipak,

Do we already have some timeframe for this fix?

[dipak]

Hi @develancer ,

I'll check with the concerned team if I can get any update on this issue. 

By the way, have you tried the latest drivers?

Thanks.

[develancer]

Hi @dipak 

As a matter of fact, I did. The example works only if the device code is compiled with -O0 or -cl-opt-disable.

Otherwise:

• on Linux (Ubuntu 20.04.2 + Radeon Software 21.10) it still produces incorrect results
• on Windows 10 (Radeon Software 21.5.2) it still crashes when compiling device code (in clBuildProgram)

[dipak]

Hi @develancer ,

As I've been informed, the issue is still under investigation and most likely the fix will be available with ROCm 4.4 as mentioned in the related ROCm issue page: Wrong int64 multiplication test case [https://github.com/RadeonOpenCompute/ROCm/issues/1383]

Thanks.

[colesdav]

Hope you havn't been using your AMD GPU for anything genetic engineering related...