OpenCL

[matszpk]

I wrote known Radeon Assembler called as CLRadeonExtender. Year ago, I added support for ROCm binary format (I got sample binaries by using 'cloc' utility). I have doubts whether that format (HSACO format) is still used by ROCm OpenCL driver. Or maybe, does  ROCm OpenCL use this same format like legacy AMDGPU-PRO OpenCL or Windows AMD OpenCL implementation? Unfortunatelly, I can't use ROCm on my computer, because my hardware is not compatible with ROCm. Dear users of the AMD DevGurus, can you help me an you give me sample binaries of the OpenCL programs built under a ROCm OpenCL? I would like to verify my supposition that ROCm binary format (HSACO) handled by my assembler is correct for current ROCm OpenCL implementation. For checking, I attached a sample binary (for ROCm 1.3 built by 'cloc' utility).

EDIT: I would like to see ROCm OpenCL binary built by its driver (not any third party tool) by using some offline compiler like GitHub - Maratyszcza/clcc: OpenCL offline compiler that uses clBuildProgram and clGetProgramInfo to get binaries. And second question: Is ROCm OpenCL binary format same as binary format used by AMDGPU-PRO ROCm OpenCL used for RX VEGA?

[sp314]

Hi there,

I've used your assembler before, and it is a very nice tool! Thanks for making it! Naturally, I'd like to help you if I can.

Throwing together a quick C program that compiles and saves the binary is not a problem, however... I've got ROCm with AMDGPU-PRO drivers. As I understand it, this isn't the driver you want, so can you or anybody else tell me how to switch back to the original ROCm driver, or, even better, compile with ROCm offline, without switching the driver? (I use this computer for work, so I can't uninstall and reinstall too much)

Alternatively, will the code compiled offline with something along the lines of

    /opt/rocm/opencl/bin/x86_64/clang -target amdgcn-amd-amdhsa-opencl -mcpu gfx803 testKernel.cl

help you? If so, I can do it real quick and send you the outputs for any OpenCL kernels that you want me to try.

Best,

sp

[matszpk]

Thank you for fast answer. Ofcourse, I will try compile offline my samples by using clang (I have on Ubuntu the driver and I can install ROCm platform). For me, doesn't matter whether it is the ROCm platform or AMDGPU-PRO with ROCm for VEGA (you can try use ROCm from AMDGPU-PRO and it will be recommend, because I want to support VEGA on current AMDGPU-PRO drivers). If you can confirm that it is same binary format as that used by clang from ROCm I will grateful.

[sp314]

Oh, I'm sorry, I didn't realize that you're asking specifically about Vega. I must have missed that part. I don't have a Vega, so in this case, I can't compile with the driver for it, my apologies. I hope somebody else can help you (or, once I get access to a Vega, I'll do it, but it may take a while and you'll probably have already figured it out by that time).

[matszpk]

No problem. If you can compile some examples under a ROCm OpenCL for any device, It is ok. I want only some binaries generated by ROCm OpenCL, if it is possible.

[sp314]

So, just to confirm, I'll be compiling OpenCL kernels on linux with ROCm and AMDGPU-PRO drivers, using clBuildProgram and clProgramGetInfo to get the binaries. If that helps you, I'll do it later today, no problem. Is that OK?

[matszpk]

Very thank you. This is that what I would.

[sp314]

Hi,

as promised, I quickly put together a utility that compiles OpenCL code and extracts the binary using clBuildProgram() and clGetProgramInfo(), as you requested, and ran it on a simple .cl kernel on a machine with ROCm. Please see the attached .zip file. There's a .cl kernel and the corresponding binary that I've got in the .bin file in the archive. If you want me to change anything or to run this on more .cl sources, I can do it.

I've also included a minimalistic C++ source that compiles and saves the binary, plus a build script for it (based on the OpenCL build example at https://github.com/RadeonOpenCompute/ROCm). Perhaps, somebody with access to Vega and a computer with ROCm and with/without AMDGPU-PRO can use it to test more binaries for you on different cards with minimal effort, or perhaps somebody can spot an error in this code or correct me if I'm completely wrong.

To provide some more info... I'm running Ubuntu 16.04 with ROCm kernel, uname -a prints Linux linuxpc 4.11.0-kfd-compute-rocm-rel-1.6-180 #1 SMP Tue Oct 10 08:15:38 CDT 2017 x86_64 x86_64 x86_64 GNU/Linux. The output of /opt/rocm/opencl/bin/x86_64/clinfo > clinfo_output.txt in the archive too. This is all on a Ryzen 3 with an R290x card and AMDGPU-PRO 17.40 drivers (but I made sure to compile the binary for the GPU, not the CPU).

Your CLRX has helped me a lot, thanks for it again. I can't 100% guarantee that I understood your question and did everything correctly, but I hope this code and the binary I've got is what you needed. I hope this helps.

Best,

sp

[matszpk]

Thank you for efforts. However, your delivered results in package is for 'legacy' AMDGPU-PRO OpenCL driver. I recognized it in delivered clinfo output. clinfo under the ROCm-OpenCL just prints:

'Driver version: 1.1 (HSA, LC)' (where is HSA,LC is important)

and

'Device OpenCL C Version: OpenCL C 2.0'

Maybe you have working ROCm, but without a ROCm-OpenCL package or you just forgot about switching between the OpenCL to a ROCm-OpenCL (just change ICD in /etc/OpenCL/vendors). Maybe somebody who have any working ROCm-OpenCL and new graphics card could help me.

Dear users of the AMD Devgurus, I encourange to use delivered code by sp314 to generate an OpenCL program binary under a ROCm-OpenCL or an AMDGPU-PRO or VEGA. Any help will be appreciated.

[sp314]

Hi matszpk,

I've managed to get the ROCm OpenCL to work (temporarily), clinfo now prints Driver version: 2545.0 (HSA1.1,LC), among other things, and I've generated a new binary for you. Please see attached.

Is this what you needed and does this help? I'll have to go back to AMDGPU-pro sometime soon, but in the meantime, if this is what you need, you can send me a few kernels and I'll compile them for you, if you want.

Best,

sp

[matszpk]

Yes. This is the ROCm OpenCL binary. Thank you for any help. Currently,  I am working over reading/writing metadata which is attached in these binaries and full support for this binary format in CLRX assembler will be ready soon.

[sp314]

Great, I'm glad I could help.

Somewhat related, I'll eventually have to switch from ROCm to regular OpenCL, and I was planning on using your assembler (I've used it before). Would you mind me tagging you in the posts on this forum if I have questions? I'm not quite sure how to set up the kernel parameters, such as VGPR and scalar register counts, and the amount of LDS that I use, things along those lines, but I'll post it later and in a separate thread.

[matszpk]

Hi. I wrote ROCm support in my the CLRadeonExtender ( in an assembler and a disassembler). Also, I wrote preliminary sample (CLRadeonExtender source code, in samples directory) 'VectorAdd' (vector add in an assembly language) which could be work under ROCm-OpenCL. Can anybody test this sample under ROCm-OpenCL environment? I will be very glad if anybody will test this sample program under that environment. If you want to test this sample, just download latest trunk (or master on github) of a CLRadeonExtender source code and build it (to build samples you need to enable BUILD_SAMPLES option while calling cmake: for example by executing 'cmake .. -DBUILD_SAMPLES=ON ...') and try to run './samples/VectorAdd'. I can not do it, because I still don't have required hardware (mainly mainboard with the PCIE atomics support) to do it.

WARNING: This sample can work incorrectly and it can hang up device due to badly generated binary code.

[matszpk]

Hi. I found way to compile OpenCL code under ROCm-OpenCL without using driver. I installed ROCm-OpenCL on my Ubuntu and I prepared script from project GitHub - RadeonOpenCompute/clang-ocl: OpenCL compilation with clang compiler.​ And it works enough good. The generated binary is readable by my clrxdisasm after small fix without any problems. This way requires only ROCm-OpenCL package without rest of the ROCm stuff. I attached sample binaries (compiled for device 'gfx803' to target 'amdgcn-amd-amdhsa-amdgizcl') to compare with binaries generated by working ROCm-OpenCL driver.