OpenCL

[avinashkrc]

I am trying to testout how well atomicity performs on APU. But my sample program does not update the variable properly hence whole system hangs as I check for updated value at either side (cpu and gpu)  in while loop and it will continue to loop until other side updates the variable.

HOST program:

 int *data =(int*)clSVMAlloc(context, CL_MEM_READ_WRITE|CL_MEM_SVM_FINE_GRAIN_BUFFER|CL_MEM_SVM_ATOMICS , sizeof(int)*256,0);

int t = 2,ind;
        for(ind = 0; ind < 256; ind++) {
               atomic_store(&data[ind], 0);
        }

 status = clEnqueueNDRangeKernel(cmdQueue, kernel, 1, NULL, indexSpaceSize, workGroupSize, 0, NULL, &kernelExeEvent);

atomic_store(p, 1);
    while(t--) {
        printf("value of t = %d\n", t);
        for(ind = 0; ind < 255; ind++) {
                printf("first load %d\n", atomic_load(&data[ind]));
               while(atomic_load(&data[ind]) != 1);
               atomic_store(&data[ind], 0);
                printf("value of ind = %d, %d\n", t, ind);
        }
    }
    clWaitForEvents(1, &kernelExeEvent);

GPU kernel:

__kernel void simple(volatile __global atomic_int *A, __global atomic_int *p) {
    int in = 0, index = 0;
    while(in || atomic_load(p) == 1) {
        in = 1;
        while(atomic_load_explicit(&A[get_global_id(0)], memory_scope_work_group, memory_scope_all_svm_devices) !=0);
        atomic_store_explicit(&A[get_global_id(0)], 1, memory_order_seq_cst, memory_scope_all_svm_devices);
        index++;
        if(index > 1)
            break;
    }
}

[cardboard]

what's your device? (some APUs don't have SVM atomics (A8-7100 APU) )

I'm not experienced and probably can't answer this but
"the variable" ?
your kernel is a little confusing to analyse with such vague guidance
maybe a little annotation of what its trying to perform would help, I can understand the kernel somewhat, but compartmentalizing your code might help separate the where communication slowdown occurs

most things I've read say that this is basically impossible in most cases, please simply check if its possible rather than debugging first.

[avinashkrc]

By variable I mean every index location of data. My APU is A12-9800 and it does support atomics.

I was trying to understand cost of atomics operation for APU.

So plan was to set ( store 1) an array of integer (size 256 = number of gpu threads) by gpu if it finds that corresponding value at index ind of data is unset. However cpu thread will wait for each index ind of data to be set, once data at ind is set then cpu will unset it.

Some of the indexes gets updated but some doesn't thats where the problem lies. Since gpu also wait in while loop for value at given index to be unset, it hangs from which i can infer that atomics are not working properly.

System config: APU A12-9800

Ubuntu 14.04

opencl version 2.0

driver: fglrx

[dipak]

Hi Avinash,

Could you please share the clinfo output, driver details and complete source code that reproduces the issue? 

Some parts are missing in the above example. Also, some codes seem incorrect to me. For example, the 2nd argument of atomic_load_explicit should be memory ordering constraints not the memory_scope

while(atomic_load_explicit(&A[get_global_id(0)], memory_scope_work_group, memory_scope_all_svm_devices) !=0);

Thanks.

[avinashkrc]

Hi dipak, complete source code is attached with the original question (i updated the original question to attach the source codes). Clinfo is below:

Number of platforms:                 1
  Platform Profile:                 FULL_PROFILE
  Platform Version:                 OpenCL 2.0 AMD-APP (1912.5)
  Platform Name:                 AMD Accelerated Parallel Processing
  Platform Vendor:                 Advanced Micro Devices, Inc.
  Platform Extensions:                 cl_khr_icd cl_amd_event_callback cl_amd_offline_devices

  Platform Name:                 AMD Accelerated Parallel Processing
Number of devices:                 2
  Device Type:                     CL_DEVICE_TYPE_GPU
  Vendor ID:                     1002h
  Board name:                    
  Device Topology:                 PCI[ B#0, D#1, F#0 ]
  Max compute units:                 8
  Max work items dimensions:             3
    Max work items[0]:                 256
    Max work items[1]:                 256
    Max work items[2]:                 256
  Max work group size:                 256
  Preferred vector width char:             4
  Preferred vector width short:             2
  Preferred vector width int:             1
  Preferred vector width long:             1
  Preferred vector width float:             1
  Preferred vector width double:         1
  Native vector width char:             4
  Native vector width short:             2
  Native vector width int:             1
  Native vector width long:             1
  Native vector width float:             1
  Native vector width double:             1
  Max clock frequency:                 1107Mhz
  Address bits:                     64
  Max memory allocation:             594742272
  Image support:                 Yes
  Max number of images read arguments:         128
  Max number of images write arguments:         64
  Max image 2D width:                 16384
  Max image 2D height:                 16384
  Max image 3D width:                 2048
  Max image 3D height:                 2048
  Max image 3D depth:                 2048
  Max samplers within kernel:             16
  Max size of kernel argument:             1024
  Alignment (bits) of base address:         2048
  Minimum alignment (bytes) for any datatype:     128
  Single precision floating point capability
    Denorms:                     No
    Quiet NaNs:                     Yes
    Round to nearest even:             Yes
    Round to zero:                 Yes
    Round to +ve and infinity:             Yes
    IEEE754-2008 fused multiply-add:         Yes
  Cache type:                     Read/Write
  Cache line size:                 64
  Cache size:                     16384
  Global memory size:                 2378969088
  Constant buffer size:                 65536
  Max number of constant args:             8
  Local memory type:                 Scratchpad
  Local memory size:                 32768
  Max pipe arguments:                 16
  Max pipe active reservations:             16
  Max pipe packet size:                 594742272
  Max global variable size:             535267840
  Max global variable preferred total size:     2378969088
  Max read/write image args:             64
  Max on device events:                 1024
  Queue on device max size:             8388608
  Max on device queues:                 1
  Queue on device preferred size:         262144
  SVM capabilities:                
    Coarse grain buffer:             Yes
    Fine grain buffer:                 Yes
    Fine grain system:                 No
    Atomics:                     Yes
  Preferred platform atomic alignment:         0
  Preferred global atomic alignment:         0
  Preferred local atomic alignment:         0
  Kernel Preferred work group size multiple:     64
  Error correction support:             0
  Unified memory for Host and Device:         1
  Profiling timer resolution:             1
  Device endianess:                 Little
  Available:                     Yes
  Compiler available:                 Yes
  Execution capabilities:                
    Execute OpenCL kernels:             Yes
    Execute native function:             No
  Queue on Host properties:                
    Out-of-Order:                 No
    Profiling :                     Yes
  Queue on Device properties:                
    Out-of-Order:                 Yes
    Profiling :                     Yes
  Platform ID:                     0x7f2b59ca5a18
  Name:                         Carrizo
  Vendor:                     Advanced Micro Devices, Inc.
  Device OpenCL C version:             OpenCL C 2.0
  Driver version:                 1912.5 (VM)
  Profile:                     FULL_PROFILE
  Version:                     OpenCL 2.0 AMD-APP (1912.5)
  Extensions:                     cl_khr_fp64 cl_amd_fp64 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_3d_image_writes cl_khr_byte_addressable_store cl_khr_gl_sharing cl_khr_gl_depth_images cl_ext_atomic_counters_32 cl_amd_device_attribute_query cl_amd_vec3 cl_amd_printf cl_amd_media_ops cl_amd_media_ops2 cl_amd_popcnt cl_khr_image2d_from_buffer cl_khr_spir cl_khr_subgroups cl_khr_gl_event cl_khr_depth_images cl_khr_mipmap_image cl_khr_mipmap_image_writes

  Device Type:                     CL_DEVICE_TYPE_CPU
  Vendor ID:                     1002h
  Board name:                    
  Max compute units:                 4
  Max work items dimensions:             3
    Max work items[0]:                 1024
    Max work items[1]:                 1024
    Max work items[2]:                 1024
  Max work group size:                 1024
  Preferred vector width char:             16
  Preferred vector width short:             8
  Preferred vector width int:             4
  Preferred vector width long:             2
  Preferred vector width float:             8
  Preferred vector width double:         4
  Native vector width char:             16
  Native vector width short:             8
  Native vector width int:             4
  Native vector width long:             2
  Native vector width float:             8
  Native vector width double:             4
  Max clock frequency:                 3800Mhz
  Address bits:                     64
  Max memory allocation:             8158393344
  Image support:                 Yes
  Max number of images read arguments:         128
  Max number of images write arguments:         64
  Max image 2D width:                 8192
  Max image 2D height:                 8192
  Max image 3D width:                 2048
  Max image 3D height:                 2048
  Max image 3D depth:                 2048
  Max samplers within kernel:             16
  Max size of kernel argument:             4096
  Alignment (bits) of base address:         1024
  Minimum alignment (bytes) for any datatype:     128
  Single precision floating point capability
    Denorms:                     Yes
    Quiet NaNs:                     Yes
    Round to nearest even:             Yes
    Round to zero:                 Yes
    Round to +ve and infinity:             Yes
    IEEE754-2008 fused multiply-add:         Yes
  Cache type:                     Read/Write
  Cache line size:                 64
  Cache size:                     32768
  Global memory size:                 32633573376
  Constant buffer size:                 65536
  Max number of constant args:             8
  Local memory type:                 Global
  Local memory size:                 32768
  Max pipe arguments:                 16
  Max pipe active reservations:             16
  Max pipe packet size:                 3863426048
  Max global variable size:             1879048192
  Max global variable preferred total size:     1879048192
  Max read/write image args:             64
  Max on device events:                 0
  Queue on device max size:             0
  Max on device queues:                 0
  Queue on device preferred size:         0
  SVM capabilities:                
    Coarse grain buffer:             No
    Fine grain buffer:                 No
    Fine grain system:                 No
    Atomics:                     No
  Preferred platform atomic alignment:         0
  Preferred global atomic alignment:         0
  Preferred local atomic alignment:         0
  Kernel Preferred work group size multiple:     1
  Error correction support:             0
  Unified memory for Host and Device:         1
  Profiling timer resolution:             1
  Device endianess:                 Little
  Available:                     Yes
  Compiler available:                 Yes
  Execution capabilities:                
    Execute OpenCL kernels:             Yes
    Execute native function:             Yes
  Queue on Host properties:                
    Out-of-Order:                 No
    Profiling :                     Yes
  Queue on Device properties:                
    Out-of-Order:                 No
    Profiling :                     No
  Platform ID:                     0x7f2b59ca5a18
  Name:                         AMD A12-9800 RADEON R7, 12 COMPUTE CORES 4C+8G
  Vendor:                     AuthenticAMD
  Device OpenCL C version:             OpenCL C 1.2
  Driver version:                 1912.5 (sse2,avx,fma4)
  Profile:                     FULL_PROFILE
  Version:                     OpenCL 1.2 AMD-APP (1912.5)
  Extensions:                     cl_khr_fp64 cl_amd_fp64 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_3d_image_writes cl_khr_byte_addressable_store cl_khr_gl_sharing cl_ext_device_fission cl_amd_device_attribute_query cl_amd_vec3 cl_amd_printf cl_amd_media_ops cl_amd_media_ops2 cl_amd_popcnt cl_khr_spir cl_khr_gl_event

driver is fglrx

2nd argument of atomic_load_explicit is corrected thanks for pointing out, but problem persists. Please check attached source code.

[dipak]

I've couple of suggestions:

• Add a clFlush() after the clEnqueueNDRangeKernel call.
• The program assumes that the host-side statement "atomic_store((atomic<int> *)p, 1)" will be executed before the kernel starts. Though it is most likely to happen, however, a waiting condition (e.g. "while(atomic_load(p) == 0);") can be added to the kernel to avoid a hang. 

Please note that fglrx series drivers are very old and I think they are already under legacy support model. So, no future release is expected for this driver.

Thanks,

[avinashkrc]

Hi dipak‌,

• Implemented your suggestion of clFlush() but problem persists.
• I did that and problem still persists.

What kind of driver would you suggest for me to use on ubuntu 14.04 with AMD-APP_SDK 3.0?

[dipak]

With the 1st change that I suggested, the code seems running fine on my Carrizo laptop on Windows 10 with Adrenalin 19.4.1.  

What kind of driver would you suggest for me to use on ubuntu 14.04 with AMD-APP_SDK 3.0?

It seems there is no latest driver available for the above configuration.  The fglrx/catalyst drivers, which used to work with the configuration, are legacy now. As per this driver download page a12-9800-apu, no driver is listed there for Linux. So, I assume the AMDGPU-Pro drivers do not support this APU yet. Also, APP SDK is no longer officially supported. Now, on Linux, libraries come with the driver package itself.

Thanks.

[dipak · Accepted solution]

By the way, I made another change that I forget to mention. It was the main reason for failing. In the original code, the host-side was not updating all the elements. Please modify the code as shown below:

for(ind = 0; ind < 255; ind++) {                                    ----> it should be < 256 or <= 255 otherwise the last element does not get updated and causing the GPU to wait forever
printf("first load %d\n", atomic_load(&data[ind]));
while(atomic_load(&data[ind]) != 1);
atomic_store(&data[ind], 0);
printf("value of ind = %d, %d\n", t, ind);
}

 Thanks.

[avinashkrc]

sorry, i don't know how i missed it. Thanks for correcting.