3 Replies Latest reply on Apr 12, 2017 3:49 PM by jlavi

    Using OpenCL from AMDGPU-PRO in Linux without the rest

    jlavi

      In Linux, instead of installing the full AMDGPU-PRO package I've installed only the few shared librries needed for OpenCL support. The idea was first meantioned in Phoronix discissions in November. I need OpenCL to speed image processing and 3D modeling in Linux.

       

      The files are:

        /opt/amdgpu-pro/lib64/

        ├── libamdocl12cl64.so

        ├── libamdocl64.so

        ├── libdrm_amdgpu.so.1 -> libdrm_amdgpu.so.1.0.0

        ├── libdrm_amdgpu.so.1.0.0

        ├── libOpenCL.so -> libOpenCL.so.1

        └── libOpenCL.so.1

      and the ICD file at /etc/OpenCL/vendors/amdocl64.icd. I am using kernel 4.9.x which has its own amdgpu driver.

       

      After setting LD_LIBRARY_PATH to point to shared libraries, I have been able to use OpenCL programs like clinfo, MemtestCL,

      Darktable, Hashcat, Boinc and Photoscan and they all use OpenCL and compile and execute their own OpenCL routines. Everything seems to work alright.

       

      MemtestCL gave memory errors with random block with shared libaries from AMDGPU-PRO. There was a bug MemtestCL that caused these error but it was fixed in 2012 by adding a memory barrier into the test program. When I ran MemtestCL with libraries from AMDGPU-PRO 16.60, there were no memory errors

       

      Are there any speed penalties in this setup?  When trying to repeat the Phoronix OpenCL benchmark, some tests were on par and some lagged way behind.

        • Re: Using OpenCL from AMDGPU-PRO in Linux without the rest
          jlavi

          In Phoronix article "A Word Of Warning When Using AMDGPU-PRO On An Unsupported Kernel" (https://www.phoronix.com/scan.php?page=news_item&px=AMDGPU-PRO-Linux-4.8-DKMS) discussed the performance degrade if kernel misses the needed functionality. Quote: "Bridgman also explained that while the hybrid OpenGL and Vulkan driver components make use of ioctls not found in the mainline AMDGPU DRM code, they can still work albeit slower."

           

          After having done some further benchmarking. I wasn't able to run the Phoronix benchmark fully as an automated test but I was able to run all but one benchmark manually from the command line. My test setup is Xeon E3-1241 3.5 GHz, MSI RX-480 8gb Gaming X (1303 Mhz).

           

          The largest gainer and loser compared to Phoronix benchmark with RX480 reference card were:

          julianGPU 113365184 vs. 81972594 (+38%)

          SHPC BusSpeedReadback 12.68 vs. 14.20 (-11%)

           

          Many test ran 2-4% faster than the reference card which can be explained by ca. 3% higher clock on my RX480 board compared to reference board.

           

          This is quite close to reference card performance. I assume the performance difference would be greater and likely to negative direction if needed IOCTLs were missing from amdgpu in kernel 4.9.

           

          With older AMDGPU-PRO 16.40 MemtestCL gives false error on random block operation but no error with 16.60. The same false random block test error had been seen in 2012 when it was fixed by adding a barrier into the test loop. Possibly the barrier implementation was ineffective in 16.40 but fixed in 16.60?

          • Re: Using OpenCL from AMDGPU-PRO in Linux without the rest
            mheieis

            Hi

            I was able to get opencl working on Fedora 25 as per jlavi's outline above. However, after yesterday's OS update, opencl seg faults. For instance, I can no longer darktable with opencl enabled and clinfo seg faults. Any thoughts?

             

             

            ~~~~~~~~~~~~~~~~~~~

            Linux copernicus 4.10.6-200.fc25.x86_64 #1 SMP Mon Mar 27 14:06:23 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux fully updated.

            ~~~~~~~~~~~~~~~~~~~

            modules updated

             

                Install  kernel-4.10.8-200.fc25.x86_64                                  @updates

                Erase    kernel-4.9.14-200.fc25.x86_64                                  @updates

                Install  kernel-core-4.10.8-200.fc25.x86_64                              @updates

                Erase    kernel-core-4.9.14-200.fc25.x86_64                              @updates

                Install  kernel-debug-devel-4.10.8-200.fc25.x86_64                      @updates

                Erase    kernel-debug-devel-4.9.14-200.fc25.x86_64                      @updates

                Install  kernel-devel-4.10.8-200.fc25.x86_64                            @updates

                Erase    kernel-devel-4.9.14-200.fc25.x86_64                            @updates

                Upgraded kernel-headers-4.10.6-200.fc25.x86_64                          @updates

                Upgrade                4.10.8-200.fc25.x86_64                          @updates

                Install  kernel-modules-4.10.8-200.fc25.x86_64                          @updates

                Erase    kernel-modules-4.9.14-200.fc25.x86_64                          @updates

                Install  kernel-modules-extra-4.10.8-200.fc25.x86_64                    @updates

                Erase    kernel-modules-extra-4.9.14-200.fc25.x86_64                    @updates

                Erase    kmod-VirtualBox-4.9.14-200.fc25.x86_64-5.1.18-1.fc25.x86_64    @@commandline

                Upgraded liba52-0.7.4-25.fc25.x86_64                                    @updates

                Upgrade        0.7.4-26.fc25.x86_64                                    @updates

                Upgraded libdrm-2.4.75-1.fc25.i686                                      @updates

                Upgraded libdrm-2.4.75-1.fc25.x86_64                                    @updates

                Upgrade        2.4.76-1.fc25.i686                                      @updates

                Upgrade        2.4.76-1.fc25.x86_64                                    @updates

             

                Upgraded xorg-x11-server-Xorg-1.19.1-3.fc25.x86_64                      @updates

                Upgrade                      1.19.3-1.fc25.x86_64                      @updates

                Upgraded xorg-x11-server-common-1.19.1-3.fc25.x86_64                    @updates

                Upgrade                        1.19.3-1.fc25.x86_64                    @updates

            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`

            darktable backtrace:

            this is darktable darktable-2.2.3-1.fc25 reporting a segfault:

             

             

            #0  0x00007f440108cfdb in waitpid () at /lib64/libpthread.so.0

            #1  0x00007f44059c3c40 in _dt_sigsegv_handler () at /usr/bin/../lib64/darktable/libdarktable.so

            #2  0x00007f43fd902990 in <signal handler called> () at /lib64/libc.so.6

            #3  0x0000000000000000 in  ()

            #4  0x00007f43a05dfb89 in  () at /lib64/libamdocl64.so

            #5  0x00007f43a05e1afe in  () at /lib64/libamdocl64.so

            #6  0x00007f43a05d1b98 in  () at /lib64/libamdocl64.so

            #7  0x00007f43a02edf69 in  () at /lib64/libamdocl64.so

            #8  0x00007f43a02bc4d9 in  () at /lib64/libamdocl64.so

            #9  0x00007f43a02bc54f in  () at /lib64/libamdocl64.so

            #10 0x00007f43a02bd2c9 in  () at /lib64/libamdocl64.so

            #11 0x00007f43a02843d8 in  () at /lib64/libamdocl64.so

            #12 0x00007f43a02858c7 in  () at /lib64/libamdocl64.so

            #13 0x00007f43a0285a56 in  () at /lib64/libamdocl64.so

            #14 0x00007f43a024140e in  () at /lib64/libamdocl64.so

            #15 0x00007f43a025cc27 in  () at /lib64/libamdocl64.so

            #16 0x00007f43a022adbe in clIcdGetPlatformIDsKHR () at /lib64/libamdocl64.so

            #17 0x00007f43e264d76e in  () at /lib64/libOpenCL.so

            #18 0x00007f43e264f647 in  () at /lib64/libOpenCL.so

            #19 0x00007f440108aba9 in __pthread_once_slow () at /lib64/libpthread.so.0

            #20 0x00007f43e264dd31 in clGetPlatformIDs () at /lib64/libOpenCL.so

            #21 0x00007f44059d158b in dt_opencl_init () at /usr/bin/../lib64/darktable/libdarktable.so

            #22 0x00007f44059772be in dt_init () at /usr/bin/../lib64/darktable/libdarktable.so

            #23 0x00005559b8e9fb86 in main ()

             

             

            Thread 11 (Thread 0x7f43e3052700 (LWP 2431)):

            #0  0x00007f4401089460 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libpthread.so.0

            #1  0x00007f44059d965f in dt_control_work_res () at /usr/bin/../lib64/darktable/libdarktable.so

            #2  0x00007f44010836ca in start_thread () at /lib64/libpthread.so.0

            #3  0x00007f43fd9d4f7f in clone () at /lib64/libc.so.6

             

             

            Thread 10 (Thread 0x7f43e3853700 (LWP 2430)):

            #0  0x00007f4401089460 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libpthread.so.0

            #1  0x00007f44059d965f in dt_control_work_res () at /usr/bin/../lib64/darktable/libdarktable.so

            #2  0x00007f44010836ca in start_thread () at /lib64/libpthread.so.0

            #3  0x00007f43fd9d4f7f in clone () at /lib64/libc.so.6

             

             

            Thread 9 (Thread 0x7f43d3fff700 (LWP 2429)):

            #0  0x00007f43fd99981d in nanosleep () at /lib64/libc.so.6

            #1  0x00007f43fd99976a in sleep () at /lib64/libc.so.6

            #2  0x00007f44059d928a in dt_control_worker_kicker () at /usr/bin/../lib64/darktable/libdarktable.so

            #3  0x00007f44010836ca in start_thread () at /lib64/libpthread.so.0

            #4  0x00007f43fd9d4f7f in clone () at /lib64/libc.so.6

             

             

            Thread 8 (Thread 0x7f43e4054700 (LWP 2428)):

            #0  0x00007f4401089460 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libpthread.so.0

            #1  0x00007f44059da8be in dt_control_work () at /usr/bin/../lib64/darktable/libdarktable.so

            #2  0x00007f44010836ca in start_thread () at /lib64/libpthread.so.0

            #3  0x00007f43fd9d4f7f in clone () at /lib64/libc.so.6

             

             

            Thread 7 (Thread 0x7f43e4855700 (LWP 2427)):

            #0  0x00007f4401089460 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libpthread.so.0

            #1  0x00007f44059da8be in dt_control_work () at /usr/bin/../lib64/darktable/libdarktable.so

            #2  0x00007f44010836ca in start_thread () at /lib64/libpthread.so.0

            #3  0x00007f43fd9d4f7f in clone () at /lib64/libc.so.6

             

             

            Thread 6 (Thread 0x7f43e5056700 (LWP 2426)):

            #0  0x00007f4401089460 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libpthread.so.0

            #1  0x00007f44059da8be in dt_control_work () at /usr/bin/../lib64/darktable/libdarktable.so

            #2  0x00007f44010836ca in start_thread () at /lib64/libpthread.so.0

            #3  0x00007f43fd9d4f7f in clone () at /lib64/libc.so.6

             

             

            Thread 5 (Thread 0x7f43e5857700 (LWP 2425)):

            #0  0x00007f4401089460 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libpthread.so.0

            #1  0x00007f44059da8be in dt_control_work () at /usr/bin/../lib64/darktable/libdarktable.so

            #2  0x00007f44010836ca in start_thread () at /lib64/libpthread.so.0

            #3  0x00007f43fd9d4f7f in clone () at /lib64/libc.so.6

             

             

            Thread 4 (Thread 0x7f43e6058700 (LWP 2424)):

            #0  0x00007f4401089460 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libpthread.so.0

            #1  0x00007f44059da8be in dt_control_work () at /usr/bin/../lib64/darktable/libdarktable.so

            #2  0x00007f44010836ca in start_thread () at /lib64/libpthread.so.0

            #3  0x00007f43fd9d4f7f in clone () at /lib64/libc.so.6

             

             

            Thread 3 (Thread 0x7f43e6859700 (LWP 2423)):

            #0  0x00007f4401089460 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libpthread.so.0

            #1  0x00007f44059da8be in dt_control_work () at /usr/bin/../lib64/darktable/libdarktable.so

            #2  0x00007f44010836ca in start_thread () at /lib64/libpthread.so.0

            #3  0x00007f43fd9d4f7f in clone () at /lib64/libc.so.6

             

             

            Thread 2 (Thread 0x7f43e705a700 (LWP 2422)):

            #0  0x00007f4401089460 in pthread_cond_wait@@GLIBC_2.3.2 () at /lib64/libpthread.so.0

            #1  0x00007f44059da8be in dt_control_work () at /usr/bin/../lib64/darktable/libdarktable.so

            #2  0x00007f44010836ca in start_thread () at /lib64/libpthread.so.0

            #3  0x00007f43fd9d4f7f in clone () at /lib64/libc.so.6

             

             

            Thread 1 (Thread 0x7f4405efe680 (LWP 2421)):

            #0  0x00007f440108cfdb in waitpid () at /lib64/libpthread.so.0

            #1  0x00007f44059c3c40 in _dt_sigsegv_handler () at /usr/bin/../lib64/darktable/libdarktable.so

            #2  0x00007f43fd902990 in <signal handler called> () at /lib64/libc.so.6

            #3  0x0000000000000000 in  ()

            #4  0x00007f43a05dfb89 in  () at /lib64/libamdocl64.so

            #5  0x00007f43a05e1afe in  () at /lib64/libamdocl64.so

            #6  0x00007f43a05d1b98 in  () at /lib64/libamdocl64.so

            #7  0x00007f43a02edf69 in  () at /lib64/libamdocl64.so

            #8  0x00007f43a02bc4d9 in  () at /lib64/libamdocl64.so

            #9  0x00007f43a02bc54f in  () at /lib64/libamdocl64.so

            #10 0x00007f43a02bd2c9 in  () at /lib64/libamdocl64.so

            #11 0x00007f43a02843d8 in  () at /lib64/libamdocl64.so

            #12 0x00007f43a02858c7 in  () at /lib64/libamdocl64.so

            #13 0x00007f43a0285a56 in  () at /lib64/libamdocl64.so

            #14 0x00007f43a024140e in  () at /lib64/libamdocl64.so

            #15 0x00007f43a025cc27 in  () at /lib64/libamdocl64.so

            #16 0x00007f43a022adbe in clIcdGetPlatformIDsKHR () at /lib64/libamdocl64.so

            #17 0x00007f43e264d76e in  () at /lib64/libOpenCL.so

            #18 0x00007f43e264f647 in  () at /lib64/libOpenCL.so

            #19 0x00007f440108aba9 in __pthread_once_slow () at /lib64/libpthread.so.0

            #20 0x00007f43e264dd31 in clGetPlatformIDs () at /lib64/libOpenCL.so

            #21 0x00007f44059d158b in dt_opencl_init () at /usr/bin/../lib64/darktable/libdarktable.so

            #22 0x00007f44059772be in dt_init () at /usr/bin/../lib64/darktable/libdarktable.so

            #23 0x00005559b8e9fb86 in main ()

              • Re: Using OpenCL from AMDGPU-PRO in Linux without the rest
                jlavi

                I have seen clinfo giving segfault when some symbolic link to shared library file is missing. Doing a segfault is a bad error message or a bug -- it is crashing after all. In a perfect world it would give a human readable error message about missing file.

                 

                Just to give credit about the idea of using few shared libs for OpenCL, I think it was first mentioned in Phoronix discussion half a year ago. Arch Linux seems to have their own install script for doing this hack.

                 

                When using 4.9 kernel I had some issues with suspend/resume. I spotted amdgpu errors like "*ERROR* suspend of IP block <uvd_v6_0> failed", "*ERROR* amdgpu_resume failed" and "*ERROR* amdgpu: ring 12 test failed". Testing with MemtestCL failed all the tests not because the tests couldn't be run.

                 

                I suspect the errors were due to mismatch between firmware and kernel versions. The firmware versions came from Debian package firmware-amd-graphics version 20161130-2. Polaris10 files are dated Dec 14. The firmware files in 16.60 dkms package were dated Dec 15.

                 

                Instead of trying different firmware versions I tried switching to 4.10 kernel. I have done several suspends and resumes during over a week now, I haven't seen yet those restore/suspend errors I saw with 4.9 kernel.