cancel
Showing results for 
Search instead for 
Did you mean: 

Archives Discussions

valerioa
Adept II

ioctl permissions on Linux

I'm running OpenCL AMD-APP-SDK-v2.8-lnx64.tgz with FirePro_9.003.3-Linux-x32x64-151130.zip on 4 FirePro 3D V5800

Device 0 : Juniper Device ID is 0x1bb83f0

Device 1 : Juniper Device ID is 0x1c88580

Device 2 : Juniper Device ID is 0x1c86c40

Device 3 : Juniper Device ID is 0x2035230

root can execute all OpenCL sample fine, but any regular user can't. Even extending permissions to anybody on /dev/aticard*

sudo chmod -fR 777 /dev/ati

a non-root user, shows with strace

stat("/dev/ati", {st_mode=S_IFDIR|0777, st_size=120, ...}) = 0

stat("/dev/ati/card0", {st_mode=S_IFCHR|0777, st_rdev=makedev(250, 0), ...}) = 0

open("/dev/ati/card0", O_RDWR)          = 6

ioctl(6, 0x80046484, 0x7fff2699714c)    = 0

ioctl(6, 0xc0106407, 0x7fff26997260)    = -1 EINVAL (Invalid argument)

ioctl(6, 0xc0106401, 0x7fff26997260)    = 0

ioctl(6, 0xc0106401, 0x7fff26997260)    = 0

ioctl(6, 0x8204648b, 0x7fff26997640)    = 0

ioctl(6, 0x80146454, 0x7fff26997850)    = -1 EACCES (Permission denied)

ioctl(6, 0x80146454, 0x7fff26997850)    = -1 EACCES (Permission denied)

ioctl(6, 0x80146454, 0x7fff26997850)    = -1 EACCES (Permission denied)

ioctl(6, 0x80146454, 0x7fff26997850)    = -1 EACCES (Permission denied)

Have premissions on ioctl()'s on card recently changed? I was able to execute OpenCL as regular user fine with previous versions of fglrx/OpenCL.

Is that the desired behavior? Will only root be able to run OpenCL programs on headless servers?

Thanks,

Valerio


0 Likes
31 Replies
himanshu_gautam
Grandmaster

Can you tell us more about your system configuration? Like which linux flavor, linux version etc..

Also, Are you trying to run APP SDK Samples? If so, can you check the permissions on the binaries?

It is possible that you installed the new APP SDK as root and some permission settings on the files are not allowing normal users to run... Can you check on these lines?

0 Likes

Distro: Welcome to Ubuntu 12.10 (GNU/Linux 3.5.0-22-generic x86_64)

Kernel: Linux hotshot 3.5.0-22-generic #34-Ubuntu SMP Tue Jan 8 21:47:00 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

CPU:AMD FX(tm)-8150 Eight-Core Processor

GPU:

* 0. 01:00.0 ATI FirePro V5800 (FireGL V)

  1. 06:00.0 ATI FirePro V5800 (FireGL V)

  2. 07:00.0 ATI FirePro V5800 (FireGL V)

  3. 08:00.0 ATI FirePro V5800 (FireGL V)

I've tried with a 3.2.0 kernel, with the same result.

Permissions look correct, all executables and libraries are 755.

even if I setuid and setgid any executable, a non-root user cannot execute the sample programs:

valerio@hotshot:~$ ls -l /opt/AMDAPP/samples/opencl/bin/x86_64/FFT

-rwsr-sr-x 1 501 501 154737 Dec 27 22:24 /opt/AMDAPP/samples/opencl/bin/x86_64/FFT

valerio@hotshot:~$ /opt/AMDAPP/samples/opencl/bin/x86_64/FFT

Platform 0 : Advanced Micro Devices, Inc.

Original Input Real

15.3732 201.81 51.9855 89.2322 92.572 34.4675 96.2478 66.3863 11.345 225.168

Original Input Img

0.0600514 0.788318 0.203068 0.348563 0.361609 0.134639 0.375968 0.259322 0.0443163 0.879562

GPU not found. Falling back to CPU device

Platform found : Advanced Micro Devices, Inc.

Selected Platform Vendor : Advanced Micro Devices, Inc.

Device 0 : AMD FX(tm)-8150 Eight-Core Processor            Device ID is 0xf786a0

Build Options are : -x clc++

Executing kernel for 1 iterations

-------------------------------------------

Output real

131643 -1085.95 -997.15 -1791.52 532.119 1659.74 -166.27 969.692 1189.76 -862.707

Output img

514.23 2289.84 936.49 -603.839 699.7 1018.18 1900.06 795.439 -1328.03 -293.334

My feeling is that these are permission checks  are performed inside firegl_compat_ioctl() or downstram of it, inside the fglrx module; they might not be related to any filesystem permissions.


0 Likes

APP SDK 2.8 requires catalyst version 12.10 (or) later.

You have to first install the driver followed by the APP SDK.

I can't read the catalyst version from FirePro_9.003.3-Linux-x32x64-151130.zip

Can you please let me know what is the catalyt version you are using?

0 Likes

Interestingly, I encountered this problem while updating my driver to catalyst 13.1 (9.003).

clinfo did not list the GPU at all.... after the update and an strace revealed a similar user-permission problem.

After downgrading to 12.10 (9.002) -- It started working fine.

So, this may be a workaround for you. But, I wont advocate this because I have not tested much.

But, After downgrading, "clinfo" worked just fine.

If you decide to un-install, you could run the script "/usr/share/ati/fglrx-uninstall.sh".

But, I leave the decision to you. I can't back this decision fully at the moment.

0 Likes

Hello Himanshu,

thank you for looking into this. FirePro 3D have no 12.10 catalyst that I can see. The previous package for Firepro #d according to this page: http://support.amd.com/us/gpudownload/fire/previous/Pages/fire_linux.aspx is this:

http://www2.ati.com/drivers/firepro/FirePro_8.982.8.3_Linux_x32x64-148581.zip

8.982.8.2 does not compile with kernel 3.5.0

Is there a 12.10 package for firepro? This:

http://www2.ati.com/drivers/linux/amd-driver-installer-catalyst-12.10-x86.x86_64.zip

will compile, but will not detect the firepro cards.

Valerio

0 Likes

Hi Valerio,

AFAIK, The generic driver amd-driver-installer-catalyst-12.10-x86.x86_64.zip  must work for firepro cards as well.

Not sure, why it does not work.

I will check with AMD Engg Team.

When you mean - "it does not detect" -- Do you mean "clinfo" does not list it? (or) the GPUs themselves are not detected by the OS?

Thanks,

0 Likes

clinfo or aticonfig --lsa will not list any of the Fire Pro Cards with catalyst 12.10

0 Likes

Hi,

I tried to reproduce the problem after making a fresh install of Linux, followed by 13.1 driver installation , followed by 2.8 APP SDK installation.

Everything is sane and normal. I don't see permission errors.

Did the problems started occuring after you upgraded your driver? (or) Did you do a fresh install of the system.

If possible, Can you share the exact sequence that led to the problem? It will be useful for me to reproduce the problem here.

Thanks,

0 Likes

HI Himanshu,

thank you very much for your help.

Here are my steps:

1) I was running ubuntu 11.0 with  the following drivers

./ati/fglrx-8.911.3.1/fglrx-amdcccle_8.911-0ubuntu1_amd64.deb

./ati/fglrx-8.911.3.1/fglrx-dev_8.911-0ubuntu1_amd64.deb

./ati/fglrx-8.911.3.1/fglrx_8.911-0ubuntu1_amd64.deb

and AMD-APP-SDK-v2.7-lnx64.tgz

All was working fine.

2) I updated ubuntu to 12.10 doing a 'do-release-update' twice

3) When in ubuntu 12.10, I uninstalled the SDK by

rm -fR /opt/AMDAPP

and

dpkg --remove fglrx fglrx-amdcccle fglrx-de

dpkg --purge fglrx fglrx-amdcccle fglrx-dev

4) reboot

5) Installed

./packages/fglrx/fglrx-9.003.3/fglrx_9.003-0ubuntu1_amd64.deb

./packages/fglrx/fglrx-9.003.3/fglrx-dev_9.003-0ubuntu1_amd64.deb

./packages/fglrx/fglrx-9.003.3/fglrx-amdcccle_9.003-0ubuntu1_amd64.deb

via

cd ./packages/fglrx

dpkg --install *deb

then I installed AMD-APP-SDK-v2.8-lnx64.tgz

Some details: this is a headless server, there are no monitors connected to the cards. Does it matter?

I tried having an Xorg running on the first card, and having a user with DISPLAY=:0.0 running the openCL programs, but the result was very much the same.

I have lightdm configured as such

[/etc/lightdn/lightdm.conf]

----------------------------------------------

[LightDM]

minimum-vt=-1

[SeatDefaults]

greeter-session=lightdm-gtk-greeter

user-session=gnome-classic

xserver-allow-tcp=true

xserver-command=Xvfb -screen 0 1900x1200x24 -I

[XDMCPServer]

enabled=true

--------------------------------------------------------------

so I can connect to the server via nomachine's nx server 4

0 Likes

Himanshu,

I think I found where the problem is.

probably somewhere in firegl_compat_ioctl() - or upstream of it, or downstream) there is some code similar to

case FIREGL_IOCTL_ID:

    if (! capable (CAP_SYS_ADMIN))

        return -EPERM;

    return do_something();

    break;

Indeed, if I do

sudo setcap cap_sys_admin+ep /opt/AMDAPP/samples/opencl/bin/x86_64/FFT

FFT works fine for a regular user:

valerio@hotshot:~$ /opt/AMDAPP/samples/opencl/bin/x86_64/FFT

Platform 0 : Advanced Micro Devices, Inc.

Original Input Real

15.3732 201.81 51.9855 89.2322 92.572 34.4675 96.2478 66.3863 11.345 225.168

Original Input Img

0.0600514 0.788318 0.203068 0.348563 0.361609 0.134639 0.375968 0.259322 0.0443163 0.879562

Platform found : Advanced Micro Devices, Inc.

Selected Platform Vendor : Advanced Micro Devices, Inc.

Device 0 : Juniper Device ID is 0x26f5f20

Device 1 : Juniper Device ID is 0x2786a40

Device 2 : Juniper Device ID is 0x27b32d0

Device 3 : Juniper Device ID is 0x291c440

Build Options are : -x clc++

Executing kernel for 1 iterations

-------------------------------------------

Output real

131643 -1085.95 -997.15 -1791.52 532.118 1659.74 -166.271 969.692 1189.76 -862.707

Output img

514.23 2289.84 936.489 -603.839 699.7 1018.18 1900.06 795.439 -1328.03 -293.334

It is not an optimal solution as some programs don't like to run with that capabilities, R being one

sudo setcap cap_sys_admin+ep /usr/local/pkg/R-2.15.2/lib64/R/bin/exec/R

valerio@hotshot:~$ /usr/local/pkg/R-2.15.2/lib64/R/bin/exec/R

/usr/local/pkg/R-2.15.2/lib64/R/bin/exec/R: error while loading shared libraries: libR.so: cannot open shared object file: No such file or directory

Now why capabilities are enforced on my kernel and not on yours, it's quite a mystery. Maye your kernel runs with drop_capabilities?


0 Likes

and the problem with R is that "setuid root" or "setcap cap_sys_admin+ep"  programs ignore LD_LIBRARY_PATH.

R uses LD_LIBRARY_PATH to find libR.so

0 Likes

Please give me sometime as I am trying out the repro case...

btw, You have said

"

1) I was running ubuntu 11.0 with  the following drivers

./ati/fglrx-8.911.3.1/fglrx-amdcccle_8.911-0ubuntu1_amd64.deb

./ati/fglrx-8.911.3.1/fglrx-dev_8.911-0ubuntu1_amd64.deb

./ati/fglrx-8.911.3.1/fglrx_8.911-0ubuntu1_amd64.deb

and AMD-APP-SDK-v2.7-lnx64.tgz

All was working fine.

"

I assume that the first one is to install CCC (Cat control center), second one is header files required for development and the third one is the actual driver.

Can you confirm this?

0 Likes

Hi valerioa,

I tried to reproduce your issue, but could not do it. Here is what I tried:

1. I had tried on a Ubuntu 12.04LTS (not 12.10). Not sure how stable 12.10 is. Also I have earlier faced issues with ubuntu 12.10 specifically when i upgraded to it using "do release upgrade".

2. I installed fglrx 9.003.3 driver downloaded from AMD's site, directly using the .run file. Rebooted. Installed AMD APP SDK 2.8. The sample's ran without any problem without any sudo privileges.

3. I removed this driver. Create debian packages out the .run file, using --buildpkg option. Installed those packages (just as you did). Again Installed AMD APP SDK 2.8. Rebooted. Sample again worked fine.

So, I am not sure where the problem is. I would recommend you to try doing it over a clean system once.

0 Likes

We met same issue on Ubuntu 12.04 clean install with Catalyst 13.1 for x86_64 and Radeon HD 7970. None except root can run any opencl code on GPU nor even clinfo shows anything except CPU. Running "strace clinfo" brings ioctl permission denied:

open("/dev/ati/card0", O_RDWR)          = 5

ioctl(5, 0xc0586450, 0x7fff12a83aa0)    = -1 EACCES (Permission denied)

While both /dev/ati and /dev/ati/card0 have correct priviledges for everyone

root@talvi:~# ls -la /dev/ati

total 0

drwxrwxrwx  2 root users     60 Feb  7 16:43 .

drwxr-xr-x 15 root root    4480 Feb  7 16:43 ..

crw-rw-rw-  1 root users 249, 0 Feb  7 16:43 card0

I've tried reinstall twice, also tried Catalyst 12.10 version, no effect.

We also run a headless node, just a server with neither X, nor monitor connected. Card is detected correctly

$ aticonfig --lsa

* 0. 04:00.0 AMD Radeon HD 7900 Series

$ lspci | grep VGA

04:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Tahiti XT [Radeon HD 7970]

Is here some small bit/after installation trick missing?

0 Likes

We also run a headless node, just a server with neither X, nor monitor connected. Card is detected correctly

$ aticonfig --lsa

* 0. 04:00.0 AMD Radeon HD 7900 Series

$ lspci | grep VGA

04:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Tahiti XT [Radeon HD 7970]

Is here some small bit/after installation trick missing?

Can you confirm that you see the same behavior when you connect a monitor to your card. I am doing experiments myself, regarding the headless configuration.

Although I am not sure if permission issues can occur because of that. What i have seen is that some devices get detected, and some not detected.

Also run these two commands after connecting the display:

1. aticonfig --adapter=all --initial

2. export DISPLAY=:0

And reboot.

Will let you know of the experiments soon.

0 Likes

Tried with a monitor connect. Run the commands you suggested then rebooted. No effect.

Further investigation: root can run any opencl code either from console or remotely (through ssh), with or without lightdm running. Regular user can't run anything in any combination.

sudo setcap cap_sys_admin+ep /usr/bin/clinfo

does the trick but not really a proper solution, since sysadmins will end up assigning admin privileges to every single OpenCL binary. Which is neither safe, nor practical.

0 Likes

While installing 13.1, Are you using the regular install (or) the distribution specific package?

0 Likes

Since my last message I reinstalled Ubuntu several more times. Found that with Ubuntu's provided deb packages (as described at https://help.ubuntu.com/community/BinaryDriverHowto/ATI) my HD 7970 card does get recognized at all. I've tried both fglrx-installer and fglrx-installer-updates.

Seems that AMD APP SDK requires one of Catalyst versions from AMD download area. Or may be I missed something.

0 Likes

imd wrote:

Found that with Ubuntu's provided deb packages (as described at https://help.ubuntu.com/community/BinaryDriverHowto/ATI) my HD 7970 card does get recognized at all. I've tried both fglrx-installer and fglrx-installer-updates.

"does not get recognized" - I believe. Correct me please.

Also, When you mean a "clean install" - I hope you are "not" referring to the "release upgrade" path. Instead, you refer to installing Ubuntu from scratch - Please confirm.

I suggest you download the graphics driver from http://support.amd.com/us/gpudownload/Pages/index.aspx - This is what the Ubuntu site also says. You should get a file like "amd-driver-installer-catalyst-13.1-linux-x86.x86_64.run".

Remove previous driver installation. You can use th "fglrx-uninstall.sh" script found in /usr/share/ati directory.

If you had installed before from Ubuntu repository, please do "apt-get remove" as suggested in the Ubuntu page.

Reboot the machine.

Run the 'run' file you got from AMD site. Build a "distribution specific" package. In my case, the driver script recognizes my system as a "debain" system and tries to build the package. During this process, the script may fail because certain utilities may not be found (like execstack, db_modaliases etc..) Just apt-get install these scripts (For db_modaliases, you have to use db-modaliases in apt-get). This script will create 3 DEB files in your current directory. Just install them one by one. Reboot the system and you should be good to go.

If you have time, try this method and see if that solves the problem.

0 Likes

"does not get recognized" - I believe. Correct me please.

Also, When you mean a "clean install" - I hope you are "not" referring to the "release upgrade" path. Instead, you refer to installing Ubuntu from scratch - Please confirm.

Jep, "does not get recognized". By clean install I mean full Ubuntu 12.04 reinstallation from scratch. I use locally prepared kickstart and boot from net.

Just did what you suggested for a second part:

# apt-get install dh-modaliases

# apt-get install execstack

# sh amd-driver-installer-catalyst-13.1-linux-x86.x86_64.run

and there I've chosen to generate .deb files instead of installation. Got in the current directory

fglrx_9.012-0ubuntu1_amd64.deb

fglrx-amdcccle_9.012-0ubuntu1_amd64.deb

fglrx-dev_9.012-0ubuntu1_amd64.deb

Got them installed with "dpkg -i" and additionally had to install dkms package. Rebooted. Set permissions once again:

# chmod 777 /dev/ati

# chmod 666 /dev/ati/card0

Still the same: card has been recognized, root can run any OpenCL code remotely or locally, while regular users still gets ioctl error and has no access to GPU.

I could provide you with "dpkg --get-selections" output.

0 Likes

I don't see a "sudo" in your cmdline.  Are you installing it directly as "root" ? I am not sure if that matters. Let me check.

okay, i have asked you to do enough. Let me try the root thing here and see if it results in an ioctl issue.

Thanks for yur time

0 Likes

Checked the root install as well. Everything is fine. Everyone is able to access and run.... so, could not reproduce the ioctl issue again..

Also, you dont need to run any "chmod" on /dev/ati.

I tried your values and that had no (ill) effect. 

Can you post the "ldd" output of any of the OpenCL programs that you are running?

Mine looks like this.

        linux-vdso.so.1 =>  (0x00007fff71b8e000)

        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd3d89de000)

        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd3d87da000)

        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fd3d85d1000)

        libOpenCL.so.1 => /usr/lib/fglrx/libOpenCL.so.1 (0x00007fd3d83cb000)

        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fd3d80c4000)

        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd3d7e3f000)

        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fd3d7c29000)

        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd3d7886000)

        /lib64/ld-linux-x86-64.so.2 (0x00007fd3d8c1c000)

0 Likes

Well, I encountered the "ioctl" issue through a different path today.

I have 13.1 installed and running on a Ubuntu 11.10 machine.

Everyone is able to run OpenCL programs.

I bring down the GDM (desktop manager) by saying "service gdm stop".

After this, only "root' is able to see  the device.

All other users are not able to see the device.

Strace reveals EACCESS on ioctl.

I wll check with Ubuntu 12.04 tomorrow.

I beleive 12.04 uses "LightDM". So the command to bring down the DM needs some amendment.

Anyway, Is the Desktop Manager up and running in your case?

0 Likes

Yes, I was logged as root for a sake of installation, that is no sudo. Then logged in separately with a regular account for a sake of testing.

Here is what clinfo is linked against:

$ ldd $(which clinfo)

          linux-vdso.so.1 =>  (0x00007fffa1bff000)

          libOpenCL.so.1 => /usr/lib/fglrx/libOpenCL.so.1 (0x00007fafceecf000)

          libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fafcecb2000)

          libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fafce9b5000)

          libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fafce7b1000)

          libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fafce59b000)

          libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fafce1db000)

          /lib64/ld-linux-x86-64.so.2 (0x00007fafcf0fe000)

regaring lightdm: original idea was having a headless server with no GUI at all. That is no lightdm and boot to text mode. But since that I tested with GUI and without, whether lightdm is running or not has no effect. Is there a chance to attach a list of packages I have installed? It seems that is some of the non-default package that makes difference in Catalyst driver installation.

0 Likes

On your system, boot the system to X and login through the X console.

Run "clinfo" from inside the X session. Are you able to see the GPU device without sudo?

0 Likes

Wow ... yes, locally logged in regular user may run clinfo normally and sees GPU. While I'm logged in locally I can open ssh session from remote computer and also has access to GPU with clinfo. As soon as I log out locally, I can't access GPUs any through ssh session. Mystery to me.

It helps a bit, but still does not solve the problem fully, since whole point was running a standalone GPU server with a number of remote users for remote test access.

We have a large farm of GPU servers with CUDA (Tesla cards on board) configured for remote access. Having them as headless nodes has never been a problem. With Radeon HD that was our first step towards OpenCL. Hopefully we will manage to install it finally

0 Likes

Can you try running "xhost +" to disable access control and then try connecting from remote?

This used to be the case before. Not sure, if this will work now. Can you just give a try and let us know?

Thanks,

check http://devgurus.amd.com/message/1287082#1287082. We were able to run a headless system here.

Message was edited by: Himanshu Gautam

0 Likes

Himanshu,

thank you for the time spent on this.

'xhost +' requires a running X11 process (X or Xorg) and a DISPLAY environment variable properly assigned. In a headless server, neither are present.

I'm in the same boat as imd, I'd like to run OpenCL apps on a headless server. If OpenCL apps can only be run in the presence of, and as a child of, an X or Xorg process, it is not a workable solution for me. My server resides in a rack in a server room. No easy access and not physical display.

Thanks,

Valerio

0 Likes

Valerio,

As stated earlier, you can run a headless configuration (without any Monitors/ Cables) attached to your system. But You need to have X running. For ubuntu 12.04 system, we worked on here, you need to have lightdm running, and need to enable proper access control for remote access.

When lightdm was disabled, GPUs will only show if we ran it using sudo previlege. With standard user previleges, only CPU was shown (with ioctl errors). When lightdm was started again, sudo as well as standard user could see the GPUs.

0 Likes

check http://devgurus.amd.com/message/1287082#1287082. We were able to run a headless system here.

This has solved the issue. Thanks!

Summary: run Ubuntu 12.04 with GUI installed and lightdm on. Config (display-setup-script= makes the trick, the rest can be any):

$ cat /etc/lightdm/lightdm.conf
[SeatDefaults]
greeter-session=unity-greeter
user-session=ubuntu
allow-guest=false
greeter-show-manual-login=true
display-setup-script=/etc/lightdm/xhost.sh

$ cat /etc/lightdm/xhost.sh
xhost +
chmod uog+rw /dev/ati/card*

Make sure that xhost.sh is executable:

sudo chmod a+x /etc/lightdm/xhost.sh

Works fine with disconnected monitor and keyborad (i.e. headless). Though the fact that we still need full GUI is confusing, especially after CUDA/Tesla installation experience.

0 Likes

Hi imd,

Thanks for confirming that you were finally able to run the headless server. AMD is working on decoupling the OpenCL and X. So we may be able to work on AMD GPUs as well without installing X. Anyways AFAIK, Tesla is a proper compute accelerator and does not do any graphics rendering. Can you tell if NVIDIA GPU (Fermi/Kepler based) work with OpenCL with out X?

0 Likes