cancel
Showing results for 
Search instead for 
Did you mean: 

Archives Discussions

pjb7687
Adept I

Driver crashes with OpenCL?

Hello,

I am Jeongbin Park, the main developer of Cas-OFFinder (snugel/cas-offinder · GitHub​, article: http://www.ncbi.nlm.nih.gov/pubmed/24463181).

Recently we are experiencing crashes with the latest AMD driver (on Ubuntu linux) with huge input data.

The symptom of 'crash' is that Cas-OFFinder runs fine with several hours, and then suddenly it hangs.

After that, running any OpenCL program makes the terminal hang and doesn't respond.

Also, when I tried to kill Cas-OFFinder or the newly created OpenCL processes,

they won't simply be killed just like zombie processes - even with SIGKILL - and the only way to terminate them is system reboot.

For your information, I designed the host-side program to make a lot of chunks of the input data,

so that the running time per each chunk of the OpenCL kernels to be in relatively short time (in several seconds).

Full source code of latest Cas-OFFinder: snugel/cas-offinder at experimental · GitHub

Source code of OpenCL kernels can be found here: cas-offinder/cas-offinder.cl at experimental · snugel/cas-offinder · GitHub

Could you see the source code and let me know why such a weird behavior happen?

For your information, it occurs with AMD APP SDK 2.9.1 and also with 3.0 beta. Also, it does not crash with CPU.

Thanks,

Jeongbin

0 Likes
31 Replies
dipak
Staff
Staff

Re: Driver crashes with OpenCL?

Thanks for reporting the problem.

Recently we are experiencing crashes with the latest AMD driver (on Ubuntu linux) with huge input data.

May I assume that the issue is not occurring on earlier drivers, only using latest one? [Or please specify the last catalyst version where it worked fine]

The symptom of 'crash' is that Cas-OFFinder runs fine with several hours, and then suddenly it hangs.

Did you observe any particular pattern and dependency on external factors such as setup, input data, system load etc.? Please provide your setup details such as OS, GPU, CPU etc.. We'll try to reproduce it.

BTW, is it possible to trigger the issue in less time? it would be helpful for our testing.

Regards,

0 Likes
pjb7687
Adept I

Re: Driver crashes with OpenCL?

Firstly I have tested it on the default version of driver shipped with Ubuntu 14.04.

I also tried it with the latest driver, but nothing different.

We are also trying to use a commercial cluster computer (Chundoong, 슈퍼컴퓨터 천둥) utilizing AMD graphic cards (7970 HD) and OpenCL, but the same problem also occurs.

The cluster uses RHEL 6.3 as its host OS.

I tested it with 7870 HD, 7970 HD, and R9 290X, but all has the same issue.

Currently our server has two of R9 290X cards with the latest AMD driver.

Unfortunately, the problem usually occurs in long time analysis, but I am not sure...

I can provide input data but how can I upload them?

Hardware summary of our server:

  OS: Ubuntu 14.04 LTS

  CPU: Intel i7 4770k

  GPU: 2x AMD R9 290X

Thank you,

Jeongbin

0 Likes
dipak
Staff
Staff

Re: Driver crashes with OpenCL?

Hi Jeongbin,

Thanks for the quick reply. By referring the catalyst version, I just wanted to confirm whether the issue is related to this version only or not.

Regarding the input data, you may attach the file here (if size within limit) or may upload to a public site and share us the link [if data file is password protected, you can send me the password via a private message]

Regards,

0 Likes
pjb7687
Adept I

Re: Driver crashes with OpenCL?

First, you can download the input file from this link:

http://www.rgenome.net/static/targets.zip

And you also need the reference genome of Human from here:

http://www.rgenome.net/static/human_hg38.zip

First you need to unzip the above two compressed files,

and then the directory structure should be like below:

./human_hg38/chr1.fa

./human_hg38/chr2.fa

....

./targets_1.txt

./targets_2.txt

...

Finally you can run Cas-OFFinder like below:

cas-offinder targets_1.txt G output.txt

(Of course you should have installed Cas-OFFinder on your system, maybe you can easily build one with CMake. Please test the 'experimental' branch of Cas-OFFinder [snugel/cas-offinder at experimental · GitHub]. Or I can also provide a compiled binary if you want)

Please note that the first line of target_??.txt is the path of directory containing genome sequences.

You can try all of the 21 files in sequence, and then maybe you can find the problem that I reported.

Thank you,

Jeongbin

0 Likes
dipak
Staff
Staff

Re: Driver crashes with OpenCL?

Thanks for providing the data files. We'll check and get back you shortly.

0 Likes
dipak
Staff
Staff

Re: Driver crashes with OpenCL?

Hi Jeongbin,

AMD has just released Catalyst 15.7 [display driver version 15.20.x]. It has many improvements and fixes compared to earlier ones. Could you please check the issue once using this driver?

As you already have the working setup, it'll be quicker for you to verify than me. If issue still exists, we'll try here.

Regards,

0 Likes
dipak
Staff
Staff

Re: Driver crashes with OpenCL?

Hi Jeongbin,

Did you manage to check it with the latest 15.7 driver? If yes, what was your observation?

Regards,

0 Likes
pjb7687
Adept I

Re: Driver crashes with OpenCL?

i tested the driver for a week, I haven't found any crash with it.

In addition, I also found that the performance of evaluation is faster than before, about 20~30%. Very good!

Thank you for your answer.

Best,

Jeongbin

0 Likes
dipak
Staff
Staff

Re: Driver crashes with OpenCL?

Thanks for your confirmation. Its really nice to hear that the latest driver is working fine.

Regards,

0 Likes