Hello everyone,
I am running a GPU program, and it looks like there is a memory leak (on the host side) somewhere. Normally I use valgrind for this, but the OpenCL library itself generates huge numbers of errors. Is it possible that the OpenCL libraries and dependencies have a memory leak inside them? Does AMD run valgrind on these libraries to ensure their quality?
i don't know about OpenCL but fglrx generate huge amount of errors in valgrind. for example running glxgears with valgrind on system with fglxr.
but someone from AMD stated that fglrx do things which valgrind don't understand and considers it as a error.
It is possible to use valgrind if you use suppression files to suppress
the messages caused by libraries etc. See the valgrind documentation.
This way you will only see stuff caused by your own code.
well for me valgrind with openCL cause frozing whole computer and i must reset via SysRq.
I did add suppression files... and running with valgrind indicated that my part of the application was clean. I have sinced turned off OpenCL profiling and the code does not appear to leak.... but it will take a bit of time to find out. This application will be running for about 4 days, so any leak will be detrimental. I made a few other changes along with turning off profiling which might cause the leak to disappear. Have others let their OpenCL applications run for days? Any experiences to share?
Also just because valgrind errors are suppressed doesn't mean that they aren't there... I did notice in another posting that SDK 2.3 is going to have better support for valgrind, which would be amazing.
I use OpenCl with ATI card for scientific reseach of browning dynamics. usually i run my OpenCl application for about 5 days. Several times i'd received defunct process. Several times after several hours of calculating. When i used NVIDIA card with this application no one defunct process appeared.
can't AMD release valgrind suppress list for using with fglrx?
so we can detect memory leaks in our OpenCL/GL applications.
At this time I'm happy to say my GPU application has been running for about 3 hours, and no sight of a memory leak yet. I think that whatever the problem was, the adjustments to my own code, or turning off profiling fixed it.
I have written a couple of scripts to generate suppression files
when using libraries you have no control over. For Debian based
distro's these scripts depend on the packages valgrind and php5-cli.
For other distro's RTM.
There are three scripts:
vg - the main script
mksup - converts valgrind output to suppression file format
dupsup - removes duplicates
Cut the scripts between the <------> and name them vg, mksup and dupsup.
Install the files in a directory in your PATH, and make them
executable (chmod 755). If necessary change the first lines of
the scripts if 'php' or 'sh' live somewhere else on your system.
Use as follows:
vg <program to test> <output suppressions file>
Example:
vg "rbl 128 100 128" OpenCL.sup
This command runs the OpenCL program 'rbl' with its parameters,
the quotes are required for a program with parameters. The output
is written to the file 'OpenCL.sup' and contains ALL suppressions.
This file must be inspected manually and any suppressions generated
for your own code removed
In the case of the above simple program I was left with 28 suppressions.
Of course use is not restricted to OpenCL libraries.
The reason for splitting the functionality among mksup and dupsup is
that you can merge files as follows and remove duplicates:
cat file1.sup file2.sup | dupsup > output.sup
I supply these scripts in the hope they will be useful but without
any guarantees. USE AT YOUR OWN RISK. As far as I am concerned these
are now in the public domain and you can do what you want with them.
Feedback via this thread please.
Harry
<----------- vg --------------> #!/bin/sh valgrind --log-fd=1 --leak-check=yes --gen-suppressions=all $1 | mksup | dupsup > $2 <-----------------------------> <----------- mksup --------------> #!/usr/bin/php <?php $count = 0; $fp = fopen("php://stdin", "r") or die("can't read stdin"); function insert_name($matches) { global $count; return "SUP" . $count++; } function remove_extra($matches) { return ""; } $insup = false; while (!feof($fp)) { $line = fgets($fp); if (false !== strpos($line, '{')){ $insup = true; } $line = preg_replace_callback('|<.*>|', "insert_name", $line); $line = preg_replace_callback('|^==.*$|', "remove_extra", $line); if ($insup && (strlen($line) > 1)){ echo $line; } if (false !== strpos($line, '}')){ $insup = false; } } fclose($fp); ?> <-----------------------------> <----------- dupsup --------------> #!/usr/bin/php <?php $fp = fopen("php://stdin", "r") or die("can't read stdin"); $sups = array(); $insup = false; while (!feof($fp)) { $line = trim(fgets($fp)); if(feof($fp)) break; if (false !== strpos($line, '{')){ $insup = true; } $supname = fgets($fp); // We don't need this but must read it $val = trim(fgets($fp)); while ($insup){ $line = trim(fgets($fp)); if (false !== strpos($line, '}')){ $insup = false; } else { $val .= '|' . $line; } } $sups[] = $val; } fclose($fp); $sups = array_unique($sups); foreach ($sups as $k => $v){ echo "{\n"; $v = ' ' . str_replace('|', "\n ", $v); echo ' SUP' . $k . "\n"; echo $v . "\n"; echo "}\n"; } ?> <----------------------------->