9 Replies Latest reply on Dec 11, 2010 8:19 AM by HarryH

    Memory Leaks?  Valgrind?

    aj_guillon

      Hello everyone,

      I am running a GPU program, and it looks like there is a memory leak (on the host side) somewhere.  Normally I use valgrind for this, but the OpenCL library itself generates huge numbers of errors.  Is it possible that the OpenCL libraries and dependencies have a memory leak inside them?   Does AMD run valgrind on these libraries to ensure their quality?

        • Memory Leaks?  Valgrind?
          nou

          i don't know about OpenCL but fglrx generate huge amount of errors in valgrind. for example running glxgears with valgrind on system with fglxr.

          but someone from AMD stated that fglrx do things which valgrind don't understand and considers it as a error.

          • Memory Leaks?  Valgrind?
            HarryH

            It is possible to use valgrind if you use suppression files to suppress

            the messages caused by libraries etc. See the valgrind documentation.

            This way you will only see stuff caused by your own code.

              • Memory Leaks?  Valgrind?
                nou

                well for me valgrind with openCL cause frozing whole computer and i must reset via SysRq.

                • Memory Leaks?  Valgrind?
                  aj_guillon

                  I did add suppression files... and running with valgrind indicated that my part of the application was clean.  I have sinced turned off OpenCL profiling and the code does not appear to leak.... but it will take a bit of time to find out.  This application will be running for about 4 days, so any leak will be detrimental.  I made a few other changes along with turning off profiling which might cause the leak to disappear.  Have others let their OpenCL applications run for days?  Any experiences to share?

                  Also just because valgrind errors are suppressed doesn't mean that they aren't there... I did notice in another posting that SDK 2.3 is going to have better support for valgrind, which would be amazing.

                    • Memory Leaks?  Valgrind?
                      zeland

                      I use OpenCl  with ATI card for scientific reseach of browning dynamics. usually i run my OpenCl application for about 5 days. Several times i'd received defunct process.  Several times after several hours of calculating. When i used NVIDIA card with this application no one  defunct process appeared.

                  • Memory Leaks?  Valgrind?
                    MicahVillmow
                    aj_guillon,
                    We run valgrind on a per changelist basis on our samples. What you are seeing for a lot of the failures are cases where the runtime allocates memory and then hands it off to the driver to manage. Valgrind considers these lost memory even though the base driver deallocates them.
                      • Memory Leaks?  Valgrind?
                        nou

                        can't AMD release valgrind suppress list for using with fglrx?

                        so we can detect memory leaks in our OpenCL/GL applications.

                          • Memory Leaks?  Valgrind?
                            aj_guillon

                            At this time I'm happy to say my GPU application has been running for about 3 hours, and no sight of a memory leak yet.  I think that whatever the problem was, the adjustments to my own code, or turning off profiling fixed it.

                            • Memory Leaks?  Valgrind?
                              HarryH

                              I have written a couple of scripts to generate suppression files
                              when using libraries you have no control over.  For Debian based
                              distro's these scripts depend on the packages valgrind and php5-cli.
                              For other distro's RTM.

                              There are three scripts:

                              vg     - the main script
                              mksup  - converts valgrind output to suppression file format
                              dupsup - removes duplicates

                              Cut the scripts between the <------> and name them vg, mksup and dupsup.
                              Install the files in a directory in your PATH, and make them
                              executable (chmod 755). If necessary change the first lines of
                              the scripts if 'php' or 'sh' live somewhere else on your system.

                              Use as follows:

                              vg <program to test> <output suppressions file>

                              Example:

                              vg "rbl 128 100 128" OpenCL.sup

                              This command runs the OpenCL program 'rbl' with its parameters,
                              the quotes are required for a program with parameters. The output
                              is written to the file 'OpenCL.sup' and contains ALL suppressions.
                              This file must be inspected manually and any suppressions generated
                              for your own code removed

                              In the case of the above simple program I was left with 28 suppressions.

                              Of course use is not restricted to OpenCL libraries.

                              The reason for splitting the functionality among mksup and dupsup is
                              that you can merge files as follows and remove duplicates:

                              cat file1.sup file2.sup | dupsup > output.sup

                              I supply these scripts in the hope they will be useful but without
                              any guarantees. USE AT YOUR OWN RISK. As far as I am concerned these
                              are now in the public domain and you can do what you want with them.

                              Feedback via this thread please.

                              Harry

                              <----------- vg --------------> #!/bin/sh valgrind --log-fd=1 --leak-check=yes --gen-suppressions=all $1 | mksup | dupsup > $2 <-----------------------------> <----------- mksup --------------> #!/usr/bin/php <?php $count = 0; $fp = fopen("php://stdin", "r") or die("can't read stdin"); function insert_name($matches) { global $count; return "SUP" . $count++; } function remove_extra($matches) { return ""; } $insup = false; while (!feof($fp)) { $line = fgets($fp); if (false !== strpos($line, '{')){ $insup = true; } $line = preg_replace_callback('|<.*>|', "insert_name", $line); $line = preg_replace_callback('|^==.*$|', "remove_extra", $line); if ($insup && (strlen($line) > 1)){ echo $line; } if (false !== strpos($line, '}')){ $insup = false; } } fclose($fp); ?> <-----------------------------> <----------- dupsup --------------> #!/usr/bin/php <?php $fp = fopen("php://stdin", "r") or die("can't read stdin"); $sups = array(); $insup = false; while (!feof($fp)) { $line = trim(fgets($fp)); if(feof($fp)) break; if (false !== strpos($line, '{')){ $insup = true; } $supname = fgets($fp); // We don't need this but must read it $val = trim(fgets($fp)); while ($insup){ $line = trim(fgets($fp)); if (false !== strpos($line, '}')){ $insup = false; } else { $val .= '|' . $line; } } $sups[] = $val; } fclose($fp); $sups = array_unique($sups); foreach ($sups as $k => $v){ echo "{\n"; $v = ' ' . str_replace('|', "\n ", $v); echo ' SUP' . $k . "\n"; echo $v . "\n"; echo "}\n"; } ?> <----------------------------->