11 Replies Latest reply on Oct 30, 2010 11:52 AM by bpurnomo

    Profiling Changes Results

    aj_guillon

      I worked with a friend to try out my OpenCL project on his Windows machine with Visual Studio, so that I can try the Stream Analyzer.  The project compiles fine... runs...  but when I try to do any profiling on it, my program crashes due to an error inside one of the kernels.  This is my own error code that is generated, but it looks as if profiling the code has caused the results to change.  It works fine under normal execution, but with profiling it dies.  What could be causing this?  There are no race-conditions in my code or anything like that.  However, if the kernel is run more than once my program would fail... is the profiler perhaps causing multiple executions of the same kernel without "rolling back" changes to global memory?

      I'm using the latest SDK, and latest catalyst drivers on Win XP SP3.

        • Profiling Changes Results
          ryta1203

          Could you post your code?

          Plus, this should probably be moved to the OpenCL forum.

            • Profiling Changes Results
              aj_guillon

              Thanks for the response, I thought this was the appropriate forum for discussions of tools.

              Unfortunately I can't post the code, because it is a fairly large commercial product.  I know this doesn't help to debug things, but if there are things I can try I will do so... also, if I can understand how the profiler will interact with my own code perhaps I can find what the issue is.  Is each kernel guaranteed to be executed only once?

                • Profiling Changes Results
                  bpurnomo

                  Thank you for using our tool, ATI Stream Profiler. This is the correct forum for discussion about GPU tools.  The tool does perform multiple executions per kernel whenever necessary to collect all the specified performance counters.  Prior to each kernel execution, buffers that are created with both read and write flag should be rolled back (the tool saves the original buffers).   Have you specified these buffers with the correct flags?  If possible, please send us (gputools.support@amd.com) a test case in case there is a problem with the tool that we can fix.

                   

                    • Profiling Changes Results
                      aj_guillon

                      I have emailed my kernel to the supplied address.  For those who come across this thread without access to the code, it basically just tries some atomic operations in a loop until they succeed.

                        • Profiling Changes Results
                          aj_guillon

                          Any ideas?  Anything I can do to help?

                            • Profiling Changes Results
                              bpurnomo

                              Please send us the complete test case.

                                • Profiling Changes Results
                                  aj_guillon

                                  What specifically do you want in a "complete test case"?  I cannot send the entire application, and building a demo application that just exhibits this problem also could be very complicated, due to dependencies... I am willing to do this, but is there anything else we can attempt or try before I do that?  Perhaps some way for me to send you debugging information, logs, or a trace ?

                                  Also, if I could tell the profiler "I don't care about these particular kernel launches"... I would probably be able to proceed... since the kernels that break the profiler are only constructing data structures for the model to run... i.e. they are run once, quickly, then never again.

                                  As I mentioned, there are many atomic operations in the code... has the profiler been tested with atomic support?

                                  Thanks.

                                    • Profiling Changes Results
                                      bpurnomo

                                      Yes, the profiler should work with atomic ops.

                                      Can you please post all source lines corresponding to the clCreateBuffer/Image calls for all the buffer objects used by the kernel?

                                        • Profiling Changes Results
                                          aj_guillon

                                          I have a factory that builds the objects... here is what it does:

                                          clCreateBuffer(q._context, flags, size, 0, &error_code);

                                          The flags passed in are: CL_MEM_READ_WRITE if I want to create a read/write array, or CL_MEM_READ_ONLY for a read-only array.  I changed the code to use only CL_MEM_READ_WRITE though to see if this had an effect, and it did not.

                                            • Profiling Changes Results
                                              aj_guillon

                                              Okay, I think I figured it out.  I spent tonight doing a comment-binary-search to try to find what line in my kernel is causing it to crash... the problem is atomics after all.  You can reproduce the problem very easily.  Write a kernel, and pass __global unsigned int* count as an argument.  Initialize the parameter to zero before you pass it into a kernel.

                                              Now inside the kernel just do atom_inc(count) or atom_add(count, 1).  You will find that the total sum is greater than the number of global work items by a factor of 4, which suggests to me that the tool runs each kernel 4 times.  None of the kernels I care to profile use atomics... but I need to get past this kernel to continue my work.  I'm going to try to emulate an atom_add using atom_cmpxchg and see if this fixes it.

                                                • Profiling Changes Results
                                                  bpurnomo

                                                  I don't think the problem is with atomics (I tried with a simple program with atomics and the Histogram SDK sample, both work with the profiler): as long as the kernel arguments and buffers are setup properly (with both READ and WRITE flag), the profiler will restore the read write buffers prior to introducing additional kernel dispatches indiscriminantly.  The problem may be due to the way you setup the program that perhaps the profiler hasn't accounted yet (or maybe there is a bug in the application). To rule out the atomics issue, you can try modifying your kernel to read a location in the buffer, add 1, and write back the result to the buffer.  I'd guess that you would have the same issue with this scenario in your application.

                                                  This is hard to debug without the source application.  If you can isolate the problem in a simple test case (perhaps by modifying one of the SDK samples), please send it to us so we can take a look.