3 Replies Latest reply on Jun 30, 2012 1:13 PM by kphillisjr

    Catalyst driver fails on piglit Opengl tests

    kphillisjr

      Since i read elsewhere in the discussion about the AMD OpenGL drivers not being tested for accuracy often, I decided to go ahead and find a test suite that is freely available that could easily shed light on the areas of the driver that need work. I already know that AMD will probably not issue fixes for the cards that already had the announcement that support will be dropped. It'd be nice if the recently removed cards at least got the basic fixes required to at least pass the accuracy tests provided by the piglit opengl test suite, however that'd be a dream.

       

       

      Anyways, I went ahead and tested the OpenGL driver on two different computers and found that the driver completely buckled under the series of tests. This should not have happened, however, the fact that the driver actually failed is more troublesome, because these basic tests that are done should not have caused the driver to bug out and crash numerous times.

       

      Test System 1:

      Radeon HD5770

      Athlon II x4 630

      Gigabyte GA-MA790XT-UD4P

      4GB DDR3 Memory

      Linux Mint 13 ( 64-bit ; Ubuntu 12.04 based, Ubuntu/Precise )

      Catalyst 12.6 beta and Catalyst 12.6 release.

       

      Test System 2:

      Asus K53TA-BBR6 ( Radeon 6720G2 listed, but used apu)

      Sabayon Linux 9 ( 64-bit )

       

      Unofficial bug report for linux:

      http://ati.cchtml.com/show_bug.cgi?id=545

       

      Piglit OpenGL test suite: ( Developers source)

      http://cgit.freedesktop.org/piglit/

       

      Also, I went ahead and attached the output files from the tests since these offer a ton of information on what needs improvement.

      on the quick driver tests using concurrent running on system 1.

       

      CountResult
      1054Passed
      298Failed
      14Crashed
      52skipped
      1418Total Tests Attempted ( Before System Failure)
      8767Total Tests Available ( Dry Run )

       

       

      Anyways, I know of a few other test profiles for OpenGL using this test suite:

       

      Test

      Count*

      Test Profile Name
      9734all
      8764r600
      7all_egl
      3all_es1
      2all_es2
      5929glsl **
      1570glsl-1.1 ***
      2192glsl-1.2 ***
      1974glsl-1.3 ***
      156glsl-1.4 ***
      17OpenGL 3 ****
      70OpenGL 1.1
      47OpenGL 2


      * Test Count is gathered by a dry run using piglet-run.py

      ** This was gathered by running the following command: ./piglit-run.py -c 0 -t spec/glsl- -d tests/all.tests all-results

      *** This was gathered by running the following command: ./piglit-run.py -c 0 -t spec/glsl-1.[Version number] -d tests/all.tests all-results

      **** This is gathered by running the following command: ./piglit-run.py -c 0 -t spec/\!OpenGL\ [Version Number] -d tests/all.tests results

        • Re: Catalyst driver fails on piglit Opengl tests
          kphillisjr

          I did a lot more testing, and found a few interesting tidbits.

           

           

          First off, The crash mentioned before is caused by a race condition when performing threaded shader compiling. This happens in particular when 4 glsl programs are compiled and/or used at the same time. This probably should not happen. To run the tests for now, disable concurrent test running as follows:

          ./piglit-run.py -c 0 tests/quick-driver.tests catalyst-12.6/quick-driver

           

          Anyways, with this test, I got a lot further into the tests, and the general results for the overall driver appears as follows:

          Tests

          Ran

          Status
          6188Passed
          1715Failed
          45crashed
          247Skipped
          8234total attempted
          8767Total tests Available

           

          This test fails to complete because of internal errors causing crashes with the EXT_framebuffer series of extensions. To test just the EXT_framebuffer set of driver tests, run piglet as follows:

          ./piglit-run.py -c 0 -t spec/EXT_framebuffer tests/quick-driver.tests ~/catalyst-12.6/tests/quick-driver-nonconcurrent-ext-framebuffer

          This will crash when the following test is called:

          spec/EXT_framebuffer_multisample/turn-on-off 2

          With this crash, the system will appear to work just fine, but when you move the window the screen will suddenly freeze.

           

          Anyways, I'll get back later with further examination after i resume the numerous tests using the command:

          ./piglit-run.py -c 0 -r -x spec/EXT_framebuffer_multisample/turn-on-off ~catalyst-12.6/quick-driver

           

          Edit: Finished the test run and now i have final results for the quick-driver tests...

           

          CountStatus
          6580Passed
          1864Failed
          53Crashed
          264Skipped
          8764Total


            • Re: Catalyst driver fails on piglit Opengl tests
              gsellers

              Hi,

               

              First, thanks! These are possibly the most helpful types of forum posts we can receive. I'm not sure where the idea that AMD doesn't test its drivers come from. Every change we make runs through several hundred tests, each with tens or hundreds of sub-tests totaling many thousands of tests all together - and that's just for the OpenGL component. Other components have similar suites. A Catalyst package then goes through an almost 3 month long QA cycle where we test every imaginable application before we ship it. With something like a graphics driver being so complex and the number of system configurations we have to support being so wide, it's obviously impossible to test absolutely everything and unfortunately, bugs do slip through.

               

              All that said, Piglit is not one of the test suites we run internally. Short term, we'll assign some engineers to look at the failures and see if we can't get closer to a complete run. More permanently, we're going to add the complete suite to our automated test systems and get it to run on every submitted change, along with the thousands of other tests we run.

               

              Thanks again for the detailed posts and in particular the follow up. It's all to easy to write a post about how much someone sucks... it's much more helpful and constructive to help us dig in and produce the best drivers we can.

               

              Cheers,

               

              Graham

                • Re: Catalyst driver fails on piglit Opengl tests
                  kphillisjr

                  Thanks for the quick response. I'm certain that the Open Source team Graphics team at AMD might be familiar with piglet, and it is possible that provide some insight. Anyways, I know that the large list of issues using piglit can be shown on the older graphics cards as well.

                   

                  I never said that AMD did not test it's drivers. It only appears that way due to how there are bugs that can actually last for over 2 years. This is actually quite annoying. I personally have one bug that went from a lower level performance bug to a major hassle because of changes in the driver over the years.

                   

                  Winehq Bug report from around June 25th, 2010 - Tested against Catalyst 10.5

                  http://bugs.winehq.org/show_bug.cgi?id=23048

                  At this point the bug was a minor performance issue where it is difficult to enter the cd key for the game because of unusually poor performance with in-game menus.

                   

                  Then there was an old bug report on the cchtml I made that was lost. So i had to refile, and over the past 2 years wine went from a performance issue to a full blown unable to play the game. However, I did refile and update along the way...

                   

                  ati unofficial bug report - last tested against Catalyst 12.6.

                  http://ati.cchtml.com/show_bug.cgi?id=54

                   

                  The funny part is that this is tested on the nearly the exact same hardware configuration.

                   

                  Anyways, I for one have seen years of complaints on the phoronix forums ( http://www.phoronix.com/ ) where a vast majority of end users will simply say to go with nvidia graphics cards due to the excessive errors found.

                   

                  Anyways, I Think that it might be worth while to see if it's possible to do a few out of band updates that are specifically targetted at fixing the numerous piglit errors that are seen.

                   

                   

                  Also, I know of other errors that should have been fixed where features fail when trying to test OpenGL ES on Ubuntu 12.04 based systems where all that is done is simply install the driver using the generated debian packages ( Ubuntu/precise to be exact) for the platform ( See: http://ati.cchtml.com/show_bug.cgi?id=126 )

                   

                   

                  Edit:

                   

                  I just completed the "all tests" profile for the catalyst 12.6 driver. However, I am unable to convert it from the raw test data to an html viewable file. The uncompressed output wound up being 893MB ( 10.3MB compressed as a 7-zip compressed tarball)

                   

                  The command for this test was:

                  ./piglit-run.py -c 0 --exclude "spec/EXT_framebuffer_multisample/turn-on-off" tests/all.tests ~/catalyst-12.6/tests/all-nonconcurrent

                  Anyways, Here's the overall breakdown of the results using a grep and word count based on lines:

                  CountResult
                  9734Total Available
                  9729Total Attempted
                  5Total Excluded
                  7244pass
                  2028Fail
                  3warn
                  58crash
                  396Skip

                   

                   

                  Edit 2: I was looking over the test Results, and can see a possible order for solving the failed tests.

                  1. Fixing the "spec/EXT_framebuffer_multisample/turn-on-off" test.  I believe the expected worst case should be that the program crashing instead of the driver crashing.
                  2. Remove "WGL_EXT_swap_control" extension being defined... I seriously doubt this extension should even be allowed in Linux. This causes the test "general/gl30basic" to fail, and could possibly cause issues with wine ( http://www.winehq.org/ ).
                  3. Fix the Security bugs. These 3 bugs are not defined in the OpenGL spec, however adding an extra layer never hurts. However, it is also possible to fail these tests due to memory errors.
                  4. Then the basic OpenGL and GLX test ( namely glean tests and glx tests ). The glean/scissor test is key because I noticed a lot of other tests rely on glScissor, and currently this function is off by 1.
                  5. Finally general tests, fbo tests, and single hiz bug.
                  6. Fix shaders
                  7. Start fixing the numerous Spec tests. Within the GLSL only, There are 1485 failed tests that can be broken down with the following errors:
                    Test setTotal Passed
                    Total Ran
                    spec/glsl44425929
                    Test Breakdown
                    spec/glsl-1.002437
                    spec/glsl-1.1013981570
                    spec/glsl-1.2017212192
                    spec/glsl-1.3012421974
                    spec/glsl-1.40107156

                  Anyways, It's sort of funny, but piglit shows that some of the AMD/ATI extensions that are supported do not behave the way that is defined by the spec...

                  • spec/AMD_conservative_depth/glsl-1.30/compiler/late-redeclaration-01.frag
                  • spec/AMD_seamless_cubemap_per_texture/amd_seamless_cubemap_per_texture
                  • spec/ATI_draw_buffers/arbfp-no-option
                  • spec/ATI_envmap_bumpmap/ati_envmap_bumpmap-bump
                  • spec/ATI_texture_compression_3dc/fbo-generatemipmap-formats