cancel
Showing results for 
Search instead for 
Did you mean: 

Archives Discussions

kphillisjr
Adept II

Catalyst driver fails on piglit Opengl tests

Since i read elsewhere in the discussion about the AMD OpenGL drivers not being tested for accuracy often, I decided to go ahead and find a test suite that is freely available that could easily shed light on the areas of the driver that need work. I already know that AMD will probably not issue fixes for the cards that already had the announcement that support will be dropped. It'd be nice if the recently removed cards at least got the basic fixes required to at least pass the accuracy tests provided by the piglit opengl test suite, however that'd be a dream.

Anyways, I went ahead and tested the OpenGL driver on two different computers and found that the driver completely buckled under the series of tests. This should not have happened, however, the fact that the driver actually failed is more troublesome, because these basic tests that are done should not have caused the driver to bug out and crash numerous times.

Test System 1:

Radeon HD5770

Athlon II x4 630

Gigabyte GA-MA790XT-UD4P

4GB DDR3 Memory

Linux Mint 13 ( 64-bit ; Ubuntu 12.04 based, Ubuntu/Precise )

Catalyst 12.6 beta and Catalyst 12.6 release.

Test System 2:

Asus K53TA-BBR6 ( Radeon 6720G2 listed, but used apu)

Sabayon Linux 9 ( 64-bit )

Unofficial bug report for linux:

http://ati.cchtml.com/show_bug.cgi?id=545

Piglit OpenGL test suite: ( Developers source)

http://cgit.freedesktop.org/piglit/

Also, I went ahead and attached the output files from the tests since these offer a ton of information on what needs improvement.

on the quick driver tests using concurrent running on system 1.

CountResult
1054Passed
298Failed
14Crashed
52skipped
1418Total Tests Attempted ( Before System Failure)
8767Total Tests Available ( Dry Run )

Anyways, I know of a few other test profiles for OpenGL using this test suite:

Test

Count*

Test Profile Name
9734all
8764r600
7all_egl
3all_es1
2all_es2
5929glsl **
1570glsl-1.1 ***
2192glsl-1.2 ***
1974glsl-1.3 ***
156glsl-1.4 ***
17OpenGL 3 ****
70OpenGL 1.1
47OpenGL 2


* Test Count is gathered by a dry run using piglet-run.py

** This was gathered by running the following command: ./piglit-run.py -c 0 -t spec/glsl- -d tests/all.tests all-results

*** This was gathered by running the following command: ./piglit-run.py -c 0 -t spec/glsl-1.[Version number] -d tests/all.tests all-results

**** This is gathered by running the following command: ./piglit-run.py -c 0 -t spec/\!OpenGL\ [Version Number] -d tests/all.tests results

0 Likes
3 Replies
kphillisjr
Adept II

Re: Catalyst driver fails on piglit Opengl tests

I did a lot more testing, and found a few interesting tidbits.

First off, The crash mentioned before is caused by a race condition when performing threaded shader compiling. This happens in particular when 4 glsl programs are compiled and/or used at the same time. This probably should not happen. To run the tests for now, disable concurrent test running as follows:

./piglit-run.py -c 0 tests/quick-driver.tests catalyst-12.6/quick-driver

Anyways, with this test, I got a lot further into the tests, and the general results for the overall driver appears as follows:

Tests

Ran

Status
6188Passed
1715Failed
45crashed
247Skipped
8234total attempted
8767Total tests Available

This test fails to complete because of internal errors causing crashes with the EXT_framebuffer series of extensions. To test just the EXT_framebuffer set of driver tests, run piglet as follows:

./piglit-run.py -c 0 -t spec/EXT_framebuffer tests/quick-driver.tests ~/catalyst-12.6/tests/quick-driver-nonconcurrent-ext-framebuffer

This will crash when the following test is called:

spec/EXT_framebuffer_multisample/turn-on-off 2

With this crash, the system will appear to work just fine, but when you move the window the screen will suddenly freeze.

Anyways, I'll get back later with further examination after i resume the numerous tests using the command:

./piglit-run.py -c 0 -r -x spec/EXT_framebuffer_multisample/turn-on-off ~catalyst-12.6/quick-driver

Edit: Finished the test run and now i have final results for the quick-driver tests...

CountStatus
6580Passed
1864Failed
53Crashed
264Skipped
8764Total


0 Likes
gsellers
Staff
Staff

Re: Catalyst driver fails on piglit Opengl tests

Hi,

First, thanks! These are possibly the most helpful types of forum posts we can receive. I'm not sure where the idea that AMD doesn't test its drivers come from. Every change we make runs through several hundred tests, each with tens or hundreds of sub-tests totaling many thousands of tests all together - and that's just for the OpenGL component. Other components have similar suites. A Catalyst package then goes through an almost 3 month long QA cycle where we test every imaginable application before we ship it. With something like a graphics driver being so complex and the number of system configurations we have to support being so wide, it's obviously impossible to test absolutely everything and unfortunately, bugs do slip through.

All that said, Piglit is not one of the test suites we run internally. Short term, we'll assign some engineers to look at the failures and see if we can't get closer to a complete run. More permanently, we're going to add the complete suite to our automated test systems and get it to run on every submitted change, along with the thousands of other tests we run.

Thanks again for the detailed posts and in particular the follow up. It's all to easy to write a post about how much someone sucks... it's much more helpful and constructive to help us dig in and produce the best drivers we can.

Cheers,

Graham

0 Likes
kphillisjr
Adept II

Re: Catalyst driver fails on piglit Opengl tests

Thanks for the quick response. I'm certain that the Open Source team Graphics team at AMD might be familiar with piglet, and it is possible that provide some insight. Anyways, I know that the large list of issues using piglit can be shown on the older graphics cards as well.

I never said that AMD did not test it's drivers. It only appears that way due to how there are bugs that can actually last for over 2 years. This is actually quite annoying. I personally have one bug that went from a lower level performance bug to a major hassle because of changes in the driver over the years.

Winehq Bug report from around June 25th, 2010 - Tested against Catalyst 10.5

http://bugs.winehq.org/show_bug.cgi?id=23048

At this point the bug was a minor performance issue where it is difficult to enter the cd key for the game because of unusually poor performance with in-game menus.

Then there was an old bug report on the cchtml I made that was lost. So i had to refile, and over the past 2 years wine went from a performance issue to a full blown unable to play the game. However, I did refile and update along the way...

ati unofficial bug report - last tested against Catalyst 12.6.

http://ati.cchtml.com/show_bug.cgi?id=54

The funny part is that this is tested on the nearly the exact same hardware configuration.

Anyways, I for one have seen years of complaints on the phoronix forums ( http://www.phoronix.com/ ) where a vast majority of end users will simply say to go with nvidia graphics cards due to the excessive errors found.

Anyways, I Think that it might be worth while to see if it's possible to do a few out of band updates that are specifically targetted at fixing the numerous piglit errors that are seen.

Also, I know of other errors that should have been fixed where features fail when trying to test OpenGL ES on Ubuntu 12.04 based systems where all that is done is simply install the driver using the generated debian packages ( Ubuntu/precise to be exact) for the platform ( See: http://ati.cchtml.com/show_bug.cgi?id=126 )

Edit:

I just completed the "all tests" profile for the catalyst 12.6 driver. However, I am unable to convert it from the raw test data to an html viewable file. The uncompressed output wound up being 893MB ( 10.3MB compressed as a 7-zip compressed tarball)

The command for this test was:

./piglit-run.py -c 0 --exclude "spec/EXT_framebuffer_multisample/turn-on-off" tests/all.tests ~/catalyst-12.6/tests/all-nonconcurrent

Anyways, Here's the overall breakdown of the results using a grep and word count based on lines:

CountResult
9734Total Available
9729Total Attempted
5Total Excluded
7244pass
2028Fail
3warn
58crash
396Skip

Edit 2: I was looking over the test Results, and can see a possible order for solving the failed tests.

  1. Fixing the "spec/EXT_framebuffer_multisample/turn-on-off" test.  I believe the expected worst case should be that the program crashing instead of the driver crashing.
  2. Remove "WGL_EXT_swap_control" extension being defined... I seriously doubt this extension should even be allowed in Linux. This causes the test "general/gl30basic" to fail, and could possibly cause issues with wine ( http://www.winehq.org/ ).
  3. Fix the Security bugs. These 3 bugs are not defined in the OpenGL spec, however adding an extra layer never hurts. However, it is also possible to fail these tests due to memory errors.
  4. Then the basic OpenGL and GLX test ( namely glean tests and glx tests ). The glean/scissor test is key because I noticed a lot of other tests rely on glScissor, and currently this function is off by 1.
  5. Finally general tests, fbo tests, and single hiz bug.
  6. Fix shaders
  7. Start fixing the numerous Spec tests. Within the GLSL only, There are 1485 failed tests that can be broken down with the following errors:
    Test setTotal Passed
    Total Ran
    spec/glsl44425929
    Test Breakdown
    spec/glsl-1.002437
    spec/glsl-1.1013981570
    spec/glsl-1.2017212192
    spec/glsl-1.3012421974
    spec/glsl-1.40107156

Anyways, It's sort of funny, but piglit shows that some of the AMD/ATI extensions that are supported do not behave the way that is defined by the spec...

  • spec/AMD_conservative_depth/glsl-1.30/compiler/late-redeclaration-01.frag
  • spec/AMD_seamless_cubemap_per_texture/amd_seamless_cubemap_per_texture
  • spec/ATI_draw_buffers/arbfp-no-option
  • spec/ATI_envmap_bumpmap/ati_envmap_bumpmap-bump
  • spec/ATI_texture_compression_3dc/fbo-generatemipmap-formats
0 Likes