Since i read elsewhere in the discussion about the AMD OpenGL drivers not being tested for accuracy often, I decided to go ahead and find a test suite that is freely available that could easily shed light on the areas of the driver that need work. I already know that AMD will probably not issue fixes for the cards that already had the announcement that support will be dropped. It'd be nice if the recently removed cards at least got the basic fixes required to at least pass the accuracy tests provided by the piglit opengl test suite, however that'd be a dream.
Anyways, I went ahead and tested the OpenGL driver on two different computers and found that the driver completely buckled under the series of tests. This should not have happened, however, the fact that the driver actually failed is more troublesome, because these basic tests that are done should not have caused the driver to bug out and crash numerous times.
Test System 1:
Radeon HD5770
Athlon II x4 630
Gigabyte GA-MA790XT-UD4P
4GB DDR3 Memory
Linux Mint 13 ( 64-bit ; Ubuntu 12.04 based, Ubuntu/Precise )
Catalyst 12.6 beta and Catalyst 12.6 release.
Test System 2:
Asus K53TA-BBR6 ( Radeon 6720G2 listed, but used apu)
Sabayon Linux 9 ( 64-bit )
Unofficial bug report for linux:
http://ati.cchtml.com/show_bug.cgi?id=545
Piglit OpenGL test suite: ( Developers source)
http://cgit.freedesktop.org/piglit/
Also, I went ahead and attached the output files from the tests since these offer a ton of information on what needs improvement.
on the quick driver tests using concurrent running on system 1.
Count | Result |
---|---|
1054 | Passed |
298 | Failed |
14 | Crashed |
52 | skipped |
1418 | Total Tests Attempted ( Before System Failure) |
8767 | Total Tests Available ( Dry Run ) |
Anyways, I know of a few other test profiles for OpenGL using this test suite:
Test Count* | Test Profile Name |
---|---|
9734 | all |
8764 | r600 |
7 | all_egl |
3 | all_es1 |
2 | all_es2 |
5929 | glsl ** |
1570 | glsl-1.1 *** |
2192 | glsl-1.2 *** |
1974 | glsl-1.3 *** |
156 | glsl-1.4 *** |
17 | OpenGL 3 **** |
70 | OpenGL 1.1 |
47 | OpenGL 2 |
* Test Count is gathered by a dry run using piglet-run.py
** This was gathered by running the following command: ./piglit-run.py -c 0 -t spec/glsl- -d tests/all.tests all-results
*** This was gathered by running the following command: ./piglit-run.py -c 0 -t spec/glsl-1.[Version number] -d tests/all.tests all-results
**** This is gathered by running the following command: ./piglit-run.py -c 0 -t spec/\!OpenGL\ [Version Number] -d tests/all.tests results
I did a lot more testing, and found a few interesting tidbits.
First off, The crash mentioned before is caused by a race condition when performing threaded shader compiling. This happens in particular when 4 glsl programs are compiled and/or used at the same time. This probably should not happen. To run the tests for now, disable concurrent test running as follows:
./piglit-run.py -c 0 tests/quick-driver.tests catalyst-12.6/quick-driver
Anyways, with this test, I got a lot further into the tests, and the general results for the overall driver appears as follows:
Tests Ran | Status |
---|---|
6188 | Passed |
1715 | Failed |
45 | crashed |
247 | Skipped |
8234 | total attempted |
8767 | Total tests Available |
This test fails to complete because of internal errors causing crashes with the EXT_framebuffer series of extensions. To test just the EXT_framebuffer set of driver tests, run piglet as follows:
./piglit-run.py -c 0 -t spec/EXT_framebuffer tests/quick-driver.tests ~/catalyst-12.6/tests/quick-driver-nonconcurrent-ext-framebuffer
This will crash when the following test is called:
spec/EXT_framebuffer_multisample/turn-on-off 2
With this crash, the system will appear to work just fine, but when you move the window the screen will suddenly freeze.
Anyways, I'll get back later with further examination after i resume the numerous tests using the command:
./piglit-run.py -c 0 -r -x spec/EXT_framebuffer_multisample/turn-on-off ~catalyst-12.6/quick-driver
Edit: Finished the test run and now i have final results for the quick-driver tests...
Count | Status |
---|---|
6580 | Passed |
1864 | Failed |
53 | Crashed |
264 | Skipped |
8764 | Total |
Hi,
First, thanks! These are possibly the most helpful types of forum posts we can receive. I'm not sure where the idea that AMD doesn't test its drivers come from. Every change we make runs through several hundred tests, each with tens or hundreds of sub-tests totaling many thousands of tests all together - and that's just for the OpenGL component. Other components have similar suites. A Catalyst package then goes through an almost 3 month long QA cycle where we test every imaginable application before we ship it. With something like a graphics driver being so complex and the number of system configurations we have to support being so wide, it's obviously impossible to test absolutely everything and unfortunately, bugs do slip through.
All that said, Piglit is not one of the test suites we run internally. Short term, we'll assign some engineers to look at the failures and see if we can't get closer to a complete run. More permanently, we're going to add the complete suite to our automated test systems and get it to run on every submitted change, along with the thousands of other tests we run.
Thanks again for the detailed posts and in particular the follow up. It's all to easy to write a post about how much someone sucks... it's much more helpful and constructive to help us dig in and produce the best drivers we can.
Cheers,
Graham
Thanks for the quick response. I'm certain that the Open Source team Graphics team at AMD might be familiar with piglet, and it is possible that provide some insight. Anyways, I know that the large list of issues using piglit can be shown on the older graphics cards as well.
I never said that AMD did not test it's drivers. It only appears that way due to how there are bugs that can actually last for over 2 years. This is actually quite annoying. I personally have one bug that went from a lower level performance bug to a major hassle because of changes in the driver over the years.
Winehq Bug report from around June 25th, 2010 - Tested against Catalyst 10.5
http://bugs.winehq.org/show_bug.cgi?id=23048
At this point the bug was a minor performance issue where it is difficult to enter the cd key for the game because of unusually poor performance with in-game menus.
Then there was an old bug report on the cchtml I made that was lost. So i had to refile, and over the past 2 years wine went from a performance issue to a full blown unable to play the game. However, I did refile and update along the way...
ati unofficial bug report - last tested against Catalyst 12.6.
http://ati.cchtml.com/show_bug.cgi?id=54
The funny part is that this is tested on the nearly the exact same hardware configuration.
Anyways, I for one have seen years of complaints on the phoronix forums ( http://www.phoronix.com/ ) where a vast majority of end users will simply say to go with nvidia graphics cards due to the excessive errors found.
Anyways, I Think that it might be worth while to see if it's possible to do a few out of band updates that are specifically targetted at fixing the numerous piglit errors that are seen.
Also, I know of other errors that should have been fixed where features fail when trying to test OpenGL ES on Ubuntu 12.04 based systems where all that is done is simply install the driver using the generated debian packages ( Ubuntu/precise to be exact) for the platform ( See: http://ati.cchtml.com/show_bug.cgi?id=126 )
Edit:
I just completed the "all tests" profile for the catalyst 12.6 driver. However, I am unable to convert it from the raw test data to an html viewable file. The uncompressed output wound up being 893MB ( 10.3MB compressed as a 7-zip compressed tarball)
The command for this test was:
./piglit-run.py -c 0 --exclude "spec/EXT_framebuffer_multisample/turn-on-off" tests/all.tests ~/catalyst-12.6/tests/all-nonconcurrent
Anyways, Here's the overall breakdown of the results using a grep and word count based on lines:
Count | Result |
---|---|
9734 | Total Available |
9729 | Total Attempted |
5 | Total Excluded |
7244 | pass |
2028 | Fail |
3 | warn |
58 | crash |
396 | Skip |
Edit 2: I was looking over the test Results, and can see a possible order for solving the failed tests.
Test set | Total Passed | Total Ran |
---|---|---|
spec/glsl | 4442 | 5929 |
Test Breakdown | ||
spec/glsl-1.00 | 24 | 37 |
spec/glsl-1.10 | 1398 | 1570 |
spec/glsl-1.20 | 1721 | 2192 |
spec/glsl-1.30 | 1242 | 1974 |
spec/glsl-1.40 | 107 | 156 |
Anyways, It's sort of funny, but piglit shows that some of the AMD/ATI extensions that are supported do not behave the way that is defined by the spec...