cancel
Showing results for 
Search instead for 
Did you mean: 

Processors

Highlighted
Adept I
Adept I

PCIE lanes stepping on eachother?

I was testing my NVME hard drives separately and simultaneously to see why my NVME raid setup was not giving me expected results. 

In this test case, I tried to start them all at as close to the same time as possible:

pastedImage_0.png

These are 4 Samsung 960 EVO on an Asus Hyper M.2 16x card.  Each drive does about the same when testing them individually NOT at the same time:

pastedImage_1.png

I bought the Threadripper because of the 64 dedicated PCIE lanes to the processor.  It seems like there is a bottleneck making these lanes feel less than dedicated.  The 4KiB Q32T1 is especially suspect, as it seems like the numbers kinda add up, despite this being 4 different processes doing single threaded tests on a 16 core processor.

I also did a 3 way simultaneous test on other M.2 slots on my board besides the Asus Hyper M.2 16x:

pastedImage_2.png

Again, it seems like there is only so much IO capability to go around, and it is being shared among the simultaneous tests.

Here are some  of the RAID tests that got me investigating this:

AMD Raid0 7 disks no cache:

pastedImage_3.png

AMD Raid0 3 disks no cache:

pastedImage_5.png

AMD Raid0 4 disks no cache:

pastedImage_6.png

On the RAID tangent, I don't seem to be getting anywhere close to the kind of results shown in material like this which prompted me to dig deeper (though I am using CrystalDiskMark instead of IOMeter):

Super fast NVMe RAID comes to Threadripper

The RAID numbers aside, my expectation was that I should see these individual drives performing just as well simultaneously as separately, considering I have 16 cores and 64 CPU PCIE lanes.  Why is my expectation invalid?  Is this a bottleneck in the Infinity Fabric?  It seems weird that it can push such high sequential numbers, yet the 4K tests are fighting each other for resources.

I think I may have to give up on AMD raid for now anyways because of other issues (Installed 17.50 NVME Raid drivers for Threadripper, now computer cannot power down normally ), but considering how much time I sunk into it (It took me several hours to get RaidXpert online), I wanted to get my results out there.

Tags (2)
0 Kudos
11 Replies
Highlighted
Esteemed Contributor III

Re: PCIE lanes stepping on eachother?

Can't compare IOMeter to Crystal DiskMark. Also, you're using Evo drives, not Pro drives, and random operations are where Evo drives suffer.

Which movies are you editing which require greater than 9.8GB/s?

Highlighted
Adept I
Adept I

Re: PCIE lanes stepping on eachother?

Evo vs Pro shouldn't matter for what I am asking, as I am comparing Evo to Evo, not Evo to Pro.  This is more algebraic:

Case 1:   1evo = 548MB/s

Case 2:   4evo = 183MB/s + 183MB/s + 205MB/s + 302.1MB/s + 302.1MB/s

                4evo = 1175MB/s

                1evo = 293MB/s

It is a pretty huge difference, and probably if I could press the buttons closer to the same time, they would all come back around 183.  Also this is without using RAID, just 4 SSDs that happen to be in use at the same time by 4 single core processes, and somehow they are stepping all over eachother despite being on separate CPU PCIE lanes on a 16 core chip.

0 Kudos
Highlighted
Esteemed Contributor III

Re: PCIE lanes stepping on eachother?

This is about how little I trust Crystal DiskMark. Top is my 960 Pro, bottom are my two 850 Evos. And no your PCIe lanes aren't "stepping on each other", it's just how that program works. Download IOMeter and run it as AMD did, but do be warned that unless you're running a server, database, or other very high end system, you will never reach QD32 in real workloads.

0 Kudos
Highlighted
Adept I
Adept I

Re: PCIE lanes stepping on eachother?

What exactly was it about your results that bothered you about CrystalDiskMark?  Did you run these simultaneously?

I downloaded IOMeter before to see if I could match results, but it seems pretty confusing to set up so I got lazy and went with CrystalDiskMark.  I guess I could work a little harder at it and try again with CrystalDiskMark.

0 Kudos
Highlighted
Adept I
Adept I

Re: PCIE lanes stepping on eachother?

** try again with IOMeter

0 Kudos
Highlighted
Adept I
Adept I

Re: PCIE lanes stepping on eachother?

OK, so IOMeter does not appear to support testing multiple disks at the same time with one instance, and it also doesn't seem to support multiple instances of the program running simultaneously, so if there was a problem with the hard drives stepping on each other, IOMeter would not provide much help for diagnosing it.

0 Kudos
Highlighted
Adept I
Adept I

Re: PCIE lanes stepping on eachother?

OK, so here is how much I trust IOMeter on my 8x array

pastedImage_6.png

pastedImage_4.png

pastedImage_2.png

Looks like the CrystalDiskMark is a bit closer to what I would expect an 8x raid to do, but still so far.  Perhaps I didn't configure IOMeter 100% perfectly, but that seems to be a pretty good reason to trust a program that gets results with a few clicks.

0 Kudos
Highlighted
Adept I
Adept I

Re: PCIE lanes stepping on eachother?

Here is the best illustration of the problem I have come up with so far, and what I would now imagine as my conclusion:

AMD Raid 3x Benchmarked Alone:

pastedImage_0.png

AMD Raid 4x Benchmarked Alone:

pastedImage_1.png

AMD Raid 4x and 3x benchmarked at the same time:

pastedImage_2.png

Basically my system has magic numbers of around:

Seq Q32T1 Read: Around 11000MB/s

Seq Q32T1 Write: Around 11000MB/s (I remember seeing this number in an 8x raid benchmark I did)

4KiB Q8T8 Read: Around 700MB/s

4KiB Q8T8 Read: Around 700MB/s

4KiB Q32T1 Write: Around 600MB/s

4KiB Q32T1 Write: Around 600MB/s

Whatever crazy strategy you can think of on this platform, everything will ultimately add up to these numbers.  So, if your drives' single performance numbers add up to more than any of these magic numbers, you will probably hit the same bottleneck and end up around the same numbers.  AMD Threadripper NVME raid seems to scale a little bit better than Windows Raid, but ultimately there exists some bottleneck that effects both (which I presume is in the Threadripper processor).  So if you mainly care about sequential read speed, pretty much 4 samsung drives is all you need to max out this platform.  If you mainly care about write speed, you can stack them to the non-advertised 8 drive limit if you are using 960 evo.  I would guess that 960 pro would cap out around 6.  If you are looking for the 4k random, pretty much don't bother, as it seems to only be capable of helping marginally, which was actually unexpected for me.  I would expect queue depth 1 to only be hurt by RAID, but it appears it can give the drives a bit of a rest between requests.  The advertising materials pump this Threadripper NVME RAID capabilities up quite a bit too far, resulting in a lot of wasted time for me, but hopefully some people can read this and save their own time.

I am including a UserBenchmark so you can better estimate how your results might compare to mine:

http://www.userbenchmark.com/UserRun/5930809

Despite better benchmarks, my system visibly performs worse when I try to OC the CPU so I don't bother, but the memory is OCed at 3466 quad channel.

0 Kudos
Highlighted
Esteemed Contributor III

Re: PCIE lanes stepping on eachother?

amdmatt​ you have any input? Don't think we have an AMD HEDT oriented rep here yet.

0 Kudos