PC Processors

[falloutboy]

I have an ASUS Rampage V edition 10 and an ASUS X399a.

On both machines I installed ASUS XGC-100C 10Gbe nics, on both machines they are seen ; however neither machine can ping one another after being setup with a class C ip address and subnet, I.E 192.168.1.x with a netmask of 255.255.255.0. - Both will work with the NICs connected to a switch at 1GB - end result ASUS said return the cards so that is what I did - which brings us to today.

I brought two Mellenox ConnectX2 SFP nics and a Cisco 5m Cable and repeated the above process, the R5E10 no problems, the X399a saw the NIC once when i first loaded up and showed an error 43, so deleted the driver and rebooted - this was the last time this card was seen.

Tried the card in all of the slots on my motherboard with and without my raid controller - not seen.

Tried the card in my R5E10 and put the card from my R5E10 on the X399 i.e swapped the cards between the two PC's

R5E10 - card found, X399a - not seen.

I am going batty here trying to sort this out and am quickly coming to the conclusion that there is either a fault in the X399 motherboard bios ( after reading about other PCIe issues with X399 online ) OR that there is a problem in the X399 chipset related to PCIe, either way I want this sorted as it has cost me a lot of time, effort, energy and money to set this up - and I'm getting to the point that I feel like returning this stuff and going with the devil I know - namely Intel - which I don't want to as they lost a lot of trust from me after the whole Spectre etc debacle. But it comes down to this - I need something that just works the way it's meant to.

[misterj]

falloutboy, I recommend your try the latest drivers from Aquantia.  I am assuming these NICs use Aquantia chips.  I have an ASRock board with an Aquantia NIC on the MB.  I cannot use the Aquantia drivers but do not know about ASUS.  I also recommend you get the latest Chipset drivers from AMD Support.  Please use Manually Select.  Enjoy, John.

[falloutboy]

I'm checking out the latest chipset drivers from the AMD website now and

will compare driver versions with what is currently installed ; however

when it comes to the Aquantia drivers the latest version from their website

made things worse, the cards refused to identify that they even had a cable

attached.

[misterj]

Sorry that was not useful, falloutboy.  It is time to open a Support Ticket with ASUS.  I think Aquantia also has a support path now.  If so, open a ticket there Also.  Please post your results here.  Thanks and good luck, John.

[falloutboy]

Additional,

I confirmed both Mellenox cards are working by plugging them individually into my R5E10 while in there i did a "mst status" from an administrative command prompt and got the expected response that provides the cards device number.

I can plug either card into the X399 and note the i don't get a PXE boot message when the machine starts up, nothing is found in control panel->administrative tools->device manager->Network devices and doing a "mst status" on from an administrative command prompt on this machine results in "no card found".

I am attacking this problem on two fronts at the moment - through asus and here through AMD, it's not just a particular card brand issue this is either a chipset, motherboard or bios issue and unless they fix it - i'll be going for a full refund.

[elstaci]

By any chance did you read the manual on the Mellenox ConnectX2 SFP cards. If not, here it is from Mellonox Website: http://www.mellanox.com/related-docs/user_manuals/ConnectX_2_%20VPI_UserManual_MHZH29.pdf .

It has links to where you can download Firmware and Drivers for the card at Mellenox Support. Here are the various drivers for ethernet for Windows OS: http://www.mellanox.com/page/software_overview_eth .

The cards need to be installed in a PCIe 2.0 x 8 (same as PCIe 3.0 x 4) motherboard slots to operate.

[falloutboy]

Yup - for reference both cards autodetect immediately when plugged into my

R5E10 and both provide PXE uefi bios startup messages before the loading of

Windows (My OS) and if you like in device manager you can see the Mellenox

card there.

Do this on the X399, no PXE boot message, nothing under device manager.

Physically the PCIe 2.0 x 8 fits quite happily into a PCie3 x 16 -

obviously only 8 lanes will be used as that is what is physically connected

- or at least it should be - I've tried the card in every slot barring the

video cards slot because of course I need that, I know the card works

because my R5E10 knows all about it - but in the X399 absolutely zero. I

have run up mst status on the card on my R5E10 I get this.

MST Devices

[elstaci]

Is this the BIOS you have installed on your X399a : PRIME X399-A | Motherboards | ASUS USA

BIOS

Version 06012018/05/025.29 MBytes

PRIME X399-A BIOS 0601
Update AGESA 1.0.0.5
recommend to keep C-state [enable]

copied this from the X399a Manual: http://dlcdnet.asus.com/pub/ASUS/mb/socketTR4/PRIME_X399-A/E13889_PRIME_X399-A_UM_V3_WEB.pdf?_ga=2.1...

Could it be possible that the slots are not running at the minimum PCIe 2.0 x 8 speed with two cards installed?

[falloutboy]

Yes Bios version 0601 and card currently running in slot 4 in x8 mode

although it refuses to work in any slot with or without any other PCIecards

installed

[misterj]

falloutboy, sounds like you have no driver at all installed.  MS will not have a driver for the card.  I think you need to go to Mellenox, DL and install the correct driver for your card.  Am I missing something obvious?  Enjoy, John.

[falloutboy]

MS does have a driver for the card, it installed by default on my rampage 5

extreme - didn't have to do a thing.. that also doesn't explain the lack of

PXE boot message and I have all the bios options for startup messages set

to UEFI an legacy reports.

Both these machines are basically identical barring a few things.

The R5E10 is running

a samsung 1TB PCIx4 SSD vs a WD 512GB PCIx4 SSD on the X399

an Intel 6900K vs AMD TR 1900x

the 6900x is running a few Sata II drives as well

Broadcom wireless Vs ASUS Intel wireless.

Barring odds and sods for differences in USB ports etc both machines are

running.

Gigabyte Aorus 1080 Ti's

Avago ( broadcom ) 9361 Raid Controller running 8 HGST 4Tb drives with

cachevault and supercap.

32 Gb 3000Mhz Corsair DDR4 4 Sticks at 8 Gig same model

Silverstone Strider ST1500 1500W power supplies

Windows 10 64 Bit pro.

Everything just works with no intervention on the R5E10 but seems to fail

on the X399

[elstaci]

Either your motherboard (X399a) is defective or it doesn't support the cards or there is a conflict with another component installed on your motherboard.

If your motherboard doesn't detect the cards, no drivers will be able to install since they don't see a compatible component installed.

[misterj]

falloutboy, I am very lost here, so my questions may be redundant.  What OS are you running (if Windows, command prompt - winver) and post the screenshot here.  Where did it come from?  If it is not a fresh install, please do one which may be a good idea anyway.  Can you see the card in the UEFI?  Are you running the latest UEFI for your MB?  Is CSM enabled?  Can you see the card in the Device Manager and if so does it have a yellow bang?  This is all weird.  Thanks and enjoy, John.

[falloutboy]

Windows 10 Professional Version 1803 Build 17134.81, it is a fresh install.

Where did it come from?? Microsoft. For reference the card is also not

shown under Ubuntu on my X399 but is shown on my R5E10. When the machine

boots and I go into bios there is no UEFI entry for the card, nor do I get

any PXE boot messages which I do on the R5E10. I have tried both with and

without CSM and Fastboot. The card showed in device manager the very first

time it was put in with a code 43 and then was never shown again after the

driver was deleted and the system rebooted to allow it to detect it again.

The card does not show up at all under windows 10 now and will not show at

all under Ubuntu on the X399.

It is very odd, either card in the R5E10 just works without me doing

anything but place them in the X399 - Zip, Nothing Nada.

[misterj]

Thanks, falloutboy.  You have shown remarkable patience with me.  What OS is running in the R5E10?  Have you tried one Mellenox card in the PCIex16_4 slot with the slot set to X8 in the UEFI?  If not, please do.  Is the connector on the Mellenox card a physical x8 size?  I assume you have done multiple Clear CMOS operations?  If not, please do, and always after changing HW (cards, etc).  I would like to see a complete list of your HW.  Here is mine, if you are curious:

ASRock Fatal1ty X399 Pro Gaming, Threadripper 1950X, 2xSamsung SSD 960 EVO RAID, 1TB & 500GB WD Black, G.SKILL [Flare X (for AMD)] F4-3200C14Q-32GFX, Windows 10 x64 Pro, Enermx Platimax 850, Enermx Liqtech TR4 CPU Cooler, Radeon RX580, BIOS 2.0

Do you have a Support Ticket open with your MB vendor?  Thanks and enjoy, John.

EDIT:  I tried code tags around my HW but they did not work.  Anyone help me here?

[falloutboy]

Windows 10 Professional 64 bit same rev and build - yes have tried that on

the X399 - no change, physically the card has an x8 connector. Yup have

done cmos resets, even went as far as battery removal. Yes I just sent asus

all the specs this morning and told them not to come back until it's fixed.

[hardcoregames_]

Those Asus 10GBASE-T cards usually work fine, I have a few of them on more mundane rigs and they work with a Cisco box I use.

If you LAN is sick this telling me your windows setup is mangled so use reset windows and see if that works with the gear you have

I use 1803 with clean installs and it works fine

[falloutboy]

The funny thing is though if I use the internal 1gb nics built into the

motherboards they work fine too... I would argue that that it's not

installed os or installation and I have done a full network reset on both.

There is some sort of board/bios/Chipset issue here

[hardcoregames_]

try a second disk, install 1803 on it and see if it works properly

[falloutboy]

I've thought about this issue very logically.

It's not the OS, I have created identical Bootable Ubuntu USB keys for my

two systems and get the same results I get from Windows 10 ( can't blame

Microsoft this time ).

If I boot up on the X399 using Ubuntu the Mellenox card is not seen and

there is nothing mentioning it the startup logs - so it's not a windows

issue.

If I boot up on the R5E10 using Ubuntu and have either of the two Mellenox

cards in that PC they show up under Ubuntu and the logs are full of

information about the configuration.

I can boot off either key on either machine and get the same result, the

R5E10 sees the card no mater which card or key I use, the X399 does not.

I also note that the R5E10 brings up PXE boot messages from the card - and

the X399 does not do this so this is either an:

X399 chipset issue

A BIOS issue

faulty motherboard ( I'm steered away from faulty motherboard simply by the

fact that I can put my 9361 Raid controller or a wireless network card in

any slot in any combination and they work, this points more toward Bios or

Chipset. )

Thanks for trying though but installing Windows 10 on a separate drive I

think you'll agree logically would be a waste of time.

On Sat, Jun 9, 2018 at 2:45 AM, hardcoregames™ <amd-external@jiveon.com>

[misterj]

Thanks, falloutboy.  I agree with you - this is not an OS issue.  This is a board/UEFI issue.  If your UEFI does not see the Mellenox card, you are dead meat, no matter the OS.  I looked at Newegg and Mellenox products are expensive except for a few from $20 - 50$.  Please take a look and see if any of the cheap ones would have a chance to duplicate your problem.  We could both purchase one each and compare results.  Thanks and enjoy, John.

[falloutboy]

Amazon sells them cheaper I got mine here in nz for 70nzd each but the

answer I feel is not to throw money at more nics when the ones ive got are

obviously fine, it's for the chipset/bios issue to be sorted.

[misterj]

OK, falloutboy.  I will give up on that path.  It looks like US Amazon has two cards for 41USD.  Please post your equivalent screenshot:

Requires CPU-Z 1.85 or above.  Thanks and enjoy, John.

[falloutboy]

As requested.

[misterj]

Thanks, falloutboy.  WOW, that AGESA is way down level!  Are you sure you are running the latest UEFI?  Please look and point me to what you are running.  As you probably know AGESA comes from AMD and we all should be running close to the same version.  ASRock has released a Beta UEFI with AGESA 1.0.0.6 (AGESA!V9.ThreadRipperPI-SP3r2-1.0.0.6) and you have 0.0.6.0 - sounds like an initial release.  This is a good candidate for your problem.  Let me hear.  Thanks and enjoy, John.

EDIT: Please look on your R5E10 and see what is there.

EDIT AGAIN:  I think this is it.  On the ASUS BIOS page version 0601 talks about AGESA 1.0.0.5 but contains 0.0.6.0.  BTW your board is an Extreme not Rampage.

EDIT#3: All the old UEFI files also show the AGESA as 0.0.6.0,  I talked to ASUS and they said they would look into it, but you could flash 0503.  You will need to use the FlaskBack (ASRock name) but it is a special USB socket on the back.

[falloutboy]

Latest version of the BIOS as per whats available on the ASUS website.

AGESA 1.0.0.5 ; however my understanding is that AGESA only deals with

initialization

of the processor cores, memory, and the HyperTransport controller - I have

not seen any reference to AGESA dealing with PCIe initialization but am

continuing to look.

[falloutboy]

Well it does look like it has something to do with PCI in the early stages

here is hoping that applying the squeeky wheel gets the grease treatment to

ASUS gets it sorted, I.E contacting them every 5 mins until its fixed.

[elstaci]

LoL, Nag Asus to death. LoL

[misterj]

falloutboy, I agree with elstaci - please give ASUS holy dickens!  Even if the AGESA version number just failed to get updated, this is an egregious error!  It shows extreme lack of attention to detail.  If anything needs careful attention it is the UEFI.  I hope I have not over hyped this, but it needs to be corrected and the underlying sloth needs to be corrected.  Running unknown code is unacceptable.  Have you considered going back to 0503?  Good hunting and enjoy, John.

[falloutboy]

I don't think it would help, I was running 0503 when I ran up their own

XGC-100Cs and while they detected they simply wouldn't work.

[misterj]

Thanks, falloutboy.  As you wish.  Have you heard anything  positive from ASUS?  The 0601 is not correct as of right now.  Enjoy, John.

[falloutboy]

So far some where between Squat and do diddlies - their support has become

so so bad.

[falloutboy]

Why are these people so SLOW, I posted them 48 hours ago, rang them

yesterday and then had to re-e-mail them to get things started... sheesh a

snail would move with more urgency than this.

On Mon, Jun 11, 2018 at 2:55 AM, Chasse Court <chasse.court@gmail.com>

[elstaci]

while browsing this AMD Forum:AMD Server Gurus I noticed something about a new tuning guide for the Mellanox NIC cards. Don't know if this applies to you are not. New Windows 2016 Mellanox 100GbE NIC Tuning Guide Posted on AMD Developer Central

[falloutboy]

Thanks for the thought - but unfortunately not - the machine would need to

be able to see the card in order to tune the card.

[elstaci]

LoL, I guess that was a "DOH" moment when I wrote that comment. I forgot that your motherboard doesn't recognize you NIC cards.

[falloutboy]

We all have those, i appreciate the thought though.

[misterj]

falloutboy, anything useful from ASUS?  Enjoy, John.

[falloutboy]

Only if you count useful as having to recount the same information to ASUS for the fifth time.

[elstaci]

I was glancing at the Mellanox Tuning guide and there was a link for Drivers for the Connectx-2,3,4 & 5.

This is what it shows for WINOF and WINOF2 Drivers: http://www.mellanox.com/page/winof_matrix?mtag=windows_sw_drivers .

Mellanox WinOF-2: Firmware - Driver Compatibility Matrix  it shows ConnectX-3,4,5 Only for Windows 8.1 and Windows 10 Client .

Mellanox WinOF: Firmware - Driver Compatibility Matrix it shows ConnectX-2 Compatible only with Windows 8.1 and 7.

Could it be because you have Windows 10 is the reason why it is not being seen? Just going by the compatibility chart for ConnectX-2,3,4 & 5.