24 Replies Latest reply on Aug 6, 2018 10:03 AM by mike_the_cfd_kitchen_hood_guy

    Need support on the AMD RAID driver for Linux Ubuntu

    tonytong

      Hi,

       

      I've recently put together a Ryzen 1700+ system with X370 chipset and installed dual boot Windows 10 and Ubuntu 16.04 systems (on a NVMe boot drive).   I have configured two RAID sets with RAID 0 and 1 respectively.  They work quite well under Windows, but I cannot get Ubuntu to find the RAID drives.  I did download the driver from the following webpage (Chipset ) and followed the instructions trying to insert the rcraid module to the Linux kernel.  The scripts seem to be successful and I could see that the rcraid.ko did get built and inserted into '/lib/modules/4.10.0-37-generic/kernel/drivers/scsi/' folder and the rest of steps.  However, after restarting, I used 'lsmod' to check, but couldn't find any 'raid' related module being loaded. 

       

      Can anyone help here?

       

      thank you,

      TT

        • Re: Need support on the AMD RAID driver for Linux Ubuntu
          mike_the_cfd_kitchen_hood_guy

          Had similar problem, but what I did that wasn't explained in the ISO readme is make the install file executable with "sudo chmod +x install".

           

          Once I rebooted, "sudo fdisk -l" saw the RAID.

          • Re: Need support on the AMD RAID driver for Linux Ubuntu
            tonytong

            * I did ran through the install script (output below)

            * I also saw that rcraid.ko got built and copied to "/lib/modules/4.10.0-37-generic/kernel/drivers/scsi/rcraid.ko"

            *  There is also load_amdraid script in folder "/usr/share/initramfs-tools/scripts/init-premount/"

            I tried to run the load_amdraid script myself, but got this error message:

            "insmod: ERROR: could not insert module /lib/modules/4.10.0-37-generic/kernel/drivers/scsi/rcraid.ko: File exists"

             

            Can you or anyone help further?  Thank you in advance!

             

            >>> output (partial) from install script on 4.10.0-37 kernel

            Make kernel for 4.10.0-37-generic

            make: Entering directory '/dd/driver_sdk/src'

            make: Leaving directory '/dd/driver_sdk/src'

            make: Entering directory '/dd/driver_sdk/src'

            ------------------------------------------------------------

            - building for kernel 4.10.0-37-generic

            ------------------------------------------------------------

            make -C /lib/modules/4.10.0-37-generic/build SUBDIRS=/dd/driver_sdk/src modules

            make[1]: Entering directory '/usr/src/linux-headers-4.10.0-37-generic'

              CC [M]  /dd/driver_sdk/src/rc_init.o

              CC [M]  /dd/driver_sdk/src/rc_msg.o

              CC [M]  /dd/driver_sdk/src/rc_mem_ops.o

              CC [M]  /dd/driver_sdk/src/rc_event.o

              CC [M]  /dd/driver_sdk/src/rc_config.o

              CC [M]  /dd/driver_sdk/src/vers.o

              LD [M]  /dd/driver_sdk/src/rcraid.o

              Building modules, stage 2.

              MODPOST 1 modules

              CC      /dd/driver_sdk/src/rcraid.mod.o

              LD [M]  /dd/driver_sdk/src/rcraid.ko

            make[1]: Leaving directory '/usr/src/linux-headers-4.10.0-37-generic'

            make: Leaving directory '/dd/driver_sdk/src'

            1 of 1 people found this helpful
              • Re: Need support on the AMD RAID driver for Linux Ubuntu
                ventman

                Haven't seen anybody help here, so I'll add to this. My board is a Gigabyte AX370-Gaming K5. When setting up the EFI, you need to make sure your not running the legacy RAID boot BIOS. you can use it to build the arrays, and boot from Windows, but your Linux system won't see your arrays. Set it to UEFI only. Check your kernel log, "dmesg | grep "rcraid" and look for a line like "rcraid: Array * not allowed by current licence level". If you see that, you know the module loaded, but it won't show disk arrays.

                2 of 2 people found this helpful
                  • Re: Need support on the AMD RAID driver for Linux Ubuntu
                    mawerick

                    I have the above mentioned "Array 1 not allowed by current license level" message and I do not know how to deal with it. Please help.

                    My board is an ASUS prime B350 plus and I am using Kubuntu 17.10. I installed Kubuntu on an external USB drive with the SATA configuration set to RAID, compiled the rcraid.ko and inserted it into the kernel. This all works fine. With help of the install script that is in the AMD package, the kernel module got inserted into the initrd and is automatically loaded at boot. The rcraid.ko module detects my disks. I have access to the CDROM via the rcraid.ko kernel module once it is loaded. However, it does not allow access to the hard drives. Looking into the UEFI BIOS, my current license level is "10". The two WD 2TB disks I have were configured as RAID 1. With Windows they are running fine. I upgraded the AMD RAID BIOS to version 8.1.0-00064. The rcraid kernel module is version 8.1.0-00039. Can this be an issue? The windows driver I was using during Windows install has a version 8.1.0-00026, and this worked fine. Here is the output when inserting the kernel module:

                     

                    <code>

                    rcraid: loading out-of-tree module taints kernel.

                    rcraid: module license 'Proprietary' taints kernel.

                    Disabling lock debugging due to kernel taint

                    rcraid: module verification failed: signature and/or required key missing - tainting kernel

                    <5>AMD, Inc. rcraid raid driver version 8.1.0 build_number 8.1.0-00039 built Mär 03 2018

                    <5>rcraid built on kubuntu by root on Sa 3. Mär 11:39:52 CET 2018

                    <5>rcraid: cmd_q_depth 512, tag_q_depth 16, max_xfer 448, use_swl 0xffffffff

                    <5>rcraid_probe_one: vendor = 0x1022 device 0x43bd

                    <5>rcraid_probe_one: vendor = 0x1022 device 0x7916

                    <5>rcraid_probe_one: Total adapters matched 2

                    <5>rcraid: rc_init_adapter 64 bit DMA enabled

                    <6>### rc_init_adapter(): RC_EnableZPODD = 0

                    <3>rcraid:0 request_threaded_irq irq 297

                    <5>rcraid: card 0: AMD, Inc. AHCI

                    <5>rcraid: rc_init_adapter 64 bit DMA enabled

                    <6>### rc_init_adapter(): RC_EnableZPODD = 0

                    <3>rcraid:1 request_threaded_irq irq 308

                    <5>rcraid: card 1: AMD, Inc. AHCI

                    rcraid: (1520073620.908596)

                    Array 1 not allowed by current license level.

                    rcraid: (1520073620.908619)

                    Array 1 not allowed by current license level.

                    <6>rcraid: rc_event: config change detected on bus 0

                    <4>rc_msg_process_srb: Bus/Target/Lun 0/0/0 OFFLINE

                    scsi host0: AMD, Inc. AMD-RAID

                    <4>rc_msg_process_srb: Bus/Target/Lun 0/0/0 OFFLINE

                    scsi 0:0:24:0: Processor         AMD-RAID Configuration    V1.2 PQ: 0 ANSI: 5

                    scsi 0:1:0:0: CD-ROM            HL-DT-ST DVDRAM GH22NS50  TN03 PQ: 0 ANSI: 0

                    scsi 0:0:24:0: Attached scsi generic sg2 type 3

                    sr 0:1:0:0: [sr0] scsi-1 drive

                    cdrom: Uniform CD-ROM driver Revision: 3.20

                    sr 0:1:0:0: Attached scsi CD-ROM sr0

                    sr 0:1:0:0: Attached scsi generic sg3 type 5

                    ISO 9660 Extensions: Microsoft Joliet Level 3

                    ISOFS: changing to secondary root

                    </code>

                      • Re: Need support on the AMD RAID driver for Linux Ubuntu
                        mdec

                        Sometimes I wish AMD forums had a way to escalate things and file JIRA tickets to their devs.

                        • Re: Need support on the AMD RAID driver for Linux Ubuntu
                          ventman

                          I looked into your boards UEFI menu's, go under the "Boot" tab, look under "Boot Configuration". Look for anything you can change that involves legacy boot, to "UEFI only". Also, under the Boot tab is "CSM" or Compatibility Support Module, try turning this on and off. If your Graphics card doesn't support UEFI, your board will keep toggling this on so it can boot.  You'll know you have "pure" UEFI when you stop seeing option ROM's related to disk controllers and ethernet devices after the first POST screen. Also, look under the "Advanced" tab, in "SATA Configuration" (or under the Advanced tab, can't tell) you should see an option to configure "AMD RAIDXpert2", once you see this, your OS should be able to mount the array as a regular "/dev/sd*" for each array.

                           

                          Another big thing, if your on any Kernel past the 4.10 branch, the driver module won't compile correctly.  Considering the fact that this software RAID wasn't developed by AMD, but an outside firm that's already been aquired twice by other companies since they made the contract with AMD, I hold little hope this get's fixed quickly.

                          1 of 1 people found this helpful
                            • Re: Need support on the AMD RAID driver for Linux Ubuntu
                              mawerick

                              ventman, thank you for your advice. I solved the problem indeed by switching off the CSM (Compatibility support module) in UEFI BIOS and installing Ubuntu in UEFI mode. The best thing ist that I could keep the installed Windows 10 without any data loss or the need to reinstall.

                               

                              Since the procedure is not easy, I created a full guide on How to dual boot Windows 10 and Ubuntu Linux on AMD RAID. Let me know if you find this helpful.

                               

                               

                                • Re: Need support on the AMD RAID driver for Linux Ubuntu
                                  ventman

                                  Glad I could help, I figure anyone that's been an AMD-only buyer hasn't had much exposure to UEFI. I pulled my hair out over this for days until I stumbled on the answer through trial and error.

                                  • Re: Need support on the AMD RAID driver for Linux Ubuntu
                                    mraust1n

                                    Wow! Quite the write-up you did there - awesome! I came across this thread as I'm having an impossible time installing Ubuntu 16.4 to a RAID (x399 Taichi MB and 1900x ThreadRipper).  I may be asking a ridiculous question here but seeing as how i'm dealing with all of the same files that you were and similar processes, I'm curious how possible it might be to follow the same steps and see the desired outcome? I saw you used the Linux drivers at one point while installing Ubuntu, which i thought was interesting. Regardless, you know much more about Linux than I do and just thought i'd throw the thought out there, as I've spent just over 20 hours on mine with no success. Then again, i'm not as well versed in Linux as you and the rest of the community.

                                     

                                    Anything is appreciated!

                                      • Re: Need support on the AMD RAID driver for Linux Ubuntu
                                        mdec

                                        Not the guy you responded to, but you might want to consider a software RAID like mdadm or the one from ZFS. When your RAID controller starts an array, it basically writes a header to each drive, manages them and offers a logical block device. The kind of header varies with implementation and vendor. For example, arrays created with an nVidia nForce 4 or AMD 970FX RAID ctrlr will not be accessible when plugged into an LSI MegaRAID ctrlr. The novelty of a standalone RAID controller is that it lifts the CPU and memory load from the motherboard's resources, as it has its own CPU and RAM. Thing is, this isn't a huge amount of resources, we're talking about 0.1% load on a Ryzen 2700 with peak use 500MB RAM. With mdadm you take that cost, but your data can go from one system to another easily and without any drivers required. I wasted 5 months looking for a perfect ctrlr (cheap, reliable, fast, upstream supported) and eventually I bit the bullet. It's not what you were looking and asking for, but think about it. Also, mdadm and ZFS do individual drive healthchecks, whereas controllers might not depending on the model.

                                          • Re: Need support on the AMD RAID driver for Linux Ubuntu
                                            mraust1n

                                            Thanks very much for the response! After having re-read ventman's write-up in detail, I realized that I had to go another route. I'm thinking of doing exactly what you said regarding a software RAID. I'd love to opt for a secondary standalone RAID controller BUT I'm all out of pcie slots. The system I'm working on is destined for deep-learning, which causes a concern for using software RAID. I need to do some additional research regarding the amount of resources mdadm or ZFS might consume. From your experience, is it something that can be estimated prior to build or is it always a case-by-case basis? If I go this route, I may upgrade my SSHD's to SSDs to improve IO.

                                             

                                            Lastly, if you wouldn't mind sharing some expertise, would you recommend the installation of the OS (Ubuntu 16.4) on that RAID or better to be on a small SSD, making available the mdadm software RAID (5x2TB SSHD/SSD) to that system for use?

                                             

                                            Lastly, thanks for chiming in. I really appreciate the help! I need to make a decision in the next 24hrs, as my return period will pass at that time on equipment I'm currently working with.

                                              • Re: Need support on the AMD RAID driver for Linux Ubuntu
                                                mdec

                                                I don't think this can be pre-calculated, but I'll give you a rough set of figures because I only tried this once. The laptop I used ZFS on has:
                                                - 1st gen Intel Core i7, 1.6GHz
                                                - 8GB RAM
                                                - 465GiB HDD, 7200RPM
                                                When the system booted and let me log in via TTY terminals, it used about 960MB of RAM and it was a little slow in the IO department.
                                                You should remember that ZFS has 2 features: ARC and L2ARC. The former is adaptive redundancy cache and it lives in RAM exclusively. Depending on the size of your ZFS pool, ARC parameters and available RAM, ZFS may eat varying amounts of RAM to sustain itself. L2ARC is a level 2 ARC and you can dedicate SSDs to be that.
                                                If you go with software RAID via mdadm or ZFS, your wise option would be to use the fastest devices for cache. mdadm does not support a thing like that though, so you will have to set up LVM on the resulting mdadm device. LVM supports cache devices
                                                Edit: forgot to mention. mdadm will keep using the same amount of RAM no matter what and for 4x 1.8TiB drives in RAID10, it used 60MB less than my laptop ZFS setup.

                                            • Re: Need support on the AMD RAID driver for Linux Ubuntu
                                              mawerick

                                              Hi mraust1n, the recipe I linked above works on Ubuntu < 18.04. With Ubuntu 18.04, you need another patch to make it work. I am not a fan of closed-source software and tainted kernel modules, as it is when using AMD RAID linux driver. Therefore I can understand people to recommend using software raid like dm-raid or other tools. For me, the decision was in favor of the AMD RAID because I wanted to access the same RAID both from Windows and Linux. I can mount Linux drives from Windows and Windows drives from Linux now, both living on the same RAID. If you do not need to access Linux drives from Windows and vice versa, do not use AMD RAID. There are better solutions.

                                    • Re: Need support on the AMD RAID driver for Linux Ubuntu
                                      mike_the_cfd_kitchen_hood_guy

                                      Guys,

                                       

                                      A lot of what your describing can be solved with the AMD raid driver. I'm running a Gigabyte X399 Aorus Gaming 7 with Ubuntu 16.04. While I did set this up nearly 8 months ago, the big thing I found with this older version of Ubuntu was that the driver needed to be loaded in to the OS before you could make good progress. I am able to use the hardware raid, and did not need to switch to a software raid to make this work.

                                       

                                      If you have questions on how to get this setup, reach out, watching many of you struggle with this is painful to watch.

                                       

                                      If the link above is broken, here it is:

                                      https://support.amd.com/en-us/download/chipset?os=Linux+x86_64

                                    • Re: Need support on the AMD RAID driver for Linux Ubuntu
                                      mdec

                                      At this point I'm considering buying a REAL array controller with REAL upstream support in the kernel under a normal license.

                                      1 of 1 people found this helpful
                                      • Re: Need support on the AMD RAID driver for Linux Ubuntu
                                        ramchyld

                                        Honestly, I think AMD should just hand the needed information over to the kernel and dmraid/kpartx people to add support. I managed to get rcraid.ko built thanks to the hint provided on this thread. However NVMe RAID doesn't work and even then SATA RAID isn't working how I expected it to (it manifested itself as /dev/sda with my boot SATA SSD pushed down to /dev/sdb, and the controller and disk isn't showing up in /dev/mapper. On the plus side tho, at least Clonezilla supports imaging this configuration).

                                         

                                        Why do they have to make things so complicated.

                                          • Re: Need support on the AMD RAID driver for Linux Ubuntu
                                            ventman

                                            Because, Like I hinted in that thread, AMD doesn't own the software we're having problems with. Honestly, AMD has really only been interested in the CPU portion of their platform.  It was only after Buying ATi, and inheriting their chipset tech that AMD was able to make a complete platform on their own (the 700 series) and have been making only slight revisions for the 800 and 900 series. It was really only with the introduction of APU's in 2011 that AMD had to actually put effort into creating a chipset it can call it's own.  Honestly, I love what AMD has done with the graphics side of ATi, putting in the effort to jump through the legal hoops to get as much technical info as possible out to the community to open source, but when you don't own what your selling, you play by their rules.

                                              • Re: Need support on the AMD RAID driver for Linux Ubuntu
                                                mdec

                                                I dived into the software with a couple debugging tools and you're absolutely right, all of the RAID software is outsourced and then released on joke licenses. It really seems that AMD only cares for free software when it comes to their GPU drivers...

                                                  • Re: Need support on the AMD RAID driver for Linux Ubuntu
                                                    ramchyld

                                                    I don't know about that. Not even sure that they care about open-sourcing the GPU drivers. I'm running Ubuntu 17.10. Blender won't let me do GPU rendering. I did reading and apparently only the AMDGPU-Pro drivers can do rendering. So I tried to install that... And it didn't work and caused my system to boot to a blank screen. Because only the LTS release is supported, which is a major joke. I don't want to use LTS because the version of Blender on that one is several years old at this point and I want to use a version that is at least less than six months old.

                                                     

                                                    In fact, is there a good reason to support Zen on LTS? Even 17.10 doesn't support the X399 temperature, voltage and fan sensors. I can't get a reading on any of them from lm-sensors. I'm thinking it would be worse on LTS which doesn't even support the Zen architecture natively, since it's kernel is from a year before Zen was introduced.