43 Replies Latest reply on Oct 21, 2018 6:34 PM by scythefwd

    Home lab, aka the 32 headed beast

    scythefwd

      Part
      Spec
      CPUThreadripper 1950X
      MOTHERBOARDASRock x399 Taichi
      RAMG.skill Trident Z 32gb ddr4-3200
      STORAGE1x seagate 1tb platter, 1x 1tb WD platter, 250 gbSamsung 850 EVO
      VIDEOXFX Rx570 4gb
      CPU COOLEREnermax Liqtek tr4 240
      PSUCoolermaster 1000 watt
      CASENZXT s340

       

      Future plans, 3x 256gb samsung 960

      1tb samsung 850

      (creates the possibility to demonstrate tiered storage)

       

      meshing the front of the case to provide better air flow through the radiator

      pics to follow

       

      She's not as clean as she could be, but she runs cool, seems to not mind being a VM host

        • Re: Home lab, aka the 32 headed beast
          scythefwd

          She's now running 3.9ghz, 1.25v

          Ram is running at 3466 vs 3200.. no ill effects yet.

           

          Front of case is half modded...rear cut in liner is done, mesh is fitted.  Front cut still needs to happen to actually make the functional port.  Debating leaving grate unpainted.. or clear coated.

          • Re: Home lab, aka the 32 headed beast
            kxuping

            Awesome homelab PC.

            What settings did you use for the ram?

            What do you run on the PC?

              • Re: Home lab, aka the 32 headed beast
                scythefwd

                This will be running win 10 pro with hyper-v role active.  As my main PC, I'll be interacting with that instance of win 10...

                 

                It also has a free ipa instance on it.  Once I decide on my network configurations, there will be a smoothwall firewall/router and 3 security zones, file share based off of windows, an AD domain controller, probably an email server.. havent decided on which open source one I'll be using, a linux client or 2 (centOS and Ubuntu most likely).  There will also probably be a squid server on there.  Current project is to attempt to get active directory federated services to authenticate with HOTP or TOTP two factor authentication using the freeipa server as the authentication server.. and then having my windows AD client authenticated using 2fa to the ipa server.. Meaning native support for HOTP or TOTP in a windows domain.. and using a central authentication server without samba to authenticate my linux clients and windows clients.  Once I get that authentication working, then its on to secure baseline configs using gpo.. would really rock if I could get the LDAP accounts recognized in AD for ad group membership.. gpo filters, etc...

                 

                IPA will probably be doing dns and dhcp roles, though I may kick dhcp over to smoothwall (using reservations.. dhcp is just practice playing with one).

                 

                There may be other uses that happen.. but first I have to get authentication setup.. then on to pxe boot servers hosting various images.. pxe boot clients that are inexpensive, remote patching capabilities (preferable to manage both windows and linux environments.. patch level reporting is required)... After that, probably Identity management tied to a sql speaking database (postgre, mysql, or ms sql lite)...

                 

                 

                In short, its starting with the host os, and 3 vm's, moving on to probably 7-10 vm's in the future.. Low load so Im gonna keep core count down to 2-4 wherever possible..

                 

                Thats the short term goals.. the environment will be fluid based on what I want to focus on...

                 

                 

                Ram settings.. it seems happy using the xmp 2.0 3466 profile that is pre-configured on the mobo.. The ram itself is rated for 3200 by gskill.. It just happens to deal with 3466 in my system.. Nothing special  was done.  I havent tweaked the settings manually.  Im at 3.9ghz and 1.25 volts, it will do 4ghz at 1.3..

                 

                Adding the grill up front raised my temps 2-3C from no front on the case (exposed radiator), so there is an air flow restriction (duh.. look at it), but its still 7-10 degrees cooler than the front on without the grill.

                1 of 1 people found this helpful
              • Re: Home lab, aka the 32 headed beast
                kxuping

                I activated Hyper-V on Windows 10 pro by mistake when installing latest version of Docker. (When activating Hyper-V windows 10 is run inside a VM, for others reading this).

                I thought my PC had a virus or was defective. All games run like crap with stuttering and crazy lag even on notepad++, videos on Utube were unwatchable even on 360p. Network did not work for the first 4 minutes that windows has the desktop up and running, and download/upload speeds were really bad compared to my laptop just in the next ethernet port. sometimes it stopped sending/receiving network packets for minutes, then it came back. changing video resolution on 3 monitors took minutes.

                I realized I was running that awful creation when trying to run a Virtualbox VM the PC crashed hard every time not even a blue screen, black screen and weird sounds, had to power off. Vmware workstation pro did not crash the OS, but did not work anymore.

                After uninstalling Hyper V all started working normally. But some Hyper V network interfaces and other disposals were left on the system so a backup restore was necessary.

                Docker toolbox still uses Virtualbox instead of Hyper-V on windows, so that is what I use to run dockers when running windows.

                After my experience I understand why Vmware is still number 1 in paid virtualization and KVM is number 1 in free virtualization.

                I use Docker in Mac and Linux and none had this type of problems, they use their native virtualization options (non-hyper-V).

                Please share your experience after you have it running.

                1 of 1 people found this helpful
                  • Re: Home lab, aka the 32 headed beast
                    scythefwd

                    I have run hyper-v on both a 3770 (with a ubuntu guest) and on the threadripper (with a centos guest) at home.  I have ran hyper-v on 2012 and 2016 supporting roughly 100 vm's in a cluster with no problems as well.

                     

                    I run hyper-v with a centos guest at work just to dink around too.. and thats on a mediocre i5...   The virtual nics that get left over.. those were used to connect into the virtual switches, and you can delete those via powershell or through device manager.

                    Now, with hyper-v.. they recommend provisioning at least 10% more on hardware.. so if your desktop was a little short on ram, and then you spin up another vm.. then bam.. you're going to have stutter problems due to low ram (and possibly swapping all to hell).  I also refuse to thin provision either disks or ram, and I run minimum 2 procs per vm period.  I have played pcsx2 in two instances (one as admin, one as unprivileged user).. running ffx-2 (have two ripped copies on my desktop just to test this... Yes I own the game, no.. it's not pirating) and keeping 60 fps on both windows (at 720p)..

                     

                    I do find I run into some slight stuttering when only running at 3.4ghz on the TR, but kicking it up to 3.6 base clock and she runs smooth. 

                  • Re: Home lab, aka the 32 headed beast
                    scythefwd

                    Alo, if you didn't have the virtualization extensions enabled in the bios, and if your proc didn't support slat, it may have activated the feature.. but been broke as all hell from the get go.

                      • Re: Home lab, aka the 32 headed beast
                        kxuping

                        Thx for answering.

                        Not being able to run vms in virtualbox or vmware is a no go, so I did not even try to check Hyper-V configuration/documentation.

                        My PC has 64GB Ram and all vm optimizations active in BIOS, core info reports:

                        AMD Ryzen 7 1700X Eight-Core Processor

                        AMD64 Family 23 Model 1 Stepping 1, AuthenticAMD

                        Microcode signature: 08001129

                        HYPERVISOR      -       Hypervisor is present

                        SVM             *       Supports AMD hardware-assisted virtualization

                        NP              *       Supports AMD nested page tables (SLAT)

                          • Re: Home lab, aka the 32 headed beast
                            scythefwd

                            Interesting.. I did notice some latency till i oc'd it.  At current clock, I ran prime 95 torture test.. not small fft... Even my aio cannot keep up with that one... but the default torture test and I only saw 60 C...

                             

                            Best of luck on the ryzen.. had I been expecting to only run 2-3 vm's, is probably had gone with an 1800x or 1700x myself.

                             

                            I have heard that running 32 bit guests in hyper v was causing soft locks of both host and guest... But I've only ran 64 bit guests.. and been fine here

                              • Re: Home lab, aka the 32 headed beast
                                kxuping

                                I managed to run 3-4 vms running windows/linux with my old FX 9590 and 32GB Ram on vmware workstation pro without problem.

                                I decided to go with 2 Ryzen PCs instead of 1 TR. both have 64GB Ram, one R7 1700 (Linux) the other R7 1700X running windows and linux.

                                  • Re: Home lab, aka the 32 headed beast
                                    scythefwd

                                    Man, you have expensive tastes.. probably have as much in your two systems as I have in mine... Maybe more.  I had considered going that route, but the clustered win 2k12 data center would have completely blown the budget lol.  Hell, this pushed the limits of my budget.. I reused some hdd's, PSU, video card.  I do want to go to 64 or 128 GB ram and Putin some ssd's, maybe a Nas with an iscsi lunch on it.

                                     

                                    But we're talking another grand or two for that...

                                • Re: Home lab, aka the 32 headed beast
                                  scythefwd

                                  I'm surprised it didnt run well with that proc.. hell, my system isn't anything really different, just two of your procs and the infinity messh between them..

                                • Re: Home lab, aka the 32 headed beast
                                  scythefwd

                                  And one more thing.. if you can disable HEPT in the bios.. that seems to have an effect on smoothness...

                                • Re: Home lab, aka the 32 headed beast
                                  toasttheoriginal

                                  Sounds like a fun project.  Best of luck with it. Keep the info coming in.

                                  • Re: Home lab, aka the 32 headed beast
                                    scythefwd

                                    decided to relocate the fans on my rad today..

                                     

                                    Took both fans from the inside of the rad, and have them on the outside (under the grill in the front cover.. they just BARELY fit, they'd be in contact if I still had that part of the liner still in place...) pushing room air through the rad vs. pushing hot air through from inside the case out (got to thinking about it.. the whole case was bass ackwards for fans.. was basically running a neg pressure system.. I still am, but much less so (2x 120 in with obstruction, 2x out without .. but the exhausts probably have lower air movement)

                                     

                                    Anyways.. after looking at idle temps being down in the 23 C (7 degree temp drop), I decided to see if it would survive a prime95 small fft test.  1 hour, max temp reported by hware monitor was 66 degrees (throttle temp is at 68 on the tdie right?) and actually ended the test at 64 C.  I'm tempted to take the 120's I have as exhaust and mount them to the inside of the rad as a pull system...

                                     

                                    Or I may get some more of the enermax fans.. they seem to really do a number.  They're loud (air whir.. I keep all fans at max, none of this variable speed stuff here lol).

                                     

                                    Spec for this was ram at 3333.. as it started throwing errors under memorytest64 at 3466 (the test app crashed with 0 errors at 3333, so I'll run that).  The proc was at 4ghz, 1.275 v.. so I was able to back off .025 volts.. and it appears to be stable (I mean, if prime cant make it puke after an hour.. its fairly stable). 

                                    • Re: Home lab, aka the 32 headed beast
                                      scythefwd

                                      Took her up to 4.1 GHz tonight and ran prime 95 on blend . Max temp was 62, and average was closer to 55.  Twas a short run of an hour.  Ram still at 3333.  1.3 volts.  Cinnebench was a 344x.. I don't remember the last, but it seems to scale linerally with core speed.  From what I'm seeing, roughly 50 pts per 100 MHz.  Possibly see more advantage with faster ram. 

                                      • Re: Home lab, aka the 32 headed beast
                                        scythefwd

                                        4.2 today, prime95 stable for blend torture test, no throttle.

                                        1.35 voltage.. it would lock up at 1.325...

                                         

                                        Not willing to try for 4.3... I could probably get it to boot and venchmark at 1.4v.. but I'm not will gto try itat that voltage. 

                                        • Re: Home lab, aka the 32 headed beast
                                          scythefwd

                                          Shes now booting off a 250gb samsung 850

                                          • Re: Home lab, aka the 32 headed beast
                                            darkwolfca

                                            very nice pc looks good

                                            • Re: Home lab, aka the 32 headed beast
                                              scythefwd

                                              Because my case only supports 4 disks, and I want to run 8 with a fan blowing across them to disperse heat...  And it would allow for better cable management.

                                              • Re: Home lab, aka the 32 headed beast
                                                alexkywalker

                                                Nicely done Sir!

                                                 

                                                And yes, we are in agreement of the pain of installing that monster ENERMAX cooler; I was hesitant to put it in the front, because I heard/read that top installation is preferred, but I had no room in top, and the only way to make mine work was to use some tools and remove the HDD and SATA built-in enclosures, which means my case is pretty much hollow inside.... I am using NVME drives so I didn't have to worry about placing SATA SSDs anywhere, but still, proud.

                                                 

                                                Great job, cheers!

                                                • Re: Home lab, aka the 32 headed beast
                                                  Obi-WanKenobi

                                                  Neat workstation yet 32 heads nah it only has 16 as being 16 cores with 32 threads stilla 16 core CPU and not a 32 Core CPU, many tend to confuse threads as cores which they actually aren't.

                                                   

                                                  Other than that, would be a nice render system with some  Radeon Pro WX9100's Are you looking in to getting a TR 2990X in the future?
                                                  Would suit this system quite well

                                                  • Re: Home lab, aka the 32 headed beast
                                                    colesdav

                                                    Hi,

                                                     

                                                    Have you considering running  Linux as your main OS and running Windows 10 on a virtual machine inside Linux in future?

                                                    Bye.

                                                      • Re: Home lab, aka the 32 headed beast
                                                        scythefwd

                                                        I have, but since I'm a windows engineer, I'd be spending my whole time in

                                                        the VM.. Booting directly to the vm the way windows 10 does saves me a

                                                        little bit of annoyance  (for those that dont know, when you install

                                                        hyper-v, it basically installs itself at the HAL, and your desktop becomes

                                                        a virtual machine.. just one you boot directly into)

                                                      • Re: Home lab, aka the 32 headed beast
                                                        scythefwd

                                                        Well, the enermax is dying.. At idle it's maintaining ~50 degrees temps.. thats way too high for a liquid cooler at idle.  Time to see if I can shoehorn a custom loop in there.. wish me luck.

                                                          • Re: Home lab, aka the 32 headed beast
                                                            colesdav

                                                            Is this the cooler you have been running?

                                                            http://www.enermax.com/home.php?fn=eng/product_a1_1_1&lv0=109&lv1=118&no=369

                                                            Strange that it should start to fail so early, are you able to RMA it?

                                                            Maybe you could get a  a larger radiator on the AIO system or add a reservoir to it?
                                                            This company has some interesting solutions for that I think ... https://www.alphacool.com/products/
                                                            Good luck.

                                                              • Re: Home lab, aka the 32 headed beast
                                                                scythefwd

                                                                colesdav - yes, that is the exact cooler.  Yes, they're going to rma it.  No, I'm not going to use the one they send back (though I may keep it as a spare /backup to my new cooler).  That or I'll sell it off for a few bucks and use it to offset the cost of my new cooler.

                                                                 

                                                                The cooler itself has a 2 year warranty, so I'm definitely within that period. 

                                                                I cannot add a reservoir or extra rad to it without voiding the warranty.. I couldn't disassemble it without it being obvious it was tampered with (looks like it uses barbed fittings and shrink wrap to keep it in place.. )

                                                                 

                                                                I ended up ordering these for my system:

                                                                watercool heatkiller iv pro nickle plated copper waterblock (ties for the best performing waterblock in every roundup I've seen, usually besting the EK by several degrees)

                                                                xspc photon x4 170 reservoir and pump (which will get upgraded to a d5 pump when it fails, the d5 is the better pump, but I'm still trying to be cheap... it's a hard habit to kick)

                                                                xspc compression fittings for flex

                                                                xspc 240mm radiator

                                                                2 meters worth of flex so I should have enough to do whatever routing I want.

                                                                 

                                                                That should be a much better flowing system vs. the enermax, but at 3x the cost too.. I'm not sure what fans I intend to run ... IF I get a wild hair, I may add a 140 mm radiator in there as well and just daisy chain them. 

                                                                1 of 1 people found this helpful
                                                            • Re: Home lab, aka the 32 headed beast
                                                              scythefwd

                                                              Shes together, no leaks, running very cool.  4ghz, 3.125v.  idles at 24 C.  Blended prime 95 maxes temps at 65 degrees C.

                                                              I can live with that.

                                                               

                                                              water_cooled.jpg

                                                              • Re: Home lab, aka the 32 headed beast
                                                                scythefwd

                                                                Leak tested and running cool.. pretty lights.woo.. matching front lights on my ek rgb fans, controlled by mobo headers.

                                                                cooler lights on.jpg

                                                                • Re: Home lab, aka the 32 headed beast
                                                                  scythefwd

                                                                  she's stable at 4.0ghz and 1.238V  65C in aida65 stress test.  Ram at 3266.  Bios version at 3.20

                                                                  • Re: Home lab, aka the 32 headed beast
                                                                    scythefwd

                                                                    bios is at 3.30, not 3.20