Log in to create and rate content, and to follow, bookmark, and share content with other members. Not a member? Join Now!

Home lab, aka the 32 headed beast

Discussion created by scythefwd on Mar 27, 2018
Latest reply on Oct 21, 2018 by scythefwd

Like • Show 5 Likes5
Comment • 44

Part	Spec
CPU	Threadripper 1950X
MOTHERBOARD	ASRock x399 Taichi
RAM	G.skill Trident Z 32gb ddr4-3200
STORAGE	1x seagate 1tb platter, 1x 1tb WD platter, 250 gbSamsung 850 EVO
VIDEO	XFX Rx570 4gb
CPU COOLER	Enermax Liqtek tr4 240
PSU	Coolermaster 1000 watt
CASE	NZXT s340

Future plans, 3x 256gb samsung 960

1tb samsung 850

(creates the possibility to demonstrate tiered storage)

meshing the front of the case to provide better air flow through the radiator

pics to follow

She's not as clean as she could be, but she runs cool, seems to not mind being a VM host

Attachments

IMG_20180329_183031.jpg878.0 KB

Outcomes

43 Replies

scythefwd Mar 31, 2018 8:12 PM
She's now running 3.9ghz, 1.25v
Ram is running at 3466 vs 3200.. no ill effects yet.

Front of case is half modded...rear cut in liner is done, mesh is fitted. Front cut still needs to happen to actually make the functional port. Debating leaving grate unpainted.. or clear coated.
Attachments
- IMG_20180331_193208.jpg1.2 MB
Like • Show 1 Like1
Actions
- scythefwd @ scythefwd on Mar 31, 2018 10:58 PM
  Well, now she can breath correctly ... Edges could be filed.. could use a paint on the front.. but she's a workhorse, not a showdog.
  Like • Show 2 Likes2
  Actions
  - kc5vdj @ scythefwd on Oct 18, 2018 11:10 PM
    Reminds me of mods people did for the corsair 750D before they came out with the high-airflow front panel replacement.
    That'll definitely get the airflow in.
    Like • Show 0 Likes0
    Actions
    - scythefwd @ kc5vdj on Oct 21, 2018 6:34 PM
      dropped temps by over 10 degrees.. and even allows me to go push/pull on the radiator if I want to.. not that I find a need for it.
      Like • Show 1 Like1
      Actions
kxuping Apr 1, 2018 4:27 PM
Awesome homelab PC.
What settings did you use for the ram?
What do you run on the PC?
Like • Show 0 Likes0
Actions
- scythefwd @ kxuping on Apr 1, 2018 5:08 PM
  This will be running win 10 pro with hyper-v role active. As my main PC, I'll be interacting with that instance of win 10...
  
  It also has a free ipa instance on it. Once I decide on my network configurations, there will be a smoothwall firewall/router and 3 security zones, file share based off of windows, an AD domain controller, probably an email server.. havent decided on which open source one I'll be using, a linux client or 2 (centOS and Ubuntu most likely). There will also probably be a squid server on there. Current project is to attempt to get active directory federated services to authenticate with HOTP or TOTP two factor authentication using the freeipa server as the authentication server.. and then having my windows AD client authenticated using 2fa to the ipa server.. Meaning native support for HOTP or TOTP in a windows domain.. and using a central authentication server without samba to authenticate my linux clients and windows clients. Once I get that authentication working, then its on to secure baseline configs using gpo.. would really rock if I could get the LDAP accounts recognized in AD for ad group membership.. gpo filters, etc...
  
  IPA will probably be doing dns and dhcp roles, though I may kick dhcp over to smoothwall (using reservations.. dhcp is just practice playing with one).
  
  There may be other uses that happen.. but first I have to get authentication setup.. then on to pxe boot servers hosting various images.. pxe boot clients that are inexpensive, remote patching capabilities (preferable to manage both windows and linux environments.. patch level reporting is required)... After that, probably Identity management tied to a sql speaking database (postgre, mysql, or ms sql lite)...
  
  In short, its starting with the host os, and 3 vm's, moving on to probably 7-10 vm's in the future.. Low load so Im gonna keep core count down to 2-4 wherever possible..
  
  Thats the short term goals.. the environment will be fluid based on what I want to focus on...
  
  Ram settings.. it seems happy using the xmp 2.0 3466 profile that is pre-configured on the mobo.. The ram itself is rated for 3200 by gskill.. It just happens to deal with 3466 in my system.. Nothing special was done. I havent tweaked the settings manually. Im at 3.9ghz and 1.25 volts, it will do 4ghz at 1.3..
  
  Adding the grill up front raised my temps 2-3C from no front on the case (exposed radiator), so there is an air flow restriction (duh.. look at it), but its still 7-10 degrees cooler than the front on without the grill.
  1 person found this helpful
  Like • Show 1 Like1
  Actions
kxuping Apr 1, 2018 5:31 PM
I activated Hyper-V on Windows 10 pro by mistake when installing latest version of Docker. (When activating Hyper-V windows 10 is run inside a VM, for others reading this).
I thought my PC had a virus or was defective. All games run like crap with stuttering and crazy lag even on notepad++, videos on Utube were unwatchable even on 360p. Network did not work for the first 4 minutes that windows has the desktop up and running, and download/upload speeds were really bad compared to my laptop just in the next ethernet port. sometimes it stopped sending/receiving network packets for minutes, then it came back. changing video resolution on 3 monitors took minutes.
I realized I was running that awful creation when trying to run a Virtualbox VM the PC crashed hard every time not even a blue screen, black screen and weird sounds, had to power off. Vmware workstation pro did not crash the OS, but did not work anymore.
After uninstalling Hyper V all started working normally. But some Hyper V network interfaces and other disposals were left on the system so a backup restore was necessary.
Docker toolbox still uses Virtualbox instead of Hyper-V on windows, so that is what I use to run dockers when running windows.
After my experience I understand why Vmware is still number 1 in paid virtualization and KVM is number 1 in free virtualization.
I use Docker in Mac and Linux and none had this type of problems, they use their native virtualization options (non-hyper-V).
Please share your experience after you have it running.
1 person found this helpful
Like • Show 1 Like1
Actions
- scythefwd @ kxuping on Apr 1, 2018 5:46 PM
  I have run hyper-v on both a 3770 (with a ubuntu guest) and on the threadripper (with a centos guest) at home. I have ran hyper-v on 2012 and 2016 supporting roughly 100 vm's in a cluster with no problems as well.
  
  I run hyper-v with a centos guest at work just to dink around too.. and thats on a mediocre i5... The virtual nics that get left over.. those were used to connect into the virtual switches, and you can delete those via powershell or through device manager.
  Now, with hyper-v.. they recommend provisioning at least 10% more on hardware.. so if your desktop was a little short on ram, and then you spin up another vm.. then bam.. you're going to have stutter problems due to low ram (and possibly swapping all to hell). I also refuse to thin provision either disks or ram, and I run minimum 2 procs per vm period. I have played pcsx2 in two instances (one as admin, one as unprivileged user).. running ffx-2 (have two ripped copies on my desktop just to test this... Yes I own the game, no.. it's not pirating) and keeping 60 fps on both windows (at 720p)..
  
  I do find I run into some slight stuttering when only running at 3.4ghz on the TR, but kicking it up to 3.6 base clock and she runs smooth.
  Like • Show 1 Like1
  Actions
scythefwd Apr 1, 2018 6:40 PM
Alo, if you didn't have the virtualization extensions enabled in the bios, and if your proc didn't support slat, it may have activated the feature.. but been broke as all hell from the get go.
Like • Show 1 Like1
Actions
- kxuping @ scythefwd on Apr 1, 2018 7:54 PM
  Thx for answering.
  Not being able to run vms in virtualbox or vmware is a no go, so I did not even try to check Hyper-V configuration/documentation.
  My PC has 64GB Ram and all vm optimizations active in BIOS, core info reports:
  AMD Ryzen 7 1700X Eight-Core Processor
  AMD64 Family 23 Model 1 Stepping 1, AuthenticAMD
  Microcode signature: 08001129
  HYPERVISOR      -       Hypervisor is present
  SVM             *       Supports AMD hardware-assisted virtualization
  NP              *       Supports AMD nested page tables (SLAT)
  Like • Show 0 Likes0
  Actions
  - scythefwd @ kxuping on Apr 1, 2018 8:53 PM
    Interesting.. I did notice some latency till i oc'd it. At current clock, I ran prime 95 torture test.. not small fft... Even my aio cannot keep up with that one... but the default torture test and I only saw 60 C...
    
    Best of luck on the ryzen.. had I been expecting to only run 2-3 vm's, is probably had gone with an 1800x or 1700x myself.
    
    I have heard that running 32 bit guests in hyper v was causing soft locks of both host and guest... But I've only ran 64 bit guests.. and been fine here
    Like • Show 0 Likes0
    Actions
    - kxuping @ scythefwd on Apr 3, 2018 10:56 PM
      I managed to run 3-4 vms running windows/linux with my old FX 9590 and 32GB Ram on vmware workstation pro without problem.
      I decided to go with 2 Ryzen PCs instead of 1 TR. both have 64GB Ram, one R7 1700 (Linux) the other R7 1700X running windows and linux.
      Like • Show 0 Likes0
      Actions
      - scythefwd @ kxuping on Apr 4, 2018 12:13 AM
        Man, you have expensive tastes.. probably have as much in your two systems as I have in mine... Maybe more. I had considered going that route, but the clustered win 2k12 data center would have completely blown the budget lol. Hell, this pushed the limits of my budget.. I reused some hdd's, PSU, video card. I do want to go to 64 or 128 GB ram and Putin some ssd's, maybe a Nas with an iscsi lunch on it.
        
        But we're talking another grand or two for that...
        Like • Show 0 Likes0
        Actions
  - scythefwd @ kxuping on Apr 3, 2018 9:53 AM
    I'm surprised it didnt run well with that proc.. hell, my system isn't anything really different, just two of your procs and the infinity messh between them..
    Like • Show 0 Likes0
    Actions
- scythefwd @ scythefwd on Apr 2, 2018 9:35 AM
  And one more thing.. if you can disable HEPT in the bios.. that seems to have an effect on smoothness...
  Like • Show 0 Likes0
  Actions
  - kxuping @ scythefwd on Apr 3, 2018 10:57 PM
    Will try in the future, but for the moment I need to run virtualbox and vmware vms, so no Hyper-V for the moment.
    Like • Show 0 Likes0
    Actions
toasttheoriginal Apr 2, 2018 9:01 AM
Sounds like a fun project. Best of luck with it. Keep the info coming in.
Like • Show 0 Likes0
Actions
scythefwd Apr 2, 2018 5:28 PM
decided to relocate the fans on my rad today..

Took both fans from the inside of the rad, and have them on the outside (under the grill in the front cover.. they just BARELY fit, they'd be in contact if I still had that part of the liner still in place...) pushing room air through the rad vs. pushing hot air through from inside the case out (got to thinking about it.. the whole case was bass ackwards for fans.. was basically running a neg pressure system.. I still am, but much less so (2x 120 in with obstruction, 2x out without .. but the exhausts probably have lower air movement)

Anyways.. after looking at idle temps being down in the 23 C (7 degree temp drop), I decided to see if it would survive a prime95 small fft test. 1 hour, max temp reported by hware monitor was 66 degrees (throttle temp is at 68 on the tdie right?) and actually ended the test at 64 C. I'm tempted to take the 120's I have as exhaust and mount them to the inside of the rad as a pull system...

Or I may get some more of the enermax fans.. they seem to really do a number. They're loud (air whir.. I keep all fans at max, none of this variable speed stuff here lol).

Spec for this was ram at 3333.. as it started throwing errors under memorytest64 at 3466 (the test app crashed with 0 errors at 3333, so I'll run that). The proc was at 4ghz, 1.275 v.. so I was able to back off .025 volts.. and it appears to be stable (I mean, if prime cant make it puke after an hour.. its fairly stable).
Like • Show 0 Likes0
Actions
scythefwd Apr 2, 2018 11:46 PM
Took her up to 4.1 GHz tonight and ran prime 95 on blend . Max temp was 62, and average was closer to 55. Twas a short run of an hour. Ram still at 3333. 1.3 volts. Cinnebench was a 344x.. I don't remember the last, but it seems to scale linerally with core speed. From what I'm seeing, roughly 50 pts per 100 MHz. Possibly see more advantage with faster ram.
Like • Show 0 Likes0
Actions
scythefwd Apr 3, 2018 7:45 PM
4.2 today, prime95 stable for blend torture test, no throttle.
1.35 voltage.. it would lock up at 1.325...

Not willing to try for 4.3... I could probably get it to boot and venchmark at 1.4v.. but I'm not will gto try itat that voltage.
Like • Show 0 Likes0
Actions
scythefwd Apr 11, 2018 10:51 AM
Shes now booting off a 250gb samsung 850
Like • Show 0 Likes0
Actions
darkwolfca May 2, 2018 4:00 PM
very nice pc looks good
Like • Show 0 Likes0
Actions
- scythefwd @ darkwolfca on May 3, 2018 9:12 PM
  Thx.. I've got some insane storage dreams involving custom brackets 3d printed.. but I doubt they'll ever be realized lol
  Like • Show 0 Likes0
  Actions
  - kxuping @ scythefwd on May 7, 2018 9:55 PM
    Why are you going to use 3d printed brackets?
    Like • Show 0 Likes0
    Actions
scythefwd May 7, 2018 10:00 PM
Because my case only supports 4 disks, and I want to run 8 with a fan blowing across them to disperse heat... And it would allow for better cable management.
Like • Show 1 Like1
Actions
- kxuping @ scythefwd on May 13, 2018 12:02 PM
  Please post more information/photos/howto of that. Very interesting. How do you plan on using the disks?
  Like • Show 0 Likes0
  Actions
  - scythefwd @ kxuping on May 17, 2018 8:49 PM
    howtwo? just design the brackets so they hold the disks..
    Plan on using the disks.. 2-4tb of raid 10 at half the cost of doing it with nmve...
    Like • Show 0 Likes0
    Actions
    - kxuping @ scythefwd on May 20, 2018 3:01 PM
      Yes, that is very interesting, to design the brackets and how its built
      Like • Show 0 Likes0
      Actions
alexkywalker Jul 16, 2018 1:36 PM
Nicely done Sir!

And yes, we are in agreement of the pain of installing that monster ENERMAX cooler; I was hesitant to put it in the front, because I heard/read that top installation is preferred, but I had no room in top, and the only way to make mine work was to use some tools and remove the HDD and SATA built-in enclosures, which means my case is pretty much hollow inside.... I am using NVME drives so I didn't have to worry about placing SATA SSDs anywhere, but still, proud.

Great job, cheers!
Like • Show 0 Likes0
Actions
Obi-WanKenobi Jul 16, 2018 1:58 PM
Neat workstation yet 32 heads nah it only has 16 as being 16 cores with 32 threads stilla 16 core CPU and not a 32 Core CPU, many tend to confuse threads as cores which they actually aren't.

Other than that, would be a nice render system with some Radeon Pro WX9100's Are you looking in to getting a TR 2990X in the future?
Would suit this system quite well
Like • Show 0 Likes0
Actions
- scythefwd @ Obi-WanKenobi on Jul 16, 2018 2:01 PM
  I'm fully aware of the difference between threads and cores.. and when
  running virualization.. threads matter more
  
  On Mon, Jul 16, 2018 at 2:59 PM, Obi-WanKenobi <amd-external@jiveon.com>
  Like • Show 1 Like1
  Actions
  - kxuping @ scythefwd on Jul 16, 2018 2:19 PM
    And you can see 32 Penguins if using Linux
    Like • Show 0 Likes0
    Actions
    - scythefwd @ kxuping on Jul 16, 2018 2:21 PM
      You see 32 processors in windows device manager too
      Like • Show 0 Likes0
      Actions
  - Obi-WanKenobi @ scythefwd on Jul 16, 2018 3:40 PM
    Cores always do more than threads, especially with virtualization apps, and of course if Multi / Hyper Threading is supported there will be a minor performance increase but then again, real cores will always do more no matter how you take it, for VF computing I'd go Dual 16 Core EPYC 7351, these have each 64MB L3 and 8 channel DDR4, which can be a major advantage over them TR's withonly 32mB L3 and 4 channel memory.
    
    But yeah if I'd want 32 threads Id go for 32 cores, just prefer the real deal man a shame AMD went the Intel way with this Multi Threading thing, a true 32 Core CPU with 32 threads and 1 FPU per core would of been something better for all us workstation junkies.
    Keep us updated on your system's progress tho, looks nice still
    Like • Show 0 Likes0
    Actions
    - scythefwd @ Obi-WanKenobi on Jul 17, 2018 6:38 AM
      True, but at the time of purchasing.. AMD provided most cores for the buck
      (still do). Intel didn't have anything that was within 500 dollars (closer
      to 700) with even the same number of cores. Yes, a dedicated core is more
      productive.. but the hyperthreaded processors does make a little overhead
      available.. I'm not saying I'd depend on it in a production environment..
      but for a home lab that doubles as my main pc.. it was the best fit.
      
      On Mon, Jul 16, 2018 at 4:40 PM, Obi-WanKenobi <amd-external@jiveon.com>
      Like • Show 1 Like1
      Actions
colesdav Jul 16, 2018 2:30 PM
Hi,

Have you considering running Linux as your main OS and running Windows 10 on a virtual machine inside Linux in future?

Bye.
Like • Show 0 Likes0
Actions
- scythefwd @ colesdav on Jul 17, 2018 6:35 AM
  I have, but since I'm a windows engineer, I'd be spending my whole time in
  the VM.. Booting directly to the vm the way windows 10 does saves me a
  little bit of annoyance (for those that dont know, when you install
  hyper-v, it basically installs itself at the HAL, and your desktop becomes
  a virtual machine.. just one you boot directly into)
  Like • Show 0 Likes0
  Actions
scythefwd Sep 25, 2018 8:52 AM
Well, the enermax is dying.. At idle it's maintaining ~50 degrees temps.. thats way too high for a liquid cooler at idle. Time to see if I can shoehorn a custom loop in there.. wish me luck.
Like • Show 0 Likes0
Actions
- colesdav @ scythefwd on Sep 25, 2018 8:51 PM
  Is this the cooler you have been running?
  http://www.enermax.com/home.php?fn=eng/product_a1_1_1&lv0=109&lv1=118&no=369
  Strange that it should start to fail so early, are you able to RMA it?
  Maybe you could get a a larger radiator on the AIO system or add a reservoir to it?
  This company has some interesting solutions for that I think ... https://www.alphacool.com/products/
  Good luck.
  Like • Show 0 Likes0
  Actions
  - scythefwd @ colesdav on Sep 26, 2018 6:17 AM
    colesdav - yes, that is the exact cooler. Yes, they're going to rma it. No, I'm not going to use the one they send back (though I may keep it as a spare /backup to my new cooler). That or I'll sell it off for a few bucks and use it to offset the cost of my new cooler.
    
    The cooler itself has a 2 year warranty, so I'm definitely within that period.
    I cannot add a reservoir or extra rad to it without voiding the warranty.. I couldn't disassemble it without it being obvious it was tampered with (looks like it uses barbed fittings and shrink wrap to keep it in place.. )
    
    I ended up ordering these for my system:
    watercool heatkiller iv pro nickle plated copper waterblock (ties for the best performing waterblock in every roundup I've seen, usually besting the EK by several degrees)
    xspc photon x4 170 reservoir and pump (which will get upgraded to a d5 pump when it fails, the d5 is the better pump, but I'm still trying to be cheap... it's a hard habit to kick)
    xspc compression fittings for flex
    xspc 240mm radiator
    2 meters worth of flex so I should have enough to do whatever routing I want.
    
    That should be a much better flowing system vs. the enermax, but at 3x the cost too.. I'm not sure what fans I intend to run ... IF I get a wild hair, I may add a 140 mm radiator in there as well and just daisy chain them.
    1 person found this helpful
    Like • Show 1 Like1
    Actions
scythefwd Sep 27, 2018 10:23 PM
Shes together, no leaks, running very cool. 4ghz, 3.125v. idles at 24 C. Blended prime 95 maxes temps at 65 degrees C.
I can live with that.
Like • Show 0 Likes0
Actions
scythefwd Sep 27, 2018 11:05 PM
Leak tested and running cool.. pretty lights.woo.. matching front lights on my ek rgb fans, controlled by mobo headers.
Like • Show 0 Likes0
Actions
scythefwd Sep 28, 2018 11:58 PM
she's stable at 4.0ghz and 1.238V 65C in aida65 stress test. Ram at 3266. Bios version at 3.20
Like • Show 0 Likes0
Actions
scythefwd Sep 29, 2018 12:01 AM
bios is at 3.30, not 3.20
Like • Show 0 Likes0
Actions

Home lab, aka the 32 headed beast

Attachments

Outcomes

Attachments

Related Content

Recommended Content