New Server Project

[TopHatProductio]

Hello! It's been a while since I last posted here with my own topic. I now have a dedicated ESXi server in the works. The server project is meant to replace (and exceed) my previous workstation - a Dell Precision T7500. Here are the specs for the hardware:

HPE ProLiant DL580 G7

    OS   :: VMware ESXi 6.5u3 Enterprise Plus
    CPU  :: 4x Intel Xeon E7-8870's (10c/20t each; 40c/80t total)
    RAM  :: 256GB (64x4GB) PC3-10600R DDR3-1333 ECC
    PCIe :: 1x HP 512843-001/591196-001 System I/O board + 
                1x HP 588137-B21; 591205-001/591204-001 PCIe Riser board
    GPU  :: 1x nVIDIA GeForce GTX Titan Xp +
                1x AMD FirePro S9300 x2 (2x "AMD Radeon Fury X's")
    SFX  :: 1x Creative Sound Blaster Audigy Rx
    NIC  :: 1x HPE NC524SFP (489892-B21) +
                2x Silicom PE310G4SPI9L-XR-CX3's
    STR  :: 1x HP Smart Array P410i Controller (integrated) +
                1x HGST HUSMM8040ASS200 MLC 400GB SSD (ESXi, vCenter Appliance, ISOs) + 
                4x HP 507127-B21 300GB HDDs (ESXi guest datastores) +
                1x Western Digital WD Blue 3D NAND 500GB SSD + 
                1x Intel 320 Series SSDSA2CW600G3 600GB SSD +
                1x Seagate Video ST500VT003 500GB HDD
    STR  :: 1x LSI SAS 9201-16e HBA SAS card +
                1x Mini-SAS SFF-8088 cable + 
                        1x Dell EMC KTN-STL3 (15x 3.5in HDD enclosure) + 
                                4x HITACHI Ultrastar HUH728080AL4205 8TB HDDs +
                                4x IBM Storewise XIV v7000 98Y3241 4TB HDDs
    I/O  :: 1x Inateck KU8212 (USB 3.2) +
                1x Logitech K845 (Cherry MX Blue) +
                1x Dell MS819 Wired Mouse
            1x Sonnet Allegro USB3-PRO-4P10-E (USB 3.X) +
                1x LG WH16NS40 BD-RE ODD
    PRP  :: 1x Samsung ViewFinity S70A UHD 32" (S32A700)
            1x Sony Optiarc BluRay drive
    PSU  :: 4x HP 1200W PSUs (441830-001/438203-001)

The details for the ProLiant DL380 Gen9 will appear here once data migration is complete. VMware Horizon (VDI) will have to wait for a future phase (if implemented at all). The current state of self-hosted VDI is Windows-centric, with second-hand support for Linux and no proper support for macOS.

The planned software/VM configurations have been moved back to the LTT post, and will be changing often for the foreseeable future.

Product links and details can be found here.

ESXi itself is usually run from a USB thumb drive, but I have a drive dedicated to it. No harm done. A small amount of thin provisioning/overbooking (RAM only) won’t hurt. macOS and Linux would have gotten a Radeon/FirePro (ie., Rx Vega 64), for best compatibility and stability, but market forces originally prevented this. Windows 10 gets the Audigy Rx and a Titan Xp. The macOS and Linux VMs get whatever audio the Titan Z FirePro S9300 x2 can provide. The whole purpose of Nextcloud is to phase out the use of Google Drive/Photos, iCloud, Box.com, and other externally-hosted cloud services (Mega can stay though).

There are three other mirrors for this project, in case you're interested in following individual conversations from the other sites (in addition to this thread).

• Level1Techs
• Linus Tech Tips
• TechPowerUp

P.S. Out of all the sites that I've ever used, this forum has one of the best WYSIWYG editors I've used in a while

Kudos to the devs!

[TopHatProductio]

Resolved the Nextcloud issue, the other issues still remain. Focusing on macOS VM and KMS for now...

[TopHatProductio]

Time for yet another troubleshooting thread:

https://communities.vmware.com/t5/ESXi-Discussions/ESXi-6-7-with-nVIDIA-Titan-V/m-p/2885992#M279617

[TopHatProductio]

More things I'm doing on the side, to streamline domain UX:

https://www.youtube.com/watch?v=sJ_E7I4CHw0

https://www.youtube.com/watch?v=YreKZqdu4fo

A lot of the benefits won't work until Windows 10 is activated, sadly. So, it's all just prep work.

• https://docs.microsoft.com/en-us/answers/questions/680930/moving-from-microsoft-accounts-to-domain-a...

[TopHatProductio]

Considering virtual audio device for Artix Linux, since the GPU's audio device didn't work well with PCI Passthrough. Still need to setup xBrowserSync and NGINX Revers Proxy Manager. YouPHPTube will undergo a final validation/testing phase after this. If the MariaDB issue can't be resolved, I'll just rely on LBRY and other existing alternatives and shift focus to YaCy Grid as the final Docker container instance. Still need to setup Windows 10 Enterprise VM (Titan Xp) for daily use. Also still troubleshooting the Titan V issue...

• https://linustechtips.com/status/313723/

[TopHatProductio]

I've decided to merge the Win10 and Remote Dev VMs. I've worked in this environment before, and it hasn't been an issue for me in the past. Saves time and resources in my case. Also, may have to kick the Threadripper from the VM server project if it can't be resolved by month's end. I can just toss Windows 10 on that thing if necessary, and the Titan V should work...

[TopHatProductio]

Getting ready to test Windows 10 Enterprise VM with this enabled:

• https://kb.vmware.com/s/article/1033435

[TopHatProductio]

Finished initial setup for xBrowserSync last night. Now looking to attack the last container - YaCy...

*YouPHPTube is being delayed until the rest of the server project is finished, to avoid unnecessary delays.

[TopHatProductio]

Welp, here are the major changes for the project thus far:

• YouPHPTube is probably getting jettisoned, in favour of using LBRY instead
• YaCy Grid is going to be a long-term experiment, and has to be built from source
• xBrowserSync is officially part of the project now - and it's got more on the way!
• PleX has Movie and Music streaming, but doesn't have external LDAP integration
  • added Trakt.tv plugin (scrobbler) for additional functionality
• Would like to add eBooks management to Nextcloud, but may need a new container instead
• Azure Active Directory was added, to enable possible MFA in the future
• The majority of the server project is complete - looking into 24/7 testing soon...

[TopHatProductio]

Why couldn't drive cloning be easy 😥

• https://www.reddit.com/r/HomeDataCenter/comments/s6idjv/error_after_cloning_to_sas_ssds/

[TopHatProductio]

The task list has been updated:

• https://linustechtips.com/status/314520/

[TopHatProductio]

Okay, you guys may get a laugh from this. I was remoted into the Windows 10 Enterprise VM (equipped with a Titan Xp). I installed 3DMark, thinking I was gonna do some benchmark runs today. Instead, the entire ESXi host rebooted. All of the VMs went offline, and I'm waiting to see if the server comes back from this in one piece. Hoping this also doesn't rule out just gaming in general. If the power draw from the Titan Xp is too much, I may have to consider other options...

[TopHatProductio]

Hmm...

https://forums.servethehome.com/index.php?threads/questions-about-sandisk-fusion-ioscale-3-2tb-pcie-...

[TopHatProductio]

The job is never finished:

• https://linustechtips.com/status/314805/

I march onward...

[TopHatProductio]

And here I was, thinking I was close to being done 😐 

• https://linustechtips.com/status/314861/

I need sleep.

[TopHatProductio]

What I've managed to get done thus far:

• install a 2nd PCIe SSD
• install a USB adapter card
• plug in a 2nd PSU
• buy 128GB of RAM
• decommission Threadripper

Off to a rough week. Still looking for a good PCIe enclosure...

[TopHatProductio]

First benchmarks!

https://linustechtips.com/status/315002/

[TopHatProductio]

Just purchased a Magma EB7-X8G2-RAS-F (7-slot PCIe 2.0 expansion enclosure), with what appears to be a two x8 ports on its expansion interface. I will most likely need to acquire the following next:

• x16 interface card
• x16 host adapter
• x16 PCIe cable

This is going to be a long one...

https://vimeo.com/46778368

[TopHatProductio]

The PCIe x16 Host adapter and PCIe x16 cable arrived today! Pictures in a few...

EDIT: There's 128GB more RAM on the way as well. And guess what's going in the enclosure?...

[TopHatProductio]

It's finally starting to warm up where I'm at. But that also means I can't run F@H anymore, due to thermal reasons😞Until next Autumn, the folding will be paused.

[TopHatProductio]

The spec sheet for the server has changed, in anticipation of the PCIe enclosure that arrived recently. Still waiting for a few more components to arrive in the mail, but this is the way for ward from here. Some parts have been moved from the DL580 G7, to the enclosure, to free up space and reduce power draw. Certain parts that don't receive regular use don't need to be in the DL580 G7 necessarily. Optional parts will be in the enclosure instead. The server has received a RAM upgrade as well, from 128GB to 256GB. The SAS HDD-to-SSD cloning operation will occur after the migration from ESXi 6.5 to 6.7. If you have any questions, feel free to ask!