New Server Project

[TopHatProductio]

Hello! It's been a while since I last posted here with my own topic. I now have a dedicated ESXi server in the works. The server project is meant to replace (and exceed) my previous workstation - a Dell Precision T7500. Here are the specs for the hardware:

HPE ProLiant DL580 G7

    OS   :: VMware ESXi 6.5u3 Enterprise Plus
    CPU  :: 4x Intel Xeon E7-8870's (10c/20t each; 40c/80t total)
    RAM  :: 256GB (64x4GB) PC3-10600R DDR3-1333 ECC
    PCIe :: 1x HP 512843-001/591196-001 System I/O board + 
                1x HP 588137-B21; 591205-001/591204-001 PCIe Riser board
    GPU  :: 1x nVIDIA GeForce GTX Titan Xp +
                1x AMD FirePro S9300 x2 (2x "AMD Radeon Fury X's")
    SFX  :: 1x Creative Sound Blaster Audigy Rx
    NIC  :: 1x HPE NC524SFP (489892-B21) +
                2x Silicom PE310G4SPI9L-XR-CX3's
    STR  :: 1x HP Smart Array P410i Controller (integrated) +
                1x HGST HUSMM8040ASS200 MLC 400GB SSD (ESXi, vCenter Appliance, ISOs) + 
                4x HP 507127-B21 300GB HDDs (ESXi guest datastores) +
                1x Western Digital WD Blue 3D NAND 500GB SSD + 
                1x Intel 320 Series SSDSA2CW600G3 600GB SSD +
                1x Seagate Video ST500VT003 500GB HDD
    STR  :: 1x LSI SAS 9201-16e HBA SAS card +
                1x Mini-SAS SFF-8088 cable + 
                        1x Dell EMC KTN-STL3 (15x 3.5in HDD enclosure) + 
                                4x HITACHI Ultrastar HUH728080AL4205 8TB HDDs +
                                4x IBM Storewise XIV v7000 98Y3241 4TB HDDs
    I/O  :: 1x Inateck KU8212 (USB 3.2) +
                1x Logitech K845 (Cherry MX Blue) +
                1x Dell MS819 Wired Mouse
            1x Sonnet Allegro USB3-PRO-4P10-E (USB 3.X) +
                1x LG WH16NS40 BD-RE ODD
    PRP  :: 1x Samsung ViewFinity S70A UHD 32" (S32A700)
            1x Sony Optiarc BluRay drive
    PSU  :: 4x HP 1200W PSUs (441830-001/438203-001)

The details for the ProLiant DL380 Gen9 will appear here once data migration is complete. VMware Horizon (VDI) will have to wait for a future phase (if implemented at all). The current state of self-hosted VDI is Windows-centric, with second-hand support for Linux and no proper support for macOS.

The planned software/VM configurations have been moved back to the LTT post, and will be changing often for the foreseeable future.

Product links and details can be found here.

ESXi itself is usually run from a USB thumb drive, but I have a drive dedicated to it. No harm done. A small amount of thin provisioning/overbooking (RAM only) won’t hurt. macOS and Linux would have gotten a Radeon/FirePro (ie., Rx Vega 64), for best compatibility and stability, but market forces originally prevented this. Windows 10 gets the Audigy Rx and a Titan Xp. The macOS and Linux VMs get whatever audio the Titan Z FirePro S9300 x2 can provide. The whole purpose of Nextcloud is to phase out the use of Google Drive/Photos, iCloud, Box.com, and other externally-hosted cloud services (Mega can stay though).

There are three other mirrors for this project, in case you're interested in following individual conversations from the other sites (in addition to this thread).

• Level1Techs
• Linus Tech Tips
• TechPowerUp

P.S. Out of all the sites that I've ever used, this forum has one of the best WYSIWYG editors I've used in a while

Kudos to the devs!

[TopHatProductio]

Folding@Home is off the table for the foreseeable future:

https://linustechtips.com/status/316035/

[TopHatProductio]

I've been forced to hold off on the OpenStreetMaps backend (routing) container. This was due to the insane memory usage, which appears to have been what crashed my once-stable Linux VM. I'd need to move beyond 32GB RAM for that one VM, which would be pretty crazy. The ESXi host only has 256GB RAM, with all slots filled (4GB sticks). To move beyond that would cost me a fortune, buying 8GB and 16GB sticks off the used market. The current market does not lend itself to that errand too easily. I'll focus on just YaCy Grid for the time being.

[TopHatProductio]

The PCIe enclosure is being removed from the project. Unable to get it working, and OEM/ODM won't communicate to assist with troubleshooting. No way to justify keeping it in the rack at this point.

[TopHatProductio]

Current ToDo's:

**Current ToDo's:**
 - Windows Server 2016:
        - UNIX/POSIX attributes in AD
                - <https://github.com/wruppelx/win2016setuid>
 - Artix OpenRC:
        - Docker container: YaCy Grid
                - <https://blog.fossasia.org/creating-a-dockerfile-for-yacy-grid-mcp/>
                - <https://github.com/yacy/yacy_grid_mcp/blob/master/docker/all-in-one/docker-compose.yml>
                - initiate web crawl
 - Windows 10 Enterprise:
        - Gaming VM troubleshooting (<https://www.reddit.com/r/VFIO/>)

**Upcoming ToDo's:**
 - Server/Networking:
        - migrate from ESXi 6.5 to 6.7 **

**Long-term ToDo's:**
 - Server/Networking:
        - clone HDDs to SAS SSDs
                - Acronis True Image
        - VDI host when?
                - pushed to 2023, due to performance requirements
        - DL580 Gen8/9 planning...

[TopHatProductio]

I've finally managed to setup wireless Time Machine backups for the MacBook. Next will be the EliteBook, if I can figure out how to do so. In addition to the other tasks I have in front of me.

WARNING: Apple only supports virtualisation of their OS, on their own hardware. I will not discuss unsupported configurations on this website.

• https://512pixels.net/2018/08/how-to-set-up-time-machine-server/

The other way of doing this task would be to get the macOS Server app (~20 USD on the App Store) and setup the role from the app.

[TopHatProductio]

Current ToDo's:
 - Windows Server 2016:
        - UNIX/POSIX attributes in AD
                - https://github.com/wruppelx/win2016setuid
 - Windows 10 Enterprise:
        - Gaming VM troubleshooting (https://www.reddit.com/r/VFIO/)

Upcoming ToDo's:
 - Server/Networking:
        - purchase/activate EaseUS ToDo Backup Center
        - purchase/activate OnlyOffice server license
        - migrate from ESXi 6.5 to 6.7 **

Long-term ToDo's:
 - Server/Networking:
        - clone HDDs to SAS SSDs
                - Acronis True Image
        - get a GitHub point-of-contact
        - VDI host when?
                - pushed to 2023, due to performance requirements
        - DL580 Gen8/9 planning...
 - Artix OpenRC:
        - Docker container: YaCy Grid
                - https://blog.fossasia.org/creating-a-dockerfile-for-yacy-grid-mcp/
                - https://github.com/yacy/yacy_grid_mcp/blob/master/docker/all-in-one/docker-compose.yml
                - initiate web crawl

[TopHatProductio]

Purchasing EaseUS ToDo Backup Center next week, so I can actually backup all Windows machines on my domain. That's a priority for this project. The other licenses can wait. I'm already incurring risk without adequate backups, as-is...

[TopHatProductio]

Best part about snow days and days below 40 F? I get free AC for my server. Not that most people would suggest it, but I'm not allowed to run AC in that room when the server's on, sadly. So, I take what I can get. May be better than overheating the poor thing and frying a component...

[TopHatProductio]

New List:

• https://linustechtips.com/status/317796/

[TopHatProductio]

In my quest to deck out Windows Server with every customisation I can throw at it, I’m going GPO crazy!

Windows Defender GPO is next…

[TopHatProductio]

Aaand ran into an error. Can’t win:

• https://communities.vmware.com/t5/ESXi-Discussions/Unable-to-List-Datastores-with-ESXCLI/m-p/2900149

[TopHatProductio]

Good news, it looks like the issue may have been resolved!

[TopHatProductio]

Current ToDo's:
 - Windows 10 Enterprise:
        - VM gaming troubleshooting
            - <https://www.reddit.com/r/VFIO/>

Upcoming ToDo's:
 - Artix OpenRC:
    - Docker container: Nextcloud
        - add redis caching??
 - Server/Networking:
    - purchase OnlyOffice server license
 - macOS Mojave:
    - Get a MacPorts point-of-contact
    - Homebrew => MacPorts
    - upgrade to Big Sur 11.2.3

Long-term ToDo's:
 - Server/Networking:
    - migrate from ESXi 6.5 to 6.7
    - clone HDDs to SAS SSDs
    - VDI host when?
        - pushed to 2024/2025, due to performance requirements
        - DL580 Gen8/9 planning?!
    - Get a GitHub point-of-contact
 - Artix OpenRC:
    - Docker container: YaCy Grid
        - <https://blog.fossasia.org/creating-a-dockerfile-for-yacy-grid-mcp/>
        - <https://github.com/yacy/yacy_grid_mcp/blob/master/docker/all-in-one/docker-compose.yml>
        - initiate web crawl

As usual, macOS topics are not permitted here. Please do not ask about them here, it's against the rules (for first-time readers/viewers).

[TopHatProductio]

Updated Plans:

Current ToDo's:
 - macOS Mojave:
    - upgrade to Big Sur 11.2.3
    - update MacPorts and Xcode
 - Artix OpenRC:
    - Docker container: Nextcloud
        - purchase OnlyOffice server license
        - add redis caching??
 - Windows 10 Enterprise:
    - purchase Adobe Acrobat Pro 2022
    - purchase DaVinci Resolve license

Upcoming ToDo's:
 - Server/Networking:
    - migrate from ESXi 6.5 to 6.7
    - AMD GPU shopping (Linux/macOS)
        - https://www.reddit.com/r/realAMD/comments/tt2hq4/in_search_of_a_gpu/

Long-term ToDo's:
 - Server/Networking:
    - clone HDDs to SAS SSDs
    - VDI host when (DL580 Gen8/9 planning)?
 - Artix OpenRC:
    - Docker container: YaCy Grid
        - https://blog.fossasia.org/creating-a-dockerfile-for-yacy-grid-mcp/
        - https://github.com/yacy/yacy_grid_mcp/blob/master/docker/all-in-one/docker-compose.yml
        - initiate web crawl

Some tasks have been completed. Others have been delayed, due to priority. As usual, macOS items will not be discussed here. 

[TopHatProductio]

The Titan Z is getting long-in-tooth and no longer receives driver updates. New day, new problems to solve:

https://www.reddit.com/r/realAMD/comments/tt2hq4/in_search_of_a_gpu/

https://www.techpowerup.com/forums/threads/amd-radeon-pro-v320-v340.293530/

[TopHatProductio]

Just moved to OnlyOffice Document Server EE (Enterprise Edition). Redis cache is next. If the Radeon Pro v320 can't be split between multiple VMs, macOS gets dibs. I can't see myself messing with GPU stuff in Linux again. Getting that to work with nVIDIA drivers was a bit of a pain. By the time I ever try again with another multi-die GPU, it'll hopefully be RDNA-based and on a server running something newer than Intel Westmere. The show must go on, and one VM can't absorb all of my time when other tasks await...

[TopHatProductio]

Future planning begins now:

https://linustechtips.com/status/318598/

[TopHatProductio]

Added redis container to docker, testing with nextcloud. The list is changing, slowly:

Current ToDo's:
 - macOS Mojave:
    - upgrade to Big Sur 11.2.3
    - update MacPorts and Xcode
 - Windows 10 Enterprise:
    - purchase Adobe Acrobat Pro 2022
    - purchase DaVinci Resolve license

Upcoming ToDo's:
 - Server/Networking:
    - migrate from ESXi 6.5 to 6.7

Long-term ToDo's:
 - Server/Networking:
    - convert the VMs (MBR => GPT, BIOS => UEFI)
    - clone HDDs to SAS SSDs
    - VDI host when (DL580 Gen8/9 planning)?
 - Artix OpenRC:
    - Docker container: YaCy Grid
        - <https://blog.fossasia.org/creating-a-dockerfile-for-yacy-grid-mcp/>
        - <https://github.com/yacy/yacy_grid_mcp/blob/master/docker/all-in-one/docker-compose.yml>
        - initiate web crawl

[TopHatProductio]

Moved a question to a new location:

• https://forum.level1techs.com/t/amd-radeon-pro-v320-v340-questions/183773

[TopHatProductio]

Okay, things have definitely taken a different turn than expected. As I learned in the 2021/2022 transition, certain cards that require Above 4G MMIO (Above 4G decoding) are off-limits for me as long as I'm using the DL580 G7. That was part of why I couldn't use the Tesla K80's in late 2021. Well, that same situation happened with the Radeon Pro v320, and I have no idea if the same would apply if I had gotten the v340 instead. I got the help of a friend to look for firmware updates, to see if the feature had possibly been introduced in later firmware versions. No such luck. Removing PCIe SSD's also didn't help. Sad part is, it looks as though the DL980 G7 didn't have this limitation. As a result, I will have to push updating the macOS VM to Big Sur further out - until I can get a DL580 Gen8/9 in-house.

I also had difficulty updating software for the secondary DNS (software by Technitium), so I'm going to have to contact them to figure out how to proceed. The handy installer stopped working for me a while back, so I've had to perform all updates by hand since mid-2021 iirc.

Hopefully, it won't be too long before I can get back on track with finishing the rest of the server project, because I miss actually getting tasks done...

Current ToDo's:
 - Server/Networking:
    - convert the VMs (MBR => GPT, BIOS => UEFI)

Upcoming ToDo's:
 - Server/Networking:
    - clone SAS HDDs to SAS SSDs
    - migrate from ESXi 6.5 to 6.7

Long-term ToDo's:
 - Server/Networking:
    - VDI host when (DL580 Gen8/9 planning)?
    - replace Titan Z with Radeon Pro v320/v340
 - macOS Mojave:
    - upgrade to Big Sur 11.2.3
    - update MacPorts and Xcode

Unconfirmed ToDo's:
 - Artix OpenRC:
    - Docker container: YaCy Grid
        - <https://blog.fossasia.org/creating-a-dockerfile-for-yacy-grid-mcp/>
        - <https://github.com/yacy/yacy_grid_mcp/blob/master/docker/all-in-one/docker-compose.yml>
        - initiate web crawl

oof...