TopHatProductio
Hot Shot
Hot Shot

New Server Project

Hello! It's been a while since I last posted here with my own topic. I now have a dedicated ESXi server in the works, and am planning to start using it 24/7 by the end of this year or early next year (2021). Here are the specs for the hardware:

 

HPE ProLiant DL580 G7

 

CPU  :: 4x Intel Xeon E7-8870's (10c/20t each; 40c/80t total)
RAM  :: 256GB (64x4GB) DDR3-1333 PC3-10600R ECC 
PCIe :: 1x HP 512843-001/591196-001 System I/O board + 
                1x HP 588137-B21; 591205-001/591204-001 PCIe Riser board +
GPU  :: 1x nVIDIA GeForce GTX Titan Xp +
                1x nVIDIA GeForce GTX Titan Z
SFX  :: 1x Creative Sound Blaster Audigy Rx
NIC  :: 1x HPE NC524SFP (489892-B21)
I/O  :: 1x HPE PCIe ioDuo MLC 1.28TB I/O Accelerator (641255-001) +
                1x SanDisk Fusion ioScale MLC 3.2TB Accelerator (F11-002-3T20-CS-0001)
STR  :: 1x HP Smart Array P410i Controller (integrated) +
                1x HGST HUSMM8040ASS200 MLC 400GB SSD (ESXi, vCenter Appliance, ISOs) + 
                4x HP 507127-B21 300GB HDDs (ESXi guest datastores) +
                1x Western Digital WD Blue 3D NAND 500GB SSD (Virtual Flash) + 
                1x Intel 320 Series SSDSA2CW600G3 600GB SSD (VFF) +
                1x Seagate Video ST500VT003 500GB HDD 
        1x LSI SAS 9201-16e HBA SAS card +
            1x Mini-SAS SFF-8088 cable + 
                1x Dell EMC KTN-STL3 (15x 3.5in HDD enclosure) + 
                    4x HITACHI Ultrastar HUH728080AL4205 8TB HDDs +
                    4x IBM Storewise XIV v7000 98Y3241 4TB HDDs
        1x Sony Optiarc BluRay drive
PRP  :: 1x AOC U2879VF (4K)
                1x Inateck KU5211 (USB 3.2) +
                    1x Logitech K845 (Cherry MX Blue) +
                    1x Dell MS819 Wired Mouse
PSU  :: 4x HP 1200W PSU's (441830-001/438203-001)

 

 

Product links and details can be found here.

 

Here is the current software configuration plan for the server:

 

*  Temporary task that will be replaced by a permanent, self-hosted solution

** Can benefit from port forwarding, but will be primarily tunnel-bound

^  Tunnel-bound (VPN/SSH) role - not port forwarded/exposed to the Internet

+ Active Directory enabled - Single Sign On (SSO)

 

Here is the current resource allocation plan for the server:

 

VMware VCSA (vCenter) :: 24/7 - true , dedicatedHDD - false, dedicatedGPU - false,   2c/4t  + 12GB
Windows Server 2016   :: 24/7 - true , dedicatedHDD - true , dedicatedGPU - false,   8c/16t + 16GB
Artix Linux - Xfce    :: 24/7 - true , dedicatedHDD - true , dedicatedGPU - false,   8c/16t + 16GB
macOS Server 10.14.X  :: 24/7 - true , dedicatedHDD - true , dedicatedGPU - true ,   8c/16t + 16GB        (NOT TO BE DISCUSSED HERE)
Windows 10 Enterprise :: 24/7 - false, dedicatedHDD - true , dedicatedGPU - true ,  12c/24t + 32GB *
Temporary/Testing VM  :: 24/7 - false, dedicatedHDD - N/a  , dedicatedGPU - N/a  ,  --c/--t + --GB *

 

 

Product links and details can be found here.

 

VMs marked with an * cannot be run at the same time. Only one of them can ever run at any given moment. MacOS and Linux would have gotten a Radeon/FirePro (ie., Rx Vega 64), for best compatibility and stability, but market forces have prevented this. Windows 10 gets the Creative Audigy Rx. The MacOS and Linux VMs get whatever audio the Titan Z provides (either that or a software solution). Windows 10, Remote Development, and the Temp/Testing VM will be put to sleep (or offed) until they are needed (Wake on LAN), since they don't host any essential services.

 

There are three other mirrors for this project, in case you're interested in following individual conversations from the other sites (in addition to this thread).

 

P.S. Out of all the sites that I've ever used, this forum has one of the best WYSIWYG editors I've used in a while Smiley Happy

Kudos to the devs!

Tags (1)
189 Replies
TopHatProductio
Hot Shot
Hot Shot

Folding@Home is off the table for the foreseeable future:

https://linustechtips.com/status/316035/

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

I've been forced to hold off on the OpenStreetMaps backend (routing) container. This was due to the insane memory usage, which appears to have been what crashed my once-stable Linux VM. I'd need to move beyond 32GB RAM for that one VM, which would be pretty crazy. The ESXi host only has 256GB RAM, with all slots filled (4GB sticks). To move beyond that would cost me a fortune, buying 8GB and 16GB sticks off the used market. The current market does not lend itself to that errand too easily. I'll focus on just YaCy Grid for the time being.

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

The PCIe enclosure is being removed from the project. Unable to get it working, and OEM/ODM won't communicate to assist with troubleshooting. No way to justify keeping it in the rack at this point.

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Current ToDo's:

 

**Current ToDo's:**
 - Windows Server 2016:
        - UNIX/POSIX attributes in AD
                - <https://github.com/wruppelx/win2016setuid>
 - Artix OpenRC:
        - Docker container: YaCy Grid
                - <https://blog.fossasia.org/creating-a-dockerfile-for-yacy-grid-mcp/>
                - <https://github.com/yacy/yacy_grid_mcp/blob/master/docker/all-in-one/docker-compose.yml>
                - initiate web crawl
 - Windows 10 Enterprise:
        - Gaming VM troubleshooting (<https://www.reddit.com/r/VFIO/>)

**Upcoming ToDo's:**
 - Server/Networking:
        - migrate from ESXi 6.5 to 6.7 **

**Long-term ToDo's:**
 - Server/Networking:
        - clone HDDs to SAS SSDs
                - Acronis True Image
        - VDI host when?
                - pushed to 2023, due to performance requirements
        - DL580 Gen8/9 planning...
0 Kudos
TopHatProductio
Hot Shot
Hot Shot

I've finally managed to setup wireless Time Machine backups for the MacBook. Next will be the EliteBook, if I can figure out how to do so. In addition to the other tasks I have in front of me.

WARNING: Apple only supports virtualisation of their OS, on their own hardware. I will not discuss unsupported configurations on this website.

The other way of doing this task would be to get the macOS Server app (~20 USD on the App Store) and setup the role from the app.

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Current ToDo's:
 - Windows Server 2016:
        - UNIX/POSIX attributes in AD
                - https://github.com/wruppelx/win2016setuid
 - Windows 10 Enterprise:
        - Gaming VM troubleshooting (https://www.reddit.com/r/VFIO/)

Upcoming ToDo's:
 - Server/Networking:
        - purchase/activate EaseUS ToDo Backup Center
        - purchase/activate OnlyOffice server license
        - migrate from ESXi 6.5 to 6.7 **

Long-term ToDo's:
 - Server/Networking:
        - clone HDDs to SAS SSDs
                - Acronis True Image
        - get a GitHub point-of-contact
        - VDI host when?
                - pushed to 2023, due to performance requirements
        - DL580 Gen8/9 planning...
 - Artix OpenRC:
        - Docker container: YaCy Grid
                - https://blog.fossasia.org/creating-a-dockerfile-for-yacy-grid-mcp/
                - https://github.com/yacy/yacy_grid_mcp/blob/master/docker/all-in-one/docker-compose.yml
                - initiate web crawl
0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Purchasing EaseUS ToDo Backup Center next week, so I can actually backup all Windows machines on my domain. That's a priority for this project. The other licenses can wait. I'm already incurring risk without adequate backups, as-is...

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Best part about snow days and days below 40 F? I get free AC for my server. Not that most people would suggest it, but I'm not allowed to run AC in that room when the server's on, sadly. So, I take what I can get. May be better than overheating the poor thing and frying a component...

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

In my quest to deck out Windows Server with every customisation I can throw at it, I’m going GPO crazy!

 

Screenshot_494

Screenshot_509

 

Windows Defender GPO is next…

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Good news, it looks like the issue may have been resolved!

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

 

Current ToDo's:
 - Windows 10 Enterprise:
        - VM gaming troubleshooting
            - <https://www.reddit.com/r/VFIO/>

Upcoming ToDo's:
 - Artix OpenRC:
    - Docker container: Nextcloud
        - add redis caching??
 - Server/Networking:
    - purchase OnlyOffice server license
 - macOS Mojave:
    - Get a MacPorts point-of-contact
    - Homebrew => MacPorts
    - upgrade to Big Sur 11.2.3

Long-term ToDo's:
 - Server/Networking:
    - migrate from ESXi 6.5 to 6.7
    - clone HDDs to SAS SSDs
    - VDI host when?
        - pushed to 2024/2025, due to performance requirements
        - DL580 Gen8/9 planning?!
    - Get a GitHub point-of-contact
 - Artix OpenRC:
    - Docker container: YaCy Grid
        - <https://blog.fossasia.org/creating-a-dockerfile-for-yacy-grid-mcp/>
        - <https://github.com/yacy/yacy_grid_mcp/blob/master/docker/all-in-one/docker-compose.yml>
        - initiate web crawl

 

As usual, macOS topics are not permitted here. Please do not ask about them here, it's against the rules (for first-time readers/viewers).

 

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Updated Plans:

Current ToDo's:
 - macOS Mojave:
    - upgrade to Big Sur 11.2.3
    - update MacPorts and Xcode
 - Artix OpenRC:
    - Docker container: Nextcloud
        - purchase OnlyOffice server license
        - add redis caching??
 - Windows 10 Enterprise:
    - purchase Adobe Acrobat Pro 2022
    - purchase DaVinci Resolve license

Upcoming ToDo's:
 - Server/Networking:
    - migrate from ESXi 6.5 to 6.7
    - AMD GPU shopping (Linux/macOS)
        - https://www.reddit.com/r/realAMD/comments/tt2hq4/in_search_of_a_gpu/

Long-term ToDo's:
 - Server/Networking:
    - clone HDDs to SAS SSDs
    - VDI host when (DL580 Gen8/9 planning)?
 - Artix OpenRC:
    - Docker container: YaCy Grid
        - https://blog.fossasia.org/creating-a-dockerfile-for-yacy-grid-mcp/
        - https://github.com/yacy/yacy_grid_mcp/blob/master/docker/all-in-one/docker-compose.yml
        - initiate web crawl

 

Some tasks have been completed. Others have been delayed, due to priority. As usual, macOS items will not be discussed here

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

The Titan Z is getting long-in-tooth and no longer receives driver updates. New day, new problems to solve:

https://www.reddit.com/r/realAMD/comments/tt2hq4/in_search_of_a_gpu/

https://www.techpowerup.com/forums/threads/amd-radeon-pro-v320-v340.293530/

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Just moved to OnlyOffice Document Server EE (Enterprise Edition). Redis cache is next. If the Radeon Pro v320 can't be split between multiple VMs, macOS gets dibs. I can't see myself messing with GPU stuff in Linux again. Getting that to work with nVIDIA drivers was a bit of a pain. By the time I ever try again with another multi-die GPU, it'll hopefully be RDNA-based and on a server running something newer than Intel Westmere. The show must go on, and one VM can't absorb all of my time when other tasks await...

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Future planning begins now:

https://linustechtips.com/status/318598/

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Added redis container to docker, testing with nextcloud. The list is changing, slowly:

Current ToDo's:
 - macOS Mojave:
    - upgrade to Big Sur 11.2.3
    - update MacPorts and Xcode
 - Windows 10 Enterprise:
    - purchase Adobe Acrobat Pro 2022
    - purchase DaVinci Resolve license

Upcoming ToDo's:
 - Server/Networking:
    - migrate from ESXi 6.5 to 6.7

Long-term ToDo's:
 - Server/Networking:
    - convert the VMs (MBR => GPT, BIOS => UEFI)
    - clone HDDs to SAS SSDs
    - VDI host when (DL580 Gen8/9 planning)?
 - Artix OpenRC:
    - Docker container: YaCy Grid
        - <https://blog.fossasia.org/creating-a-dockerfile-for-yacy-grid-mcp/>
        - <https://github.com/yacy/yacy_grid_mcp/blob/master/docker/all-in-one/docker-compose.yml>
        - initiate web crawl
0 Kudos
TopHatProductio
Hot Shot
Hot Shot

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Okay, things have definitely taken a different turn than expected. As I learned in the 2021/2022 transition, certain cards that require Above 4G MMIO (Above 4G decoding) are off-limits for me as long as I'm using the DL580 G7. That was part of why I couldn't use the Tesla K80's in late 2021. Well, that same situation happened with the Radeon Pro v320, and I have no idea if the same would apply if I had gotten the v340 instead. I got the help of a friend to look for firmware updates, to see if the feature had possibly been introduced in later firmware versions. No such luck. Removing PCIe SSD's also didn't help. Sad part is, it looks as though the DL980 G7 didn't have this limitation. As a result, I will have to push updating the macOS VM to Big Sur further out - until I can get a DL580 Gen8/9 in-house.

I also had difficulty updating software for the secondary DNS (software by Technitium), so I'm going to have to contact them to figure out how to proceed. The handy installer stopped working for me a while back, so I've had to perform all updates by hand since mid-2021 iirc.

Hopefully, it won't be too long before I can get back on track with finishing the rest of the server project, because I miss actually getting tasks done...

 

 

Current ToDo's:
 - Server/Networking:
    - convert the VMs (MBR => GPT, BIOS => UEFI)

Upcoming ToDo's:
 - Server/Networking:
    - clone SAS HDDs to SAS SSDs
    - migrate from ESXi 6.5 to 6.7

Long-term ToDo's:
 - Server/Networking:
    - VDI host when (DL580 Gen8/9 planning)?
    - replace Titan Z with Radeon Pro v320/v340
 - macOS Mojave:
    - upgrade to Big Sur 11.2.3
    - update MacPorts and Xcode

Unconfirmed ToDo's:
 - Artix OpenRC:
    - Docker container: YaCy Grid
        - <https://blog.fossasia.org/creating-a-dockerfile-for-yacy-grid-mcp/>
        - <https://github.com/yacy/yacy_grid_mcp/blob/master/docker/all-in-one/docker-compose.yml>
        - initiate web crawl

 

 

oof...

0 Kudos