TopHatProductio
Hot Shot
Hot Shot

New Server Project

Hello! It's been a while since I last posted here with my own topic. I now have a dedicated ESXi server in the works, and am planning to start using it 24/7 by the end of this year or early next year (2021). Here are the specs for the hardware:

 

HPE ProLiant DL580 G7

 

CPU  :: 4x Intel Xeon E7-8870's (10c/20t each; 40c/80t total)
RAM  :: 256GB (64x4GB) DDR3-1333 PC3-10600R ECC 
PCIe :: 1x HP 512843-001/591196-001 System I/O board + 
                1x HP 588137-B21; 591205-001/591204-001 PCIe Riser board +
GPU  :: 1x nVIDIA GeForce GTX Titan Xp +
                1x nVIDIA GeForce GTX Titan Z
SFX  :: 1x Creative Sound Blaster Audigy Rx
NIC  :: 1x HPE NC524SFP (489892-B21)
I/O  :: 1x HPE PCIe ioDuo MLC 1.28TB I/O Accelerator (641255-001) +
                1x SanDisk Fusion ioScale MLC 3.2TB Accelerator (F11-002-3T20-CS-0001)
STR  :: 1x HP Smart Array P410i Controller (integrated) +
                1x HGST HUSMM8040ASS200 MLC 400GB SSD (ESXi, vCenter Appliance, ISOs) + 
                4x HP 507127-B21 300GB HDDs (ESXi guest datastores) +
                1x Western Digital WD Blue 3D NAND 500GB SSD (Virtual Flash) + 
                1x Intel 320 Series SSDSA2CW600G3 600GB SSD (VFF) +
                1x Seagate Video ST500VT003 500GB HDD 
        1x LSI SAS 9201-16e HBA SAS card +
            1x Mini-SAS SFF-8088 cable + 
                1x Dell EMC KTN-STL3 (15x 3.5in HDD enclosure) + 
                    4x HITACHI Ultrastar HUH728080AL4205 8TB HDDs +
                    4x IBM Storewise XIV v7000 98Y3241 4TB HDDs
        1x Sony Optiarc BluRay drive
PRP  :: 1x AOC U2879VF (4K)
                1x Inateck KU5211 (USB 3.2) +
                    1x Logitech K845 (Cherry MX Blue) +
                    1x Dell MS819 Wired Mouse
PSU  :: 4x HP 1200W PSU's (441830-001/438203-001)

 

 

Product links and details can be found here.

 

Here is the current software configuration plan for the server:

 

*  Temporary task that will be replaced by a permanent, self-hosted solution

** Can benefit from port forwarding, but will be primarily tunnel-bound

^  Tunnel-bound (VPN/SSH) role - not port forwarded/exposed to the Internet

+ Active Directory enabled - Single Sign On (SSO)

 

Here is the current resource allocation plan for the server:

 

VMware VCSA (vCenter) :: 24/7 - true , dedicatedHDD - false, dedicatedGPU - false,   2c/4t  + 12GB
Windows Server 2016   :: 24/7 - true , dedicatedHDD - true , dedicatedGPU - false,   8c/16t + 16GB
Artix Linux - Xfce    :: 24/7 - true , dedicatedHDD - true , dedicatedGPU - false,   8c/16t + 16GB
macOS Server 10.14.X  :: 24/7 - true , dedicatedHDD - true , dedicatedGPU - true ,   8c/16t + 16GB        (NOT TO BE DISCUSSED HERE)
Windows 10 Enterprise :: 24/7 - false, dedicatedHDD - true , dedicatedGPU - true ,  12c/24t + 32GB *
Temporary/Testing VM  :: 24/7 - false, dedicatedHDD - N/a  , dedicatedGPU - N/a  ,  --c/--t + --GB *

 

 

Product links and details can be found here.

 

VMs marked with an * cannot be run at the same time. Only one of them can ever run at any given moment. MacOS and Linux would have gotten a Radeon/FirePro (ie., Rx Vega 64), for best compatibility and stability, but market forces have prevented this. Windows 10 gets the Creative Audigy Rx. The MacOS and Linux VMs get whatever audio the Titan Z provides (either that or a software solution). Windows 10, Remote Development, and the Temp/Testing VM will be put to sleep (or offed) until they are needed (Wake on LAN), since they don't host any essential services.

 

There are three other mirrors for this project, in case you're interested in following individual conversations from the other sites (in addition to this thread).

 

P.S. Out of all the sites that I've ever used, this forum has one of the best WYSIWYG editors I've used in a while Smiley Happy

Kudos to the devs!

Tags (1)
189 Replies
TopHatProductio
Hot Shot
Hot Shot

It's getting close to that time again - where I have to look into cloning drives for the server again. And we all know how it went last time 😂

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Updated ToDo List, once more (since I just gave myself more work - funny how that works):

Current ToDo's:
 - Windows Server 2016:
    - FreeSWITCH (vPBX) configuration
        - https://freeswitch.org/confluence/display/FREESWITCH/XML+Switch+Configuration
        - https://freeswitch.org/confluence/display/FREESWITCH/Directory
        - https://freeswitch.org/confluence/display/FREESWITCH/mod_ldap
        - https://freeswitch.org/confluence/display/FREESWITCH/mod_voicemail
        - https://freeswitch.org/confluence/display/FREESWITCH/mod_sms

Upcoming ToDo's:
 - Server/Networking:
    - convert the VMs (MBR => GPT, BIOS => UEFI)
    - clone SAS HDDs to SAS SSDs (Storage vMotion?)
    - migrate from ESXi 6.5 to 6.7

Long-term ToDo's:
 - Server/Networking:
    - VDI host when (DL580 Gen8/9 planning)?
    - replace Titan Z with Radeon Pro v320/v340
 - macOS Mojave:
    - upgrade to Big Sur 11.2.3
    - update MacPorts and Xcode
    - sNTP client configuration (maybe)?

Unconfirmed ToDo's:
 - Artix OpenRC:
    - Docker container: YaCy Grid
        - https://blog.fossasia.org/creating-a-dockerfile-for-yacy-grid-mcp/
        - https://github.com/yacy/yacy_grid_mcp/blob/master/docker/all-in-one/docker-compose.yml
        - initiate web crawl

 

I might never be finished. On the other hand, the original plan did include a VoIP/PBX. I guess this is where it happens...

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Just kicked FreeSWITCH for izPBX, and actually got izPBX working this time. That was pretty tedious to get working. Now to see if it can survive a container/VM reboot. Last time, it didn't go so well...

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

FreePBX has been jettisoned from the project, permanently:

Getting ready to close out the SIP trunks in a few hours...

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

New ToDo List, adjusted for final removal of telecommunications from the project:

Current ToDo's:
 - Server/Networking:
    - convert the VMs (MBR => GPT, BIOS => UEFI)
    - clone SAS HDDs to SAS SSDs (Storage vMotion?)
    - migrate from ESXi 6.5 to 6.7

Upcoming ToDo's:
 - Server/Networking:
    - VDI host when (DL580 Gen8/9 planning)?
    - replace Titan Z with Radeon Pro v320/v340 *
 - macOS Mojave:
    - upgrade to Big Sur 11.2.3
    - update MacPorts and Xcode
    - sNTP client configuration (maybe)?

Long-term ToDo's:
 - Artix OpenRC:
    - Docker container: YaCy Grid
        - https://blog.fossasia.org/creating-a-dockerfile-for-yacy-grid-mcp/
        - https://github.com/yacy/yacy_grid_mcp/blob/master/docker/all-in-one/docker-compose.yml
        - initiate web crawl
0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Currently looking to see if I can get Cisco CUCM. Might not be likely, but it's my last attempt at managing a phone system on-prem. Otherwise, I won't be looking into it again until the next version of this project...

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Reorganised ToDo's to reflect current priorities and project direction:

Current ToDo's:
 - Server/Networking:
    - convert Windows VMs (MBR => GPT, BIOS => UEFI)
    - migrate from vSphere 6.5 to 6.7 (ESXi)
    - VDI host when (DL580 Gen8/9 planning) ?

Upcoming ToDo's:
 - Server/Networking:
    - purchase HPE ProLiant DL580 Gen8
    - replace Titan Z with Radeon Pro v340
    - Move VMs to new ESXi host (Storage vMotion)
 - macOS Mojave:
    - upgrade to Big Sur 11.2.3
    - update MacPorts and Xcode
    - sNTP client configuration (maybe) ?

Long-term ToDo's:
 - Artix OpenRC:
    - Docker container: YaCy Grid
        - https://blog.fossasia.org/creating-a-dockerfile-for-yacy-grid-mcp/
        - https://github.com/yacy/yacy_grid_mcp/blob/master/docker/all-in-one/docker-compose.yml
        - initiate web crawl
 - Install Arch (OpenRC+ZFS on UEFI) from scratch
    - move all Docker containers to new Arch host

 

Yep - planning on possibly moving from Artix to pure Arch, to see if I can bake in the ZFS support I've been wanting this whole time. Haven't found anything (yet) that allows for easy conversion from other filesystems to ZFS, so might be easier to go with ZFS from the beginning in any case. Perhaps My Timeshift backups will be enhanced by this as well...

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Last ToDo List of the Week, too tired...

Current ToDo's:
 - Cisco CUCM
 - Artix OpenRC:
    - Docker container: YaCy Grid
        - https://blog.fossasia.org/creating-a-dockerfile-for-yacy-grid-mcp/
        - https://github.com/yacy/yacy_grid_mcp/blob/master/docker/all-in-one/docker-compose.yml
        - initiate web crawl

Upcoming ToDo's:
 - Server/Networking:
    - convert Windows VMs (MBR => GPT, BIOS => UEFI)
    - migrate from vSphere 6.5 to 6.7 (ESXi)

Long-term ToDo's:
 - Server/Networking:
    - VDI host when (DL580 Gen8/9 planning)
    - purchase HPE ProLiant DL580 Gen8
    - replace Titan Z with Radeon Pro v340
    - Move VMs to new ESXi host (Storage vMotion)
 - macOS Mojave:
    - upgrade to Big Sur 11.2.3
    - update MacPorts and Xcode
    - sNTP client configuration (maybe) ?
 - Artix OpenRC Reborn
    - Reinstall on UEFI, w/ OpenZFS from the start
    - New partition scheme (see below)

 

Partition scheme:

400GB SAS SSD
    - _ESP   512MB, FAT32, /efi         (esp)
    - root   320GB,  EXT4, /            (root,system)
    - home    64GB,  EXT4, /home        (home)
    - swap    16GB,  swap, [!mnt_point] (swap)
8TB SAS HDD
    - services 2TB,   ZFS, /srv         (srv)
    - variable 5TB,   ZFS, /var         (var)
8TB SAS HDD
    - backup   8TB,   ZFS, [!mnt_point] [!flag]

ZFS on root partition is off-limits until this is resolved:

On a side note:

 

Still wondering if I should test out a BSD VM one day, once I move to a newer host...

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Last ToDo List of the Week, too tired EDITED...

Current ToDo's:
 - Cisco CUCM demo/pricing
 - Windows Server 2016:
    - convert (MBR => GPT, BIOS => UEFI)
    - Upgrade from 2016 to 2019 (friggin' update times)
 - Artix OpenRC:
    - Docker container: Tor node(s)/relay(s)
    - Docker container: Discord bridge (matterbridge)
        - https://nextcloud.com/blog/bridging-chat-services-in-talk/
 - Windows 10 Enterprise:
    - convert (MBR => GPT, BIOS => UEFI)
 - Server/Networking:
    - migrate from vSphere 6.5 to 6.7 (ESXi)

Upcoming ToDo's:
 - Server/Networking:
    - purchase HPE ProLiant DL580 Gen8
    - VDI host when (DL580 Gen8/9 planning)
    - replace Titan Z with Radeon Pro v340
    - Move VMs to new ESXi host (Storage vMotion)
 - macOS Mojave:
    - upgrade to Big Sur 11.2.3
    - update MacPorts and Xcode
    - sNTP client configuration (maybe) ?

Long-term ToDo's:
 - Artix OpenRC:
    - Docker container: YaCy Grid
        - https://blog.fossasia.org/creating-a-dockerfile-for-yacy-grid-mcp/
        - https://github.com/yacy/yacy_grid_mcp/blob/master/docker/all-in-one/docker-compose.yml
        - initiate web crawl
 - Install Arch (OpenRC+ZFS on UEFI) from scratch
    - move all Docker containers to new Arch host
    - or convert existing VM ?!
    - New partition scheme (see below)

 

Partition/Volume arrangement:

400GB SAS SSD
    - _ESP   512MB, FAT32, /efi         (esp)
    - root   320GB,  EXT4, /            (root,system)
    - home    64GB,  EXT4, /home        (home)
    - swap    16GB,  swap, [!mnt_point] (swap)
8TB SAS HDD
    - services 2TB,   ZFS, /srv         (srv)
    - variable 5TB,   ZFS, /var         (var)
8TB SAS HDD
    - backup   8TB,   ZFS, [!mnt_point] [!flag]

 

ZFS on root partition is off-limits until this is resolved:

On a side note:

 

Still wondering if I should test out a BSD VM one day, once I move to a newer host. Finances are currently very tight, and I still need to purchase a few more EaseUS licenses. The CUCM idea may have to be put  off for a while, as I try to figure out the rest of the objectives here. Installing Windows 10 Enterprise on the Threadripper has proven to be more difficult than originally expected. Also considering switching cell service providers in the next 30 days. Life comes at ya fast...

0 Kudos
Miriammorri
Contributor
Contributor

400GB SAS SSD
    - _ESP   512MB, FAT32, /efi         (esp)
    - root   320GB,  EXT4, /            (root,system)
    - home    64GB,  EXT4, /home        (home)
    - swap    16GB,  swap, [!mnt_point] (swap)

facing issue on this site: https://keyboardtestt.com/

0 Kudos
BigMike23
Enthusiast
Enthusiast

Thanks for this link

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Getting back on topic, from whatever happened previously...

2022 is proving to be a tougher year than the previous one, when it comes to getting major tasks done. I may end up focusing more on the Docker host for a while, since that’s where I’ll be able to make the most progress without breaking the bank. Gotta take a slower pace, to determine a few alternative routes for some of these tasks...

Current ToDo's:
 - Cisco CUCM demo/pricing (on hold, due to finances)
 - Artix OpenRC:
    - Docker container: Discord bridge (matterbridge)
        - https://nextcloud.com/blog/bridging-chat-services-in-talk/
    - Docker container: YaCy Grid
        - https://blog.fossasia.org/creating-a-dockerfile-for-yacy-grid-mcp/
        - https://github.com/yacy/yacy_grid_mcp/blob/master/docker/all-in-one/docker-compose.yml
        - initiate web crawl
    - Docker container: Tor node(s)/relay(s)
 - Windows 10 Enterprise:
    - convert (MBR => GPT, BIOS => UEFI)

Upcoming ToDo's:
 - Windows Server 2016:
    - convert (MBR => GPT, BIOS => UEFI) w/ AOMEI license
    - Upgrade from 2016 to 2019 (friggin' update times)
 - Server/Networking:
    - migrate from vSphere 6.5 to 6.7 (ESXi)

Long-term ToDo's:
 - Server/Networking:
    - purchase HPE ProLiant DL580 Gen8
    - VDI host when (DL580 Gen8/9 planning)
    - replace Titan Z with Radeon Pro v320/v340
    - Move VMs to new ESXi host (Storage vMotion)
 - macOS Mojave:
    - upgrade to Big Sur 11.2.3
    - update MacPorts and Xcode
    - sNTP client configuration (maybe) ?
 - Install Arch (OpenRC+ZFS on UEFI) from scratch
    - move all Docker containers to new Arch host
    - or convert existing VM ?!

 

Finances are still tight, and converting Windows Server from MBR to GPT will cost money since Microsoft left that conversion tool out on Windows Server. AOMIE software license will need to be purchased as a result. Can’t purchase the previously planned EaseUS Backup licenses, due to other expenses that came up recently. I’m at least 2 paychecks (a full month) behind on these acquisitions, and I’m still trying to navigate talks with Cisco, pertaining to CUCM. I’m gonna try to install Windows 10 Enterprise on the Threadripper one last time, sometime in June. Here’s to hoping things lighten up later, maybe in the Autumn…

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Boy, things have gotten pretty weird lately. About halfway through May, I learned that I'll need more licenses for EaseUS (to manage Windows backups). Also had to delay a demo meeting with Cisco, pertaining to CUCM/WebEx. Wanted to add Tor nodes and Matterbridge to the project (as Docker containers), only for the potential risks of the former and complications (between Discord and Nextcloud chat) of the latter to dissuade me on the limited value/returns I'd receive. Days later (05/15), Windows 10 Enterprise refused to install on the Threadripper. Slightly after that (05/19), I find out that Ms left their free MBR-to-GPT conversion tool out of Windows Server 2016 - necessitating the purchase of another software license (AOMIE). On 05/21 (midnight), I moved to Nextcloud 22.2.8, in preparation for the move to version 23. On that same day, the air quality was so bad that I couldn't run the server later. It stayed off until the 26th iirc. During that time, I re-realised that I'll need a rackmount AC at some point in the future (had already looked into it before). Between May 27 and 29, I was updating Portainer and purchasing my current domain from Freenom, outright. Also considered hosting an Invidious instance. During the May/June transition, I had to move my mobile devices from StraightTalk to T-Mobile, because the former decided to drop support for the Asus ZenFone 6. They also decided that their VoLTE implementation didn't need to support said device. And they had a nerve to waste my time (days of it) with poor customer service as I tried and failed to transfer my phone number to T-Mobile (to go easy on people's contacts lists). Needless to say, Google Voice is the only reason that didn't end up becoming a complete mess. I stopped giving out my real phone number a while ago, because telephone companies don't have to play nice. At least T-Mobile has better customer service from what I've seen. On June 2nd, I was supposed to look into group policy to reduce Windows 10 telemetry - but got sidestepped by a potential File Explorer bug! On June 4th, I did more research on the requirements for Matterbridge and YaCy Grid. Matterbridge was going to require a ton of work, for something that I'd probably get little-to-no use out of. YaCy Grid is not ready for prime time, and the current config examples only show off Elastisearch. Not to say that they haven't been working on it, but I'd have to build most of it by hand, which would devour time required for other tasks already planned. The Tor node ended up getting scrapped as well, due to potential risks of hosting said instance. The only thing that survived all of that was the desire to get vanilla Yacy running. On June 7th, I finally managed to give away the Threadripper, freeing up rackspace and the Titan V. On the 10th, I was looking into Docker for Windows containers (no, not that one - the other one with less cringe), for hosting multiple isolated instances of MariaDB. Today, the update to Chromium v102 reigned down on my setup with SSL pains and the terror of frantically scrambling to download/install certificates, while being forced to switch some instances back to HTTP because self-signed certificates are everywhere and sometimes tough to replace. And establishing a proper, self-hosted CA that won't drain you of your funds is tough. Between all of these, I was battling with tight finances because or a college loan and a certificate loan, and stupidly poor air quality due to temperatures+humidity+allergens preventing me from turning on the server (software and driver updates have to happen sometime). This weekend, I was originally supposed to be 1) configuring group policy for restricting Ms telemetry, and 2) setting up a YaCy instance. It's 4am. I still have a ToDo list to update!

/endrant

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

The ToDo List for the next 24 months:

Current ToDo's:
 - Artix OpenRC:
    - Docker container: YaCy (non-Grid)
        - https://hub.docker.com/r/yacy/yacy_search_server
 - Windows Server 2016:
    - purchase AOMEI Partition Assistant Server license
        - https://www.diskpart.com/partition-manager-server-edition.html
    - purchase (2) more EaseUS Backup licenses for Windows client PCs

Upcoming ToDo's:
 - Windows 10 Enterprise:
    - convert (MBR => GPT, BIOS => UEFI)
 - Windows Server 2016:
    - convert (MBR => GPT, BIOS => UEFI) with AOMEI
    - Upgrade from 2016 to 2019 (friggin' update times)
 - Server/Networking:
    - migrate from vSphere 6.5 to 6.7 (ESXi)

Long-term ToDo's:
 - Server/Networking:
    - purchase HPE ProLiant DL580 Gen9
    - purchase AMD Radeon Pro v340
    - replace Titan Z with Radeon Pro v320/v340
    - Move VMs to new vSphere host (Storage vMotion)
    - VDI host when (DL580 Gen9 planning)
 - macOS Mojave:
    - upgrade to Big Sur 11.2.3
    - update MacPorts and Xcode
 - Artix OpenRC:
    - Docker container: OpenStreetMaps
    - Docker container: OSMR Backend
    - Docker container: izPBX (FreePBX), or
 - FreePBX Distro VM!!
    - https://www.freepbx.org/downloads/
 - Install Arch (OpenRC+ZFS on UEFI) from scratch
    - move all Docker containers to new Arch host
    - or convert existing VM ?!

 

Just got new info about macOS Ventura, thanks to an amazing Reddit user who had the stones to install it in a VM themselves:

AVX2 is the new hurdle, so Haswell is the minimum. ProLiant Gen8 is out, going for the Gen9 instead - which means that I move to DDR4 earlier than anticipated. The platform cost is higher, since I get to reuse less of my current hardware. The move to Windows Server 2019 should make managing updates for that VM much easier. I'm also considering removing Technitium, since I've been mostly relying on Active Directory DNS for the longest time. It was great for when I first started out, since I didn't have any other DNS source on my local network. But, for anyone out there not using AD or similar, Technitium is beautiful! Finances are still tight, and I've had to change some objectives. CUCM is probably out-of-reach at this point, and I still wanna grab a small group of IPv4 addresses. That will have to wait until after I get the new Gen9 server in-house. Also, I need to buy more movies to throw onto PleX - Movie Night won't make itself, ya know!


P.S. Chrome v102 blew out all of my self-signed SSL certificates. I'll be remaking those for the next few weeks, since I decided to be my own CA...

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Time to troubleshoot OnlyOffice+Nextcloud​​​​​​​. Happened after moving everything to HTTPS.

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

So far, things have been moving at a sporadic pace, depending heavily on my own capabilities. With that said, I guess that 5 years isn't too long for a tech project XD

Spent 2018 experimenting (and breaking things) on a smaller host. 2019 was a major planning year, and funds were almost non-existent. It was also when I finally decided on the form factor - 4U rackmount. 2020 was the first year I could actually start buying most of the hardware. 2021 was when the pace picked up, and I started spinning up more VMs iirc. Now, I'm in 2022, planning the next host.

Time really does fly...

0 Kudos
TopHatProductio
Hot Shot
Hot Shot

I wanna replace the MikroTik RB4011iGS+RM and Audience RBD25GR-5HPac with an RB4011iGS+5HacQ2HnD-IN. Simpler setup, and I get to move a WAP somewhere else. 

0 Kudos