New Server Project

[TopHatProductio]

Hello! It's been a while since I last posted here with my own topic. I now have a dedicated ESXi server in the works. The server project is meant to replace (and exceed) my previous workstation - a Dell Precision T7500. Here are the specs for the hardware:

HPE ProLiant DL580 G7

    OS   :: VMware ESXi 6.5u3 Enterprise Plus
    CPU  :: 4x Intel Xeon E7-8870's (10c/20t each; 40c/80t total)
    RAM  :: 256GB (64x4GB) PC3-10600R DDR3-1333 ECC
    PCIe :: 1x HP 512843-001/591196-001 System I/O board + 
                1x HP 588137-B21; 591205-001/591204-001 PCIe Riser board
    GPU  :: 1x nVIDIA GeForce GTX Titan Xp +
                1x AMD FirePro S9300 x2 (2x "AMD Radeon Fury X's")
    SFX  :: 1x Creative Sound Blaster Audigy Rx
    NIC  :: 1x HPE NC524SFP (489892-B21) +
                2x Silicom PE310G4SPI9L-XR-CX3's
    STR  :: 1x HP Smart Array P410i Controller (integrated) +
                1x HGST HUSMM8040ASS200 MLC 400GB SSD (ESXi, vCenter Appliance, ISOs) + 
                4x HP 507127-B21 300GB HDDs (ESXi guest datastores) +
                1x Western Digital WD Blue 3D NAND 500GB SSD + 
                1x Intel 320 Series SSDSA2CW600G3 600GB SSD +
                1x Seagate Video ST500VT003 500GB HDD
    STR  :: 1x LSI SAS 9201-16e HBA SAS card +
                1x Mini-SAS SFF-8088 cable + 
                        1x Dell EMC KTN-STL3 (15x 3.5in HDD enclosure) + 
                                4x HITACHI Ultrastar HUH728080AL4205 8TB HDDs +
                                4x IBM Storewise XIV v7000 98Y3241 4TB HDDs
    I/O  :: 1x Inateck KU8212 (USB 3.2) +
                1x Logitech K845 (Cherry MX Blue) +
                1x Dell MS819 Wired Mouse
            1x Sonnet Allegro USB3-PRO-4P10-E (USB 3.X) +
                1x LG WH16NS40 BD-RE ODD
    PRP  :: 1x Samsung ViewFinity S70A UHD 32" (S32A700)
            1x Sony Optiarc BluRay drive
    PSU  :: 4x HP 1200W PSUs (441830-001/438203-001)

The details for the ProLiant DL380 Gen9 will appear here once data migration is complete. VMware Horizon (VDI) will have to wait for a future phase (if implemented at all). The current state of self-hosted VDI is Windows-centric, with second-hand support for Linux and no proper support for macOS.

The planned software/VM configurations have been moved back to the LTT post, and will be changing often for the foreseeable future.

Product links and details can be found here.

ESXi itself is usually run from a USB thumb drive, but I have a drive dedicated to it. No harm done. A small amount of thin provisioning/overbooking (RAM only) won’t hurt. macOS and Linux would have gotten a Radeon/FirePro (ie., Rx Vega 64), for best compatibility and stability, but market forces originally prevented this. Windows 10 gets the Audigy Rx and a Titan Xp. The macOS and Linux VMs get whatever audio the Titan Z FirePro S9300 x2 can provide. The whole purpose of Nextcloud is to phase out the use of Google Drive/Photos, iCloud, Box.com, and other externally-hosted cloud services (Mega can stay though).

There are three other mirrors for this project, in case you're interested in following individual conversations from the other sites (in addition to this thread).

• Level1Techs
• Linus Tech Tips
• TechPowerUp

P.S. Out of all the sites that I've ever used, this forum has one of the best WYSIWYG editors I've used in a while

Kudos to the devs!

[TopHatProductio]

Well, that was fun:

• https://forum.mikrotik.com/viewtopic.php?p=988778

Looks as though I will have to experiment on my own with VLANs from here on...

[TopHatProductio]

I installed the 2x Silicom PE310G4SPI9L-XR-CX3's yesterday, and they appear to be working. Next task will be implementing VLANs on the new MikroTik RB4011iGS+5HacQ2HnD-IN-US, which I purchased recently to replace the RBD25G-5HPacQD2HPnD and RB4011iGS+. Having the Router and WAP in a single, rackmountable appliance should make cleaning and managing the server rack much easier in long-term. After this, I will be focusing all of my attention on Project ArcZ. Once ArcZ is complete (and has replaced Rocky Linux), I will be working on LXC/LDX containers (as described above). No new containers will be added until ArcZ is ready for daily use. This is now the primary focus of 2023. Some of this won't be possible until I move out (anything requiring resources only present on the Gen9). I will make ArcZ's setup script(s) publicly available in a few months. Do keep in mind, it's built for use in an ESXi VM. This is a massive change in direction for the project, and will push back things quite a bit...

[TopHatProductio]

​Updated Task List (03/26/2023):

Current ToDo's:
 - Server/Networking:
    - Set RFC2307 attributes for AD Users
        - https://wiki.samba.org/index.php/Setting_up_Samba_as_a_Domain_Member#Configuring_Samba
        - https://www.server-world.info/en/note?os=Windows_Server_2019&p=active_directory&f=12
        - https://github.com/assen-totin/powershell-unixattributes
        - https://github.com/hkbakke/ad-posix-attrs
    - Project vNet (network isolation via vLANs)
        - https://forum.mikrotik.com/viewtopic.php?p=988254
 - Artix OpenRC (Xfce):
    - Docker Stack: Portainer EE
        - MFA for AD users, via Azure AD OAuth
        - https://docs.portainer.io/admin/settings/authentication/oauth
    ** Replace with Project ArcZ (Artix OpenRC on ZFS root) **

Upcoming ToDo's:
 - Rocky Linux (Wayland/XFS):
    ** Replace with Project ArcZ-EE **
 - Server/Networking:
    - Project New Client
        - Clean install Windows 10 Enterprise LTSC
        - Install Samsung 850 Pro 2TB SSD

Long-term ToDo's:
 - DL580 Gen9 transition (24/7 instances)
 - Artix OpenRC (ArcZ):
    - Docker Stack: Pleroma (federated)
        - https://github.com/explodingcamera/docker-pleroma
    - Docker Stack: YaCy Grid
        - https://github.com/yacy/yacy_grid_mcp/blob/master/docker-compose.yml
        - https://community.searchlab.eu/t/pertaining-to-how-yacy-crawls-websites/1090
 - FreePBX Distro:
    - Port out Google Voice phone# to VoIP.ms (DID)
        - https://support.google.com/voice/answer/10130510?hl=en
        - https://support.google.com/voice/answer/1065667?hl=en
        - https://wiki.voip.ms/article/Porting_FAQ#How_to_port_my_Google_Voice_Number_to_VoIP.ms

I'll be studying for VMware VCP-DCV and CompTIA Server+ when I'm not working on Projects vNet and ArcZ. Cisco CCNA will have to wait, though. While I'm constantly picking up knowledge in that arena (esp. IPv6 and VLANs), that thing will take a while to prep for. Certification maintenance/renewal costs also exist, and CompTIA's fees can go up as I get higher-level certs.

All of the major Wazuh XDR and Nextcloud changes are being put on hold for now, due to time limitations (there's currently only one admin). Blackbird deployment and Malwarebytes removal (Windows 10 Enterprise VM and Project New Client) are delayed as well (dependent on progress with Wazuh).

This probably will end up becoming a major project overhaul...

[TopHatProductio]

Updated Task List (04/04/2023):

Current ToDo's:
    - Project vNet 1.0
        - Tag ether1 for VLAN ID 51 (Management)
        - Configure static IPv4 address in iLO3 settings
        - Test ESXi DCUI for emergency troubleshooting/access
        - Create aux. vmkernel adapter and portgroup (VLAN ID 51)
        - Configure static IPv4 address for aux. vmkernel adapter
        - Configure vCSA to use DHCP and migrate to VLAN 51
 - Server/Networking:
    - Renew OnlyOffice EE server license
    - Renew vCSA SSL certificates (10/2023)
    - Remove LibreNMS, in favour of MikroTik's "The Dude"
    - Set RFC2307 attributes for AD Users
        - https://wiki.samba.org/index.php/Setting_up_Samba_as_a_Domain_Member#Configuring_Samba
        - https://www.server-world.info/en/note?os=Windows_Server_2019&p=active_directory&f=12
        - https://github.com/assen-totin/powershell-unixattributes
        - https://github.com/hkbakke/ad-posix-attrs

Upcoming ToDo's:
 - Artix OpenRC (Xfce):
    ** Replace with Project ArcZ (Artix OpenRC on ZFS root) **
 - Rocky Linux (Wayland/XFS):
    ** Replace with Project ArcZ-EE **
    - Setup UrBackup Server

Long-term ToDo's:
 - DL580 Gen9 transition (24/7 instances)
 - Artix OpenRC (ArcZ):
    - Docker Stack: Portainer EE
        - https://docs.portainer.io/admin/settings/authentication/oauth#microsoft
    - Docker Stack: Pleroma (federated)
        - https://github.com/explodingcamera/docker-pleroma
    - Docker Stack: YaCy Grid
        - https://github.com/yacy/yacy_grid_mcp/blob/master/docker-compose.yml
        - https://community.searchlab.eu/t/pertaining-to-how-yacy-crawls-websites/1090
 - FreePBX Distro:
    - Port out Google Voice phone# to VoIP.ms (DID)
        - https://support.google.com/voice/answer/10130510?hl=en
        - https://support.google.com/voice/answer/1065667?hl=en
        - https://wiki.voip.ms/article/Porting_FAQ#How_to_port_my_Google_Voice_Number_to_VoIP.ms

Replacing ConnMan with NetworkManager was a fun little side task this morning.

While most of the VMs have been successfully migrated, backend infrastructure/services have not yet migrated. That will happen this coming weekend. I've also noticed some concerning performance degradation when attempting to access services like Nextcloud.

Today's time with LibreNMS has shown me that I need to slim things down. I have no more time to mess with it, and am replacing its VM. That time needs to go into Project ArcZ. Cronicle and Wazuh XDR are here to stay. Blackbird deployment and Malwarebytes removal (Windows 10 Enterprise VM and Project New Client) are scheduled for the 2023/2024 transition.

After seeing what Framework has been up to recently, I don't think I'll be continuing Project New Client with the EliteBook 8770w. Their upcoming 16-inch Ryzen mobile workstation (w/ dGPU option) would be more than a worthy replacement.

I'll be studying for VMware VCP-DCV and CompTIA Server+ when I'm not working on Project vNet or ArcZ. Once I've finished that, I may move onto Cisco CCNA. That thing will take a while to prep for, and I'm busy learning VLANS. Certification maintenance/renewal costs exist, and CompTIA's fees can go up as I get higher-level certs iirc.

Cloudflared, Pleroma, and PeerTube are back on the unconfirmed list. Gotta get more prior tasks done first.

I still have a Chateau 5G ax to unbox...

[TopHatProductio]

Getting closer to finishing Project vNet:

https://forum.mikrotik.com/viewtopic.php?t=195197

[TopHatProductio]

The following services have been moved into the Unconfirmed list:

• Pleroma
• PeerTube
• YaCy Grid

Pleroma and PeerTube are federated services, and would entail (potentially) increased exposure for the server. This is a potential security concern, especially for services that are not guaranteed to receive consistent use from the current user base. For PeerTube specifically, we would also potentially need dedicated hardware to have decent media encode/transcode performance (either many CPU cores w/ AVX support, GPUs with ASICs, or dedicated PCIe media cards). YaCy Grid, as a peer-driven service, could run behind the VPN without issue. But, the quality of its service(s) would at least somewhat rely on the work/data of other peers on the YaCy network. While I was willing to play around with this years ago, I now find myself having to decide what apps and services can have dedicated RAM allocated to them. For the potentially large amount of resources that YaCy may consume, I would need to know that there are enough peers in my area to make it usable when compared to other alternatives. Not to mention, the amount of persistent data that these can generate if people actually start to use them. As such, these services will most likely be delayed to a later phase of the project, when there are more servers running in-house.

[TopHatProductio]

The CRS317-1G-16S+RM is here. If this goes well, the CRS326-24G-2S+RM​​​​​​​ might be getting sold or given away...

[TopHatProductio]

Currently attempting to redo the VLAN setup:

• https://www.reddit.com/r/mikrotik/comments/12ybec4/vlan_troubleshooting/

[TopHatProductio]

Some news and updates:

• The 8TB SAS HDD for the Windows 10 VM died recently
• The ISP router threw a fit and knocked me offline for a weekend
• The current SMTP relay decided to temp. suspend my account
• I've found a replacement SMTP relay as a result, and am switching over this week
• FreePBX can no longer do outbound calls after the ISP router spat
• Haven't determined what caused the ISP router to crap itself yet
• Got three more SAS HDDs for the next iteration of the server project
• One of them is to replace the one for the Windows 10 VM
• Bridge VLAN filtering is coming soon, and will be used to move iLO3 to a VLAN
• Still need to figure out how to move vCenter Appliance to the new VLAN
• Determining if Nextcloud needs to move to an LXD container

And there's more coming soon...

[TopHatProductio]

More VLAN configuration:
https://communities.vmware.com/t5/vCenter-Server-Discussions/Attempting-to-Move-vCenter-Server-Appli...

[TopHatProductio]

Getting closer to migrating vCenter...

• https://communities.vmware.com/t5/vCenter-Server-Discussions/Attempting-to-Move-vCenter-Server-Appli...

New comments added on how to manually configure VLAN adapter.

[TopHatProductio]

vCenter has been successfully migrated. iLO3 is the only one left. Moving to ArcZ is next...

[TopHatProductio]

hMailServer has finally been configured with the new SMTP relay. Initial configuration for ArcZ (reduced target) has been completed. Now I need to create a template out of it, for future VM deployments. In opposed to configuring ZSF on a per-VM basis, I have decided to look into getting a TrueNAS VM or appliance for an iSCSI SAN. This should reduce time and complexity cost for managing storage arrays/pools, which is a must at this point. Upgrading macOS has been set back to post-DL580Gen9 transition, due to recent PSOD's -- which were strikingly similar to what happened with BlissOS. I'm still deciding on whether to renew OnlyOffice EE, which is currently ~2.4k USD for 3 years (since I count as an org). My decision is due by the end of this month. Also trying to determine whether to move Nextcloud to an LXD container.

[TopHatProductio]

It's been a while since I last updated everyone. Here's the summary:

• Originally wanted to create a vSphere template of Project ArcZ, but ran into storage constraints
• Started looking into Linux-native cloning/templating tools
  • They all either can't clone while online, don't support BTRFS, don't support non-systemd distros, can't run in OCI container, or some combo of the former(s)
  • Sticking to TimeShift and EaseUS
• I'm going to end up buying more 8TB SAS HDDs instead, to have space for everything
• I wanna have all of my large SAS HDDs managed by TrueNAS Scale, as ZFS-backed iSCSI storage pools
  • And have ESXi be the primary iSCSI initiator
• Still need to setup Cronicle Edge, Fleet MDM, and Wazuh XDR

Summer is insanely hot -- no fun cooling the server room. Also, software licenses for NT domain environments are very expensive.

[TopHatProductio]

With the move to Cronicle Edge (Docker container), I was never able to retrieve and apply the secret_key value needed to link all Cronicle hosts across my network. No matter what I tried, I was never able to find the value that the manager host was using. As such, it became impossible for me to manage the other worker hosts. I even considered using Cronicle Edge's SSH plugin as an agentless solution in the meantime, but that was simply a bandage over the wound. Ended up wiping out Cronicle in its entirety, and am now learning crontab basics instead. The sad part is, I truly liked using Cronicle when it worked. But this lost day of progress burned me. I was supposed to be working on cloning the ArcZ VM today, but that never happened. Also considered bringing back LibreNMS, but I've eaten through too much time messing around with Cronicle. For now, I need one more 8TB SAS HDD and more RAM to kick OSRM to a dedicated VM with ~96GB RAM...

[TopHatProductio]

Just grabbed a DL380 Gen9 for TrueNAS Scale on baremetal. ZFS via iSCSI coming soon...

[TopHatProductio]

Since last time, I've managed to order:

• 1x ProLiant DL380 Gen9 (2x Intel Xeon E5-2670v3's, 64GB DDR3 ECC)
• 16x Toshiba THNSNJ256GCSY 256GB SSDs
• 3x HGST He8/He10 8TB SAS HDDs
• 1x Radeon Pro v340 16GB
• 1x LSI SAS9200-16E HBA

The majority of those are meant for a iSCSI/ZFS build that will be running TrueNAS Scale in the future. However, the shipping company appears to have lost my SSDs. So now I'm sitting here with no SLOG or L2ARC -- just spinning disks with small ARC. This delays the transition away from vSphere RDMs to proper VMDKs (Storage vMotion to iSCSI/ZFS backend) and replacement of the current Artix OpenRC VM with Project ArcZ (BTRFS stopgap). I haven't received the SSDs or a refund yet, because it's still under investigation.

I've managed to work on other tasks in the meantime:

• added ClamAV file scanning (Nextcloud)
• configured proper Redis cache (Nextcloud)
• added LibreSign app (Nextcloud)
• configured video calling (FreePBX)
• compose file draft for Spreed/HPB (Nextcloud)

Haven't properly tested the Radeon Pro v340 with ESXi 6.5u3. The DL380 Gen9 will possibly need more RAM for the amount of storage it'll be managing, so that will be one of my next purchases (in a few months). Once I finish inventorying the DL380 Gen9, I'll add the details to the project configuration.

[TopHatProductio]

USPS seems to have lost the 16x Toshiba THNSNJ256GCSY 256GB SSDs, so I had to replace them. Still waiting on a refund. I'll be testing the DL380 for the first time this weekend, if time permits it.

[TopHatProductio]

​Focus is now split between configuring the Chateau, upgrading macOS, and the configuring the DL380 Gen9. The transition to Project:ArcZ depends on the TrueNAS Scale now. Once the Chateau is ready, all users will connect to WLAN with AD credentials only.

Current ToDo's:
 - Server/Networking:
    - Configure MikroTik Chateau 5G ax for RADIUS authentication
    - DL380 Gen9: Install TrueNAS Scale, install Wazuh agent, setup iSCSI target(s)
 - macOS Mojave:
    - Change unlockers (DrDonk => shanyungyang)
    - Upgrade to Catalina
    - Install *Arr suite
    - OPTIONAL: Test Radeon Pro v340

Upcoming ToDo's:
 - ArcZ0:
    - Clone to ArcZ1:
    - Setup Samba (DomainJoin), Podman + Compose
    - Podman Migration:
        - PortainerBE, NGINX Proxy Manager, Wazuh XDR, ClamAV
        - OnlyOfficeEE, xBrowserSync, RustDesk, IT-tools
 - ArtixServ => ArcZ1:
    - Setup Samba (DomainJoin), LXD
    - LXD Migration:
        - LXD Dashboard, Lxdocker, Nextcloud + Spreed/HPB, OSRM

Long-term ToDo's:
 - Server/Networking:
    - Migrate iLO3 to VLAN 51 (Project vNet)
    - DL580 Gen9 transition
 - FreePBX:
    - Port out Google Voice phone# to VoIP.ms (DID)
        - https://support.google.com/voice/answer/10130510?hl=en
        - https://support.google.com/voice/answer/1065667?hl=en
        - https://wiki.voip.ms/article/Porting_FAQ#How_to_port_my_Google_Voice_Number_to_VoIP.ms

Other Long-Term Tasks:

• https://linustechtips.com/status/337848/
• https://linustechtips.com/status/337856/

[TopHatProductio]

​Updated Task List (09/25/2023):

Current ToDo's:
 - Server/Networking:
    - RADUIS/PEAP on Chateau 5Gax
    - DL380 Gen9: Setup TrueNAS Scale, Wazuh agent, iSCSI target(s)

Upcoming ToDo's:
 - ArcZ0:
    - Clone to ArcZ1:
    - Setup Samba (DomainJoin), Podman + Compose
    - Podman Migration:
        - PortainerBE, NGINX Proxy Manager, Keycloak, Wazuh XDR, ClamAV
        - OnlyOfficeEE, xBrowserSync, RustDesk, IT-tools
 - ArtixServ => ArcZ1:
    - Setup Samba (DomainJoin), LXD
    - LXD Migration:
        - LXD Dashboard, Lxdocker, Nextcloud + Spreed/HPB, OSRM

Long-term ToDo's:
 - Server/Networking:
    - Migrate iLO3 to VLAN 51 (Project vNet)
    - DL580 Gen9 transition
 - FreePBX:
    - Port out Google Voice phone# to VoIP.ms (DID)
        - https://support.google.com/voice/answer/10130510?hl=en
        - https://support.google.com/voice/answer/1065667?hl=en
        - https://wiki.voip.ms/article/Porting_FAQ#How_to_port_my_Google_Voice_Number_to_VoIP.ms

Focus is currently split between configuring the Chateau (RADIUS), configuring the DL380 Gen9 (TrueNAS Scale), and researching Incus. The move to Project:ArcZ depends on the TrueNAS Scale. The release of Incus (LXD fork) is forcing me to consider delaying the move to LXD.

Other Long-Term Tasks:

• https://linustechtips.com/status/337848/
• https://linustechtips.com/status/337856/