VMware Cloud Community
TopHatProductio
Hot Shot
Hot Shot

New Server Project

Hello! It's been a while since I last posted here with my own topic. I now have a dedicated ESXi server in the works. The server project is meant to replace (and exceed) my previous workstation - a Dell Precision T7500. Here are the specs for the hardware:

 

HPE ProLiant DL580 G7

 

 

    OS   :: VMware ESXi 6.5u3 Enterprise Plus
    CPU  :: 4x Intel Xeon E7-8870's (10c/20t each; 40c/80t total)
    RAM  :: 256GB (64x4GB) PC3-10600R DDR3-1333 ECC
    PCIe :: 1x HP 512843-001/591196-001 System I/O board + 
                1x HP 588137-B21; 591205-001/591204-001 PCIe Riser board
    GPU  :: 1x nVIDIA GeForce GTX Titan Xp +
                1x AMD FirePro S9300 x2 (2x "AMD Radeon Fury X's")
    SFX  :: 1x Creative Sound Blaster Audigy Rx
    NIC  :: 1x HPE NC524SFP (489892-B21) +
                2x Silicom PE310G4SPI9L-XR-CX3's
    STR  :: 1x HP Smart Array P410i Controller (integrated) +
                1x HGST HUSMM8040ASS200 MLC 400GB SSD (ESXi, vCenter Appliance, ISOs) + 
                4x HP 507127-B21 300GB HDDs (ESXi guest datastores) +
                1x Western Digital WD Blue 3D NAND 500GB SSD + 
                1x Intel 320 Series SSDSA2CW600G3 600GB SSD +
                1x Seagate Video ST500VT003 500GB HDD
    STR  :: 1x LSI SAS 9201-16e HBA SAS card +
                1x Mini-SAS SFF-8088 cable + 
                        1x Dell EMC KTN-STL3 (15x 3.5in HDD enclosure) + 
                                4x HITACHI Ultrastar HUH728080AL4205 8TB HDDs +
                                4x IBM Storewise XIV v7000 98Y3241 4TB HDDs
    I/O  :: 1x Inateck KU8212 (USB 3.2) +
                1x Logitech K845 (Cherry MX Blue) +
                1x Dell MS819 Wired Mouse
            1x Sonnet Allegro USB3-PRO-4P10-E (USB 3.X) +
                1x LG WH16NS40 BD-RE ODD
    PRP  :: 1x Samsung ViewFinity S70A UHD 32" (S32A700)
            1x Sony Optiarc BluRay drive
    PSU  :: 4x HP 1200W PSUs (441830-001/438203-001)

 

 


The details for the ProLiant DL380 Gen9 will appear here once data migration is complete. VMware Horizon (VDI) will have to wait for a future phase (if implemented at all). The current state of self-hosted VDI is Windows-centric, with second-hand support for Linux and no proper support for macOS.

The planned software/VM configurations have been moved back to the LTT post, and will be changing often for the foreseeable future.

Product links and details can be found here.

 

ESXi itself is usually run from a USB thumb drive, but I have a drive dedicated to it. No harm done. A small amount of thin provisioning/overbooking (RAM only) won’t hurt. macOS and Linux would have gotten a Radeon/FirePro (ie., Rx Vega 64), for best compatibility and stability, but market forces originally prevented this. Windows 10 gets the Audigy Rx and a Titan Xp. The macOS and Linux VMs get whatever audio the Titan Z FirePro S9300 x2 can provide. The whole purpose of Nextcloud is to phase out the use of Google Drive/Photos, iCloud, Box.com, and other externally-hosted cloud services (Mega can stay though).

 

There are three other mirrors for this project, in case you're interested in following individual conversations from the other sites (in addition to this thread).

 

P.S. Out of all the sites that I've ever used, this forum has one of the best WYSIWYG editors I've used in a while Smiley Happy

Kudos to the devs!

Tags (1)
258 Replies
TopHatProductio
Hot Shot
Hot Shot

Now to address the SSD issue in the background, while I look at security patches and initial VM setup.

Reply
0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Currently trying to make a new datastore via SSH. More info here:

Reply
0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Reply
0 Kudos
TopHatProductio
Hot Shot
Hot Shot

While I'm waiting on comments for the previous issue, and a few ISOs to upload to my server, I can start working on investigating this:

Gotta search for the patches through this page, by entering the details mentioned in the last 3 kb pages:

On a side note, also ran into this when setting up my first VM:

Reserve Memory beforehand, I guess

Reply
0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Just finished installing the vCenter Appliance, and will be using the FLEX (Flash web) client to setup the SSD for virtual flash in a bit. Stay tuned Smiley Happy One step closer...

TXP-Network Does :: ESXi Test Stream! - YouTube

Reply
0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Solved the SSD/Virtual Flash issue! Now onto the next one​​​​​​​...

Reply
0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Removed the HP 491838-001 (NC375i)​​​​​​​https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-c01951393 due to space constraints, increased RAM to 128GB, purchased 4TB HDDs to replace the 2TB HUA722020ALA330's, and delaying the addition of the SolarFlare NIC.

Reply
0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Currently working on DNS, after which I'll focus on setting up the first VPN solution - SoftEther.

Reply
0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Reply
0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Just got a new GPU in the mail, which may end up replacing the GTX 1060 6GB. Still troubleshooting this issue...

Reply
0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Reply
0 Kudos
TopHatProductio
Hot Shot
Hot Shot

The K80's are coming...

Reply
0 Kudos
TopHatProductio
Hot Shot
Hot Shot

I've got a GRID K520 coming in the mail in about 2 weeks, to replace the Tesla K10. Perhaps I can buy a GRID K2 in the near future, so that I can have all three of the major variants for this card. GRID K520 looks like a GeForce card from inside a VM, if my memory isn't failing me. GRID K2 would be the Quadro variant. Tesla K10 is a pure compute version. I wonder if anything like that exists for Tesla K80...

Reply
0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Just solved another looming issue for the server project. Now to get that SSD working and added to the Virtual Flash resource pool...

TXP-Network Does :: ESXi Server - HBA Storage Array Update! - YouTube

Reply
0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Time for a long-overdue project update. Omitting a lot of steps/details here, for relative brevity. A friend of mine, from Discord (the same one who was kind enough to help me troubleshoot the many of the issues I encountered), had me run a Linux LiveCD on the server to troubleshoot the LSI HBA. For those of you who did not know, the LSI HBA wasn’t working as expected until a few hours ago (late last night). I tested it in my current workstation (Precision T7500 - Windows 10), the server (DL580 G7 – ESXi 6.5u3), and even on my laptop (EliteBook 8770w - Windows 10). When tested on the T7500, the HBA showed up – but none of the 4TB hard drives showed up. The same for the laptop and the server. After a bit of Googling (as the cool kids say), I decided that it may behoove me to try flashing it with the IT firmware, to see if that would fix it. I did so from my laptop, by making use of a powered PCIe dock (to prevent further downtime on the T7500 – running a Minecraft server). I did so, using a GUI application called MegaRAID Storage Manager. The HBA was on v17.X, and now it's on v20.X. The drives also appeared in Windows Device Manager for once. However, they didn't stay in Device Manager for long. They popped in and out, sporadically. I was instructed to reboot after the firmware update was applied. MegaRAID Storage Manager stopped being able to connect to the local server after the reboot it said to do, for the firmware update to take hold. That meant that, if the firmware I flashed was the wrong one, I’d have to resort to using sas2flash. After no luck checking on the HBA from my laptop, I decided to put it in the server, with the Linux LiveCD (as mentioned earlier). The Linux LiveCD was running an older build of Manjaro, and managed to see all of the drives in gparted. However, we were unable to get SMART data for most of the HDDs. If you look closely at the HDD models, you may or may not be able to tell why. However, while I was in the LiveCD, I decided to also try GPT scheming the Intel SSD as well, since messing with it in Windows simply did not work for some reason. A short while later, we tried the latest Manjaro LiveCD available (because Manjaro is my preferred distro with sysemd). That one didn’t see the drives at all, but did still see the HBA. At this point, I saw no other way to validate the HDDs further. I made the decision to test them in ESXi and try to pull SMART data from esxcli. The drives showed up in ESXi, and even allowed for us to pull SMART data – but it was limited, in a different format than most common drives on the market. I was able to add the Intel SSD to the Virtual Flash pool for once, though. As such, this is strictly a partial victory. We have the drives ready for use, presumably. But we don’t know how the drives are doing – which is very different from all of my previous experiences, where I could pull up SMART data immediately after installing the drives. The game is afoot.

Reply
0 Kudos
TopHatProductio
Hot Shot
Hot Shot

On a side note, the results of last night's livestreaming attempt are tempting me to make YouPHPTube part of the project again. If this keeps up, I might actually go for it...

Reply
0 Kudos
TopHatProductio
Hot Shot
Hot Shot

Reply
0 Kudos
TopHatProductio
Hot Shot
Hot Shot

ToDo List for the next few days:

  • Figure out Split Horizon DNS records (Technitium)
  • Setup ejabberd and hMailServer
    • FQDNs and subdomains
    • AD/LDAP integrations
  • Setup Artix Linux VM
    • secondary Technitium instance (AD DNS forwarding)
Reply
0 Kudos