tl;dr
I have a basic home server with one NVMe drive, multiple SSDs, and multiple (SATA) HDDs, plus multiple USB3 ports and an SD card reader.
What’s the best boot / storage configuration for good performance with a basic degree of resilience? Thank you.
More
First, a disclaimer:
In an ideal world I’d have multiple hosts, for redundancy and failover. I’d also have separate storage: a well-spec’d FreeNAS box with a decent SAS card and a healthy chunk of ECC memory, serving shares via NFS. But at the moment I don’t have those things; I’m trying to do the best I can with what I’ve got.
So:
BOOT
I’ve read some documentation and blog posts. I’ve seen recommendations to boot from SD card, but I’m a bit wary of this: my experience with SBCs (e.g. RPi) is that after a while - when the card starts to run out of write cycles - they can fail in interesting and unpredictable ways. I have experience booting OSes from USB as well (see FreeNAS, above); do you think this would be a better option?
STORAGE
In terms of “speed”: NVMe > SSD > HDD
…So I’m assuming:
SSDs (mirrored): VMs
HDDs (mirrored): other persistent data
NVMe: cache and ephemeral data (etc.)
Does that make sense? Is there a better way?
Thank you for reading 
Apologies for a long post.
P.S. Out-of-scope-but-still-relevant: the server has 32G of (non-ECC) RAM, and 5 x 1G Intel ethernet. My goal is to host no more than three “permanent” VMs (e.g. LAMP/LEMP), plus occasional others for experiments I can’t do - or don’t want to do - in a container / virtual environment. Thanks again.
Accepted Solutions
Welcome to the Community,
Hi a.p., and thank you for such a fast reply
ESXi doesn't require a fast storage for its boot device, because it will run in memory.
Thank you - that confirms what I'd read elsewhere
...Given that, though, might it not be better to boot from SD or USB...?
use the NVMe for test VMs
...because there's no redundancy, you mean?
ESXi only supports Hardware RAID controller[...]
Ah, that's important. (How did I miss that? Oops...)
I don't have any more slots for a RAID card, unfortunately, so that limits my options a bit. One solution would be: mdadm (sw raid) in a lightweight Linux, then run everything on top of that. But that's an extra abstraction layer (etc.), which wasn't the plan: the idea was to run VMs in a type 1 (bare metal) hypervisor. Hmmmm...
HDD RAID:
LUN1 = 200GB for ESXi, and a small datastore for e.g. installation/ISO images
LUN2 = remaining space used as a datastore
SSD RAID:
LUN1 = all space used as a datastore
That's really interesting, and it turns my scheme on its head a bit :smileyblush:
Thanks again for the information and recommendations - very helpful!
scoobz