Where to store system logs when persistent storage is all vSAN?

[drheim]

Hi guys,

I have a 6.7u2 cluster with an error on the hosts that "system logs are stored on non-persistent storage."  ESXi was installed on a flash and when I SSH in and run df, I am only seeing 108MB free on the largest of the local volumes.  Everything else is vSAN.  Where is the best place to store the system logs?  I can also pop in some additional disks to the hosts, but hate to waste large disks for this, and would try to partition them for logging/vSAN. I do not like extending the vSAN across extra disks when they are not needed as not sure if vSAN is smart enough to consolidate the data down to a less disks, when possible, in the event of a single disk failure. 

https://kb.vmware.com/s/article/2147541

[TheBobkin · Accepted solution]

Hello drheim​

"Where is the best place to store the system logs?"

Any persistent VMFS or NFS datastore either locally attached or remote.

If no remote-storage is available (and/or a small local NAS for all the hosts to share is not an option), then that leaves SATADOM, HDDs or SSDs attached locally to each host as viable options.

If you are using HDD/SSD on the same controller as the vSAN devices then ensure they are using the same access mode and other caveats are noted:

VMware Knowledge Base

VMware Knowledge Base

"I can also pop in some additional disks to the hosts, but hate to waste large disks for this, and would try to partition them for logging/vSAN."

Using disks for vSAN requires them to have no partitions on them so this won't be possible.

"I do not like extending the vSAN across extra disks when they are not needed as not sure if vSAN is smart enough to consolidate the data down to a less disks, when possible, in the event of a single disk failure."

Please clarify what you mean here - you linked to a kb stating not to log to vsanDatastore which you of course shouldn't do.

Bob

[drheim]

So I guess I have 2 issues

1 - Finding persistent disk to store system logs as everything is currently going to vSAN.  Guessing I will probably try and set up some network attached NFS

2 - Adding additional disks to current vSAN

For 2nd issue, If I have a 1TB of used space(total of all vmdks) on 2TB vSAN, and I add an additional disk to each node to increase vSAN to 4TB, will a host go down if 1 host loses one disk and goes back down to 2TB?  In other words, will the vSAN be smart enough to stop striping across the additional disk, when 1 disk fails, and move all of the data to a single disk on one of the nodes(when the data will fit) in order to continue to keep both nodes online while the failed disk is waiting for replacement? I do not want to fail down to a single node unless it is absolutely critical.  If the answer of this is yes, I might try to add additional disks to each node and try to partition some of the space for system logs, and the rest for vSAN(if possible)

Thanks,

Dan

[drheim]

That 2nd URL you sent over above is extremely helpful.   Looking through it now.

[TheBobkin]

Hello Dan,

While unrelated and *should* probably be a separate thread so as not to cause confusion, regarding your 2nd question:

"If I have a 1TB of used space(total of all vmdks) on 2TB vSAN, and I add an additional disk to each node to increase vSAN to 4TB, will a host go down if 1 host loses one disk and goes back down to 2TB?  In other words, will the vSAN be smart enough to stop striping across the additional disk, when 1 disk fails, and move all of the data to a single disk on one of the nodes(when the data will fit) in order to continue to keep both nodes online while the failed disk is waiting for replacement?"

No, a failed disk means you have a failed disk and will mean this storage space and the copy of the data on this will not be available until it has been replaced+resynced or otherwise fixed.

Assuming you are running an FTT=1 Storage Policy, the data will remain available (though now FTT=0) from the other nodes replica and it will be rebuilt back to FTT=1 on disk(s) of the node that lost a disk (provided there is enough available space to do so).

I am a bit confused as to what you mean going from 4TB to 2TB if you lose a disk as what you said indicates you would be adding 2x1TB Capacity-tier, one to each node (e.g. you would have 2TB on one node and 1TB on the other node after you lose a disk).

Either way - provided you have a complete replica of the data (as it is by default in a 2+1 or 1+1+1 cluster) and the Witness then the data will remain available (though FTT=0) if you lost all the storage on one node.

"I might try to add additional disks to each node and try to partition some of the space for system logs, and the rest for vSAN(if possible)"

As I said above, using disks for vSAN requires them to have no partitions on them so this won't be possible.

Bob

[drheim]

Thank-you for this response.  It was perfect. I hate adding disks before they are really needed because to me it just increases the chance that one thing breaks to knock one node completely offline.  Especially when these are HDD and not SSD. I had read somewhere else that if you have a 2 node vSan cluster with FTT=1 and you lose a disk on one of those nodes, but the data will still fit on the remaining working disks it would somehow keep the node online(FTT=1) even though there is a bad disk.  Thank-you for confirming that is not the case.

[TheBobkin]

Hello Dan,

"I had read somewhere else that if you have a 2 node vSan cluster with FTT=1 and you lose a disk on one of those nodes, but the data will still fit on the remaining working disks it would somehow keep the node online(FTT=1) even though there is a bad disk.  Thank-you for confirming that is not the case."

Just to clarify what I said above - if for instance you have just a single failed Capacity-tier device, it WILL keep the node online and it WILL try to repair the data back to FTT=1 (or whatever the Storage Policy has defined) resyncing the data on that node if there is adequate available space. However, it won't try to resync the data back to compliance onto the failed disk as this will have been marked as offline or otherwise failed.

Bob