Hello all,
First, I'm new to this forum and to VMWare so your kindness is appreciated.
Here's the background:
I have 4 servers in different locations that are running ESXi 6.0, two are Dell R220, one HP DL 380 G9, and one is HP DL 160 G9.
The servers were built a few years ago by a previous VoIP vendor and are hosting ShoreTel servers and virtual switches.
The problem is that the vendor selected too small of hard drives and there isn't enough space for snapshots.
We have QNAPs available in each location.
I was wondering if it's possible to use them as storage locations for snapshots?
If so, can I be guide how to do that?
I tried consulting with QNAP support but they weren't helpful.
If more information is required, I will gladly provide it.
Thanks in advance.
Hi and welcome. When you say "snapshots" are you talking about a literal VM snapshot, or are you referring to an image-based backup of those VMs? If you mean actual VM snapshots, what are you trying to do, exactly? I ask because generally telephony systems don't take kindly to being snapshotted (and many vendors don't support it anyway), that plus snapshots absolutely should not be left hanging around on VMs. That gets folks into all sorts of trouble. If you can explain a bit more about what you're trying to accomplish, it'll better put into context what you have any why it's a problem.
When virtual machine snapshot is taken, there are files in the folder in its datastore. If you want to take the virtual machine's snapshots to a different location, you can check the KB below.
But I was wondering why you want to do something like this.
If you want to take the virtual machine's snapshots to a different location, you can check the KB below.
...HOWEVER before you even think of doing that, please better explain what you're thinking you want to do. Because if you indeed want or think you need to do as is outlined in the KB by tayfundeger then you are going to put yourself in a very precarious situation (ESXi 6.0 being out of support, you probably don't have support anyway, you're a newbie to VMware technology, and you want to put snapshots on a consumer-grade NAS == recipe for disaster). That's the reason I didn't lead with step-by-step instructions on how to shoot yourself in the foot in the first place.
I absolutely agree with daphnissov.
Snapshots are by no means replacement for backups! Snapshots on production VMs should be avoided whenever possible.
Even though that KB article mentioned by tayfundeger is correct regarding the relocation for snapshots, this is definitely not what you want to do in this case.
If disk space is low then either check if you can free up disk space (e.g. by deleting unnecessary snapshots), or add more storage.
Please take a look at e.g. https://kb.vmware.com/s/article/1015180 which explains how snapshots work. This will explain why it's a not a good idea to keen snapshots for a long time, and also why - if needed - they should be stored on the same tier.
André
Thank you everyone for your quick replies.
So as you mentioned, rather than trying to describe “the solution”, I will describe the issue (which I should have done in the first place – my apologies)
As mentioned earlier, there are 4 ESXi 6.0 servers.
Our vendor needs to perform upgrades to the VoIP applications on these servers.
I need a rollback method in case something goes wrong. The vendor recommended on taking snapshots.
I was under the impression that snapshots (which I’m aware are not a replacement for backups) are something like a “system restore” to allow a quick rollback.
Since we have storage limitations (and performing storage upgrade is something we cannot do now for various reasons but will in the future), snapshots cannot be taken.
That is why I mentioned QNAPs, but I understand that this is not recommended.
With the storage issue described above what would be a good practice that is not too complicated to be able to rollback if needed?
Again, the reason I mentioned QNAP is because it is advertised as VMWare compatible.
With that said, my intention is not to use for hosting. If not for snapshots, then perhaps to make full copies of VMs assuming it’s one of legit methods and if so, how?
Other ideas are always appreciated.
- Dror
This is a much more complete picture of your problem statement and what you're trying to accomplish, so thanks for that. At the end of the day, you're going to require storage no matter what, whether that be for snapshots or for backups (which leverage snapshots to perform them). Let's get to your statements/questions:
I need a rollback method in case something goes wrong. The vendor recommended on taking snapshots.
I was under the impression that snapshots (which I’m aware are not a replacement for backups) are something like a “system restore” to allow a quick rollback.
Yes, that is so. However, the danger this gets people into is when they're taken, forgotten about, and they blow up a datastore. It's easy to abuse this technology, and many do.
Since we have storage limitations (and performing storage upgrade is something we cannot do now for various reasons but will in the future), snapshots cannot be taken.
On this point, when you say "snapshots cannot be taken" is this something that literally will not work, or is this something you don't *think* will work? Is this limitation lack of free space on the datastores housing the VMs? If so, how much are we talking here? I ask because maybe it is possible to snapshot them in a strategic way, either to give you that one-time backup before this project starts, or to give you enough of a safety net for a rolling upgrade process to occur. Basically anything that'll accomplish your upgrade without having to complicate things further by diverting snapshots to another location.
With the storage issue described above what would be a good practice that is not too complicated to be able to rollback if needed?
Here's what I'd recommend:
The long and short of it is this: You need to be able to perform snapshots. This is a normal part of operating a vSphere environment and they form the basis of VM-based backups. If you can't perform them, you immediately have a problem to address. Farming those snapshots out to a separate storage system because you don't have capacity on the primary one should be the absolute last resort. If it becomes the last resort, it should be a stop-gap solution until you address your primary storage concerns.
Personal opinion:
You can store some of your illegal MP3 on a consumer NAS like QNAP and the others, but not your production VMs. Yes i´am aware that they have some higher end models as well.
Snaps are part of the VM and on vSphere ESXi a VM snapshot contains your running data which needs the best performance and data security you can deliver.
You should temporarily move some VMs which a offline or your Templates/ISO from the local datastore to the QNAP.
Some thoughs:
Regards,
Joerg
Thank you daphnissov. I'll answer your questions inline the best I can:
Explain your limitation. Is this a perceived limitation or an actual one? How are you measuring it? What have you tried or not tried?
As I mentioned, there are 4 servers but I will start with the most challenging one.
The server was built per a previous vendor's guidelines and approved by them to be used as a ShoreTel HQ server.
Then a new vendor suggested to rebuild that server as a VM. They did it for us.
The server has a total capacity of 148.5 GB.
It has ESXi 6.0 installed on it with one guest that the HQ server was rebuilt on (running Win 2012 R2).
Datastore1 capacity is 141 GB where free space is 18.5 GB.
Provisioned storage size is 116.17 GB
I gathered that information from the client app. I can attach screenshots if needed.
I have not tried anything yet.
There is another issue with storage but it has to do with the server running on that guest where the software upgrade requires a minimum of 40 GB free space.
The vendor configured the server with 2 drives, drive 1 with 80 GB and drive 2 with 20 GB. I can remove drive 2 + the 18.5 GB mentioned above, and with that I have 38 GB to add to drive 1.
Based on the limitation, identify whether you need a separate storage device or not.
My guess is that there is a need for a separate storage solution based on the info above. Is that a fair guess?
Perform in-guest, file-based or config-based backup of needed items to restore if all else fails.
I will need more guidance here.
We backup daily the ShoreTel server database which can be used to rebuild the server if the worst happens, of course we want to avoid that.
If I have to use a 3rd party solution, what would be a good option (meaning, that it's not only backing up, but can also successfully recover...)?
Hey Joerg, I think you answered a few of my questions on Dell's forum.
I have a question in regards to snapshots.
You wrote that snapshots contains running data.
With that said, one of my servers has disk's properties grayed out and max size in disk provisioning showing N/A so I can't expand its hard disk space.
I searched and found out that in order to resolve this issue I need to remove snapshots.
I did find out in Snapshot manager that there was one taken by the previous vendor before a ShoreTel server upgrade.
Is it safe to remove that snapshot?
Here's the link to where it says that the solution is to remove the snapshot:
Yes a greyed out vDisks size field is always an strong indicator that the vDisk is running on an snapshot. You have to look to he vmdk filename to be 100% sure because when using SATA controller for the VM its also greyed out when the VM is on. So check the filename.
For deleting a Snapshot you need free space! How much depends on ESXi version and of course the size of the snapshot. Please run ls -lisa and df -h from the command line.
Regards,
Joerg
The ls -lisah command needs to be executed in the VM folder and not on / level. Also resize your shell window so that there are no line breakes, A screenshot is not needed because you can c&p the output directly into your post.
Regards,
Joerg
First, thank you so much for your patience and guidance.
It looks like I have snapshots on two machines only.
Here's what I got:
ShoreTel-ENC-DVS1
[root@ESXi-Encino:/vmfs/volumes/56a6b74e-cffcbcd4-4c2a-9457a5527304/ShoreTel-ENC-DVS1] ls -lisa
total 280505376
8429188 8 drwxr-xr-x 1 root root 3220 Jan 13 22:34 .
4 1024 drwxr-xr-t 1 root root 1960 Mar 18 2016 ..
520134276 37028864 -rw------- 1 root root 37916921856 May 13 18:55 ShoreTel-ENC-DVS1-000001-delta.vmdk
524328580 0 -rw------- 1 root root 338 Jan 13 22:34 ShoreTel-ENC-DVS1-000001.vmdk
515939972 16777216 -rw------- 1 root root 17179869184 Nov 4 2016 ShoreTel-ENC-DVS1-Snapshot1.vmem
511745668 2048 -rw------- 1 root root 1301090 Nov 4 2016 ShoreTel-ENC-DVS1-Snapshot1.vmsn
448831108 16777216 -rw------- 1 root root 17179869184 Jan 13 22:34 ShoreTel-ENC-DVS1-e78a5214.vswp
16817796 209715200 -rw------- 1 root root 214748364800 Nov 4 2016 ShoreTel-ENC-DVS1-flat.vmdk
50372228 1024 -rw------- 1 root root 8684 May 12 20:08 ShoreTel-ENC-DVS1.nvram
21012100 0 -rw------- 1 root root 507 Oct 20 2016 ShoreTel-ENC-DVS1.vmdk
25206404 0 -rw-r--r-- 1 root root 479 Nov 4 2016 ShoreTel-ENC-DVS1.vmsd
12623492 8 -rwxr-xr-x 1 root root 2998 Jan 13 22:34 ShoreTel-ENC-DVS1.vmx
436248196 0 -rw------- 1 root root 0 Jan 13 22:34 ShoreTel-ENC-DVS1.vmx.lck
54566532 8 -rw------- 1 root root 3864 Jan 26 2016 ShoreTel-ENC-DVS1.vmxf
440442500 8 -rwxr-xr-x 1 root root 2996 Jan 13 22:34 ShoreTel-ENC-DVS1.vmx~
163618436 1024 -rw-r--r-- 1 root root 303285 Nov 24 2018 vmware-13.log
234921604 1024 -rw-r--r-- 1 root root 269420 Nov 24 2018 vmware-14.log
272670340 1024 -rw-r--r-- 1 root root 329779 Dec 6 2018 vmware-15.log
339779204 1024 -rw-r--r-- 1 root root 350954 Dec 24 2018 vmware-16.log
394305156 1024 -rw-r--r-- 1 root root 990171 Jun 1 2019 vmware-17.log
415276676 1024 -rw-r--r-- 1 root root 963882 Jan 13 21:09 vmware-18.log
444636804 1024 -rw-r--r-- 1 root root 661578 May 13 15:14 vmware.log
432053892 195584 -rw------- 1 root root 200278016 Jan 13 22:34 vmx-ShoreTel-ENC-DVS1-3884601876-1.vswp
[root@ESXi-Encino:/vmfs/volumes/56a6b74e-cffcbcd4-4c2a-9457a5527304/ShoreTel-ENC-DVS1] df -h
Filesystem Size Used Available Use% Mounted on
VMFS-5 458.2G 375.7G 82.5G 82% /vmfs/volumes/datastore1
vfat 4.0G 23.6M 4.0G 1% /vmfs/volumes/56a6b74f-ea979f22-3bc9-9457a5527304
vfat 249.7M 183.5M 66.2M 74% /vmfs/volumes/77d56276-6242e12b-3bb3-5e40eb924faa
vfat 249.7M 8.0K 249.7M 0% /vmfs/volumes/55016d7b-9fdca45f-9666-00de496181c6
vfat 285.8M 208.1M 77.7M 73% /vmfs/volumes/56a6b74e-87f7a1c2-b0f7-9457a5527304
[root@ESXi-Encino:/vmfs/volumes/56a6b74e-cffcbcd4-4c2a-9457a5527304/ShoreTel-ENC-DVS1]
ShoreTel-WLA-DVS1
[root@ESXi-WLA:/vmfs/volumes/56a62e3a-f12aed25-f905-5820b1068e18/ShoreTel-WLA-DVS1] ls -lisa
total 168979488
8403204 8 drwxr-xr-x 1 root root 3220 Nov 5 2019 .
4 1024 drwxr-xr-t 1 root root 1960 Nov 3 2016 ..
696269060 26166272 -rw------- 1 root root 26793463808 May 13 18:47 ShoreTel-WLA-DVS1-000001-delta.vmdk
700463364 0 -rw------- 1 root root 338 Nov 5 2019 ShoreTel-WLA-DVS1-000001.vmdk
692074756 8388608 -rw------- 1 root root 8589934592 Nov 4 2016 ShoreTel-WLA-DVS1-Snapshot1.vmem
687880452 2048 -rw------- 1 root root 1294078 Nov 4 2016 ShoreTel-WLA-DVS1-Snapshot1.vmsn
494942468 8388608 -rw------- 1 root root 8589934592 Jul 21 2019 ShoreTel-WLA-DVS1-eaac7caa.vswp
16791812 125829120 -rw------- 1 root root 128849018880 Nov 4 2016 ShoreTel-WLA-DVS1-flat.vmdk
50346244 1024 -rw------- 1 root root 8684 May 12 19:25 ShoreTel-WLA-DVS1.nvram
20986116 0 -rw------- 1 root root 507 Jul 9 2016 ShoreTel-WLA-DVS1.vmdk
25180420 0 -rw-r--r-- 1 root root 481 Nov 4 2016 ShoreTel-WLA-DVS1.vmsd
12597508 8 -rwxr-xr-x 1 root root 2740 Nov 5 2019 ShoreTel-WLA-DVS1.vmx
482359556 0 -rw------- 1 root root 0 Jul 21 2019 ShoreTel-WLA-DVS1.vmx.lck
75512068 8 -rw------- 1 root root 3864 Jan 25 2016 ShoreTel-WLA-DVS1.vmxf
562051332 8 -rwxr-xr-x 1 root root 2740 Nov 5 2019 ShoreTel-WLA-DVS1.vmx~
109066500 1024 -rw-r--r-- 1 root root 908206 Dec 15 2018 vmware-25.log
184563972 1024 -rw-r--r-- 1 root root 306515 Dec 24 2018 vmware-26.log
251672836 1024 -rw-r--r-- 1 root root 273116 Dec 24 2018 vmware-27.log
339753220 1024 -rw-r--r-- 1 root root 502491 Mar 25 2019 vmware-28.log
440416516 1024 -rw-r--r-- 1 root root 902540 Jul 20 2019 vmware-29.log
490748164 1024 -rw-r--r-- 1 root root 614289 Nov 4 2019 vmware-30.log
566245636 1024 -rw-r--r-- 1 root root 824368 May 13 14:32 vmware.log
478165252 195584 -rw------- 1 root root 200278016 Jul 21 2019 vmx-ShoreTel-WLA-DVS1-3937172650-1.vswp
[root@ESXi-WLA:/vmfs/volumes/56a62e3a-f12aed25-f905-5820b1068e18/ShoreTel-WLA-DVS1] df -h
Filesystem Size Used Available Use% Mounted on
VMFS-5 271.8G 233.7G 38.0G 86% /vmfs/volumes/datastore1
vfat 285.8M 208.1M 77.7M 73% /vmfs/volumes/56a62e3a-c10256a5-dcda-5820b1068e18
vfat 4.0G 17.9M 4.0G 0% /vmfs/volumes/56a62e3a-fd7e600d-028f-5820b1068e18
vfat 249.7M 183.5M 66.2M 74% /vmfs/volumes/a37c227f-0ef7f916-59c1-edbe21b399ae
vfat 249.7M 8.0K 249.7M 0% /vmfs/volumes/11a27e75-aa4e1c8d-ffd8-2ec23eefaa07
[root@ESXi-WLA:/vmfs/volumes/56a62e3a-f12aed25-f905-5820b1068e18/ShoreTel-WLA-DVS1]
It looks like someone created snapshots in Nov 2016, and forgot about them!?
Fortunately the VMs' have thick provisioned virtual disks, and you've got free disk space on the datastores, so there shouldn't be any issues with deleting the snapshots. This may take quite some time due to their sizes, so please remain patient. Deleting the snapshots will free up ~50GB on ShoreTel-ENC-DVS1, and ~30GB on ShoreTel-WLA-DVS1, which gives you sufficient free disk space faor a new snapshot for the upcoming backup.
Two hints:
André
Snapshot age is 3.5Y?!? Congrats.... my personal new record holder
Just press the "delete all" within Snapshot manager. Yes shutting down the VM when possible is a good idea. Please keep notice that you cant start the VM until snapshot is deleted.
Regards,
Joerg
Thank you Andre for your reply, I appreciate the tips.
It was our VoIP vendor who created it before a previous software upgrade.
The previous vendor set it up. Before that, we didn't use VMWare for our VoIP servers and switches and that vendor pushed for it, but it looks like they didn't know exactly what they were doing.
LOL...Thank you?...:smileylaugh:
OK, so I take it that Delete All will consolidate the data from the snapshot and will actually bring the parent VM to the working state the snapshot was in.
I also understand that it is recommended to take snapshots when the VM is powered down. Is it also recommended to Delete all when the VM is shutdown as well, or it doesn't matter in this case?