DrorAmbar
Enthusiast
Enthusiast

ESXi 6.0, and using QNAP for snapshots storage

Hello all,

First, I'm new to this forum and to VMWare so your kindness is appreciated.

Here's the background:

I have 4 servers in different locations that are running ESXi 6.0, two are Dell R220, one HP DL 380 G9, and one is HP DL 160 G9.

The servers were built a few years ago by a previous VoIP vendor and are hosting ShoreTel servers and virtual switches.

The problem is that the vendor selected too small of hard drives and there isn't enough space for snapshots.

We have QNAPs available in each location.

I was wondering if it's possible to use them as storage locations for snapshots?

If so, can I be guide how to do that?

I tried consulting with QNAP support but they weren't helpful.

If more information is required, I will gladly provide it.

Thanks in advance.

31 Replies
daphnissov
Immortal
Immortal

Hi and welcome. When you say "snapshots" are you talking about a literal VM snapshot, or are you referring to an image-based backup of those VMs? If you mean actual VM snapshots, what are you trying to do, exactly? I ask because generally telephony systems don't take kindly to being snapshotted (and many vendors don't support it anyway), that plus snapshots absolutely should not be left hanging around on VMs. That gets folks into all sorts of trouble. If you can explain a bit more about what you're trying to accomplish, it'll better put into context what you have any why it's a problem.

tayfundeger
Hot Shot
Hot Shot

When virtual machine snapshot is taken, there are files in the folder in its datastore. If you want to take the virtual machine's snapshots to a different location, you can check the KB below.

But I was wondering why you want to do something like this.

VMware Knowledge Base

--
Blog: https://www.tayfundeger.com
Twitter: https://www.twitter.com/tayfundeger

vBlogger, vExpert, Cisco Champions

Please, if this solution helped your problem, "Helpful" if it solves your problem "Correct Answer" to mark.
0 Kudos
daphnissov
Immortal
Immortal

If you want to take the virtual machine's snapshots to a different location, you can check the KB below.

...HOWEVER before you even think of doing that, please better explain what you're thinking you want to do. Because if you indeed want or think you need to do as is outlined in the KB by tayfundeger​ then you are going to put yourself in a very precarious situation (ESXi 6.0 being out of support, you probably don't have support anyway, you're a newbie to VMware technology, and you want to put snapshots on a consumer-grade NAS == recipe for disaster). That's the reason I didn't lead with step-by-step instructions on how to shoot yourself in the foot in the first place.

0 Kudos
a_p_
Leadership
Leadership

I absolutely agree with daphnissov​.

Snapshots are by no means replacement for backups! Snapshots on production VMs should be avoided whenever possible.

Even though that KB article mentioned by tayfundeger​ is correct regarding the relocation for snapshots, this is definitely not what you want to do in this case.

If disk space is low then either check if you can free up disk space (e.g. by deleting unnecessary snapshots), or add more storage.

Please take a look at e.g. https://kb.vmware.com/s/article/1015180​ which explains how snapshots work. This will explain why it's a not a good idea to keen snapshots for a long time, and also why - if needed - they should be stored on the same tier.

André

0 Kudos
DrorAmbar
Enthusiast
Enthusiast

Thank you everyone for your quick replies.

So as you mentioned, rather than trying to describe “the solution”, I will describe the issue (which I should have done in the first place – my apologies)

As mentioned earlier, there are 4 ESXi 6.0 servers.

Our vendor needs to perform upgrades to the VoIP applications on these servers.

I need a rollback method in case something goes wrong. The vendor recommended on taking snapshots.

I was under the impression that snapshots (which I’m aware are not a replacement for backups) are something like a “system restore” to allow a quick rollback.

Since we have storage limitations (and performing storage upgrade is something we cannot do now for various reasons but will in the future), snapshots cannot be taken.

That is why I mentioned QNAPs, but I understand that this is not recommended.

With the storage issue described above what would be a good practice that is not too complicated to be able to rollback if needed?

Again, the reason I mentioned QNAP is because it is advertised as VMWare compatible.

With that said, my intention is not to use for hosting. If not for snapshots, then perhaps to make full copies of VMs assuming it’s one of legit methods and if so, how?

Other ideas are always appreciated.

- Dror

0 Kudos
daphnissov
Immortal
Immortal

This is a much more complete picture of your problem statement and what you're trying to accomplish, so thanks for that. At the end of the day, you're going to require storage no matter what, whether that be for snapshots or for backups (which leverage snapshots to perform them). Let's get to your statements/questions:

I need a rollback method in case something goes wrong. The vendor recommended on taking snapshots.

I was under the impression that snapshots (which I’m aware are not a replacement for backups) are something like a “system restore” to allow a quick rollback.

Yes, that is so. However, the danger this gets people into is when they're taken, forgotten about, and they blow up a datastore. It's easy to abuse this technology, and many do.

Since we have storage limitations (and performing storage upgrade is something we cannot do now for various reasons but will in the future), snapshots cannot be taken.

On this point, when you say "snapshots cannot be taken" is this something that literally will not work, or is this something you don't *think* will work? Is this limitation lack of free space on the datastores housing the VMs? If so, how much are we talking here? I ask because maybe it is possible to snapshot them in a strategic way, either to give you that one-time backup before this project starts, or to give you enough of a safety net for a rolling upgrade process to occur. Basically anything that'll accomplish your upgrade without having to complicate things further by diverting snapshots to another location.

With the storage issue described above what would be a good practice that is not too complicated to be able to rollback if needed?

Here's what I'd recommend:

  1. Explain your limitation. Is this a perceived limitation or an actual one? How are you measuring it? What have you tried or not tried?
  2. Based on the limitation, identify whether you need a separate storage device or not.
  3. Perform in-guest, file-based or config-based backup of needed items to restore if all else fails.

The long and short of it is this:  You need to be able to perform snapshots. This is a normal part of operating a vSphere environment and they form the basis of VM-based backups. If you can't perform them, you immediately have a problem to address. Farming those snapshots out to a separate storage system because you don't have capacity on the primary one should be the absolute last resort. If it becomes the last resort, it should be a stop-gap solution until you address your primary storage concerns.

IRIX201110141
Champion
Champion

Personal opinion:

You can store some of your illegal MP3 on a consumer NAS like QNAP and the others, but not your production VMs. Yes i´am aware that they have some higher end models as well.

Snaps are part of the VM and on vSphere ESXi a VM snapshot contains your running data which needs the best performance and data security you can deliver.

You should temporarily move some VMs which a offline or your Templates/ISO from the  local datastore to the QNAP.

Some thoughs:

  • Apply the Upgrade packages source through a ISO or additional vDisks which can be stored on the QNAP. This actions have to be done prior to the creation of the Snapshot!
  • A snapshot is a thinprovisioning file. Consume the installation of your application 1Gb of data i expect that the snap increased by that amount of data. If there is data which needs to be converted than it depends....
  • Yes, there is a adv. VM setting to tell the VM to create the snapshot on another location
  • Why not create a copy of your VM, storing them on the QNAP, and test the upgrade there?
  • How do you run your VM Backups?
  • Support of ESXi 6.0 runs out last month

Regards,
Joerg

DrorAmbar
Enthusiast
Enthusiast

Thank you daphnissov. I'll answer your questions inline the best I can:

Explain your limitation. Is this a perceived limitation or an actual one? How are you measuring it? What have you tried or not tried?

As I mentioned, there are 4 servers but I will start with the most challenging one.

The server was built per a previous vendor's guidelines and approved by them  to be used as a ShoreTel HQ server.

Then a new vendor suggested to rebuild that server as a VM. They did it for us.

The server has a total capacity of 148.5 GB.

It has ESXi 6.0 installed on it with one guest that the HQ server was rebuilt on (running Win 2012 R2).

Datastore1 capacity is 141 GB where free space is 18.5 GB.

Provisioned storage size is 116.17 GB

I gathered that information from the client app. I can attach screenshots if needed.

I have not tried anything yet.

There is another issue with storage but it has to do with the server running on that guest where the software upgrade requires a minimum of 40 GB free space.

The vendor configured the server with 2 drives, drive 1 with 80 GB and drive 2 with 20 GB. I can remove drive 2 + the 18.5 GB mentioned above, and with that I have 38 GB to add to drive 1.

Based on the limitation, identify whether you need a separate storage device or not.

My guess is that there is a need for a separate storage solution based on the info above. Is that a fair guess?

Perform in-guest, file-based or config-based backup of needed items to restore if all else fails.

I will need more guidance here.

We backup daily the ShoreTel server database which can be used to rebuild the server if the worst happens, of course we want to avoid that.

If I have to use a 3rd party solution, what would be a good option (meaning, that it's not only backing up, but can also successfully recover...)?

0 Kudos
DrorAmbar
Enthusiast
Enthusiast

Hey Joerg, I think you answered a few of my questions on Dell's forum.

I have a question in regards to snapshots.

You wrote that snapshots contains running data.

With that said, one of my servers has disk's properties grayed out and max size in disk provisioning showing N/A so I can't expand its hard disk space.

I searched and found out that in order to resolve this issue I need to remove snapshots.

I did find out in Snapshot manager that there was one taken by the previous vendor before a ShoreTel server upgrade.

Is it safe to remove that snapshot?

Here's the link to where it says that the solution is to remove the snapshot:

Disk Provisioned Size Greyed Out

0 Kudos
IRIX201110141
Champion
Champion

Yes  a greyed out vDisks size field is always an strong indicator that the vDisk is running on an snapshot. You have to look to he vmdk filename to be 100% sure because when using SATA controller for the VM its also greyed out when the VM is on. So check the filename.

For deleting a Snapshot you need free space! Smiley Happy  How much depends on ESXi version and of course the size of the snapshot.  Please run  ls -lisa  and df -h from the command line.

Regards,

Joerg

0 Kudos
DrorAmbar
Enthusiast
Enthusiast

Hi Joerg,

I took screenshots of the results for the commands you asked me to run.

I hope that this is helpful

0 Kudos
DrorAmbar
Enthusiast
Enthusiast

Hey Joerg, there are actually 3 more servers so here are the results of the 2 others.

0 Kudos
DrorAmbar
Enthusiast
Enthusiast

And this is the last one.

0 Kudos
IRIX201110141
Champion
Champion

The ls -lisah  command needs to be executed in  the VM folder and not on / level.  Also resize your shell window so that there are no line breakes, A screenshot is not needed because you can c&p the output directly into your post.

Regards,
Joerg

0 Kudos
DrorAmbar
Enthusiast
Enthusiast

First, thank you so much for your patience and guidance.

It looks like I have snapshots on two machines only.

Here's what I got:

ShoreTel-ENC-DVS1

[root@ESXi-Encino:/vmfs/volumes/56a6b74e-cffcbcd4-4c2a-9457a5527304/ShoreTel-ENC-DVS1] ls -lisa

total 280505376

8429188      8 drwxr-xr-x    1 root     root          3220 Jan 13 22:34 .

      4   1024 drwxr-xr-t    1 root     root          1960 Mar 18  2016 ..

520134276 37028864 -rw-------    1 root     root     37916921856 May 13 18:55 ShoreTel-ENC-DVS1-000001-delta.vmdk

524328580      0 -rw-------    1 root     root           338 Jan 13 22:34 ShoreTel-ENC-DVS1-000001.vmdk

515939972 16777216 -rw-------    1 root     root     17179869184 Nov  4  2016 ShoreTel-ENC-DVS1-Snapshot1.vmem

511745668   2048 -rw-------    1 root     root       1301090 Nov  4  2016 ShoreTel-ENC-DVS1-Snapshot1.vmsn

448831108 16777216 -rw-------    1 root     root     17179869184 Jan 13 22:34 ShoreTel-ENC-DVS1-e78a5214.vswp

16817796 209715200 -rw-------    1 root     root     214748364800 Nov  4  2016 ShoreTel-ENC-DVS1-flat.vmdk

50372228   1024 -rw-------    1 root     root          8684 May 12 20:08 ShoreTel-ENC-DVS1.nvram

21012100      0 -rw-------    1 root     root           507 Oct 20  2016 ShoreTel-ENC-DVS1.vmdk

25206404      0 -rw-r--r--    1 root     root           479 Nov  4  2016 ShoreTel-ENC-DVS1.vmsd

12623492      8 -rwxr-xr-x    1 root     root          2998 Jan 13 22:34 ShoreTel-ENC-DVS1.vmx

436248196      0 -rw-------    1 root     root             0 Jan 13 22:34 ShoreTel-ENC-DVS1.vmx.lck

54566532      8 -rw-------    1 root     root          3864 Jan 26  2016 ShoreTel-ENC-DVS1.vmxf

440442500      8 -rwxr-xr-x    1 root     root          2996 Jan 13 22:34 ShoreTel-ENC-DVS1.vmx~

163618436   1024 -rw-r--r--    1 root     root        303285 Nov 24  2018 vmware-13.log

234921604   1024 -rw-r--r--    1 root     root        269420 Nov 24  2018 vmware-14.log

272670340   1024 -rw-r--r--    1 root     root        329779 Dec  6  2018 vmware-15.log

339779204   1024 -rw-r--r--    1 root     root        350954 Dec 24  2018 vmware-16.log

394305156   1024 -rw-r--r--    1 root     root        990171 Jun  1  2019 vmware-17.log

415276676   1024 -rw-r--r--    1 root     root        963882 Jan 13 21:09 vmware-18.log

444636804   1024 -rw-r--r--    1 root     root        661578 May 13 15:14 vmware.log

432053892 195584 -rw-------    1 root     root     200278016 Jan 13 22:34 vmx-ShoreTel-ENC-DVS1-3884601876-1.vswp

[root@ESXi-Encino:/vmfs/volumes/56a6b74e-cffcbcd4-4c2a-9457a5527304/ShoreTel-ENC-DVS1] df -h

Filesystem   Size   Used Available Use% Mounted on

VMFS-5     458.2G 375.7G     82.5G  82% /vmfs/volumes/datastore1

vfat         4.0G  23.6M      4.0G   1% /vmfs/volumes/56a6b74f-ea979f22-3bc9-9457a5527304

vfat       249.7M 183.5M     66.2M  74% /vmfs/volumes/77d56276-6242e12b-3bb3-5e40eb924faa

vfat       249.7M   8.0K    249.7M   0% /vmfs/volumes/55016d7b-9fdca45f-9666-00de496181c6

vfat       285.8M 208.1M     77.7M  73% /vmfs/volumes/56a6b74e-87f7a1c2-b0f7-9457a5527304

[root@ESXi-Encino:/vmfs/volumes/56a6b74e-cffcbcd4-4c2a-9457a5527304/ShoreTel-ENC-DVS1]

ShoreTel-WLA-DVS1

[root@ESXi-WLA:/vmfs/volumes/56a62e3a-f12aed25-f905-5820b1068e18/ShoreTel-WLA-DVS1] ls -lisa

total 168979488

8403204      8 drwxr-xr-x    1 root     root          3220 Nov  5  2019 .

      4   1024 drwxr-xr-t    1 root     root          1960 Nov  3  2016 ..

696269060 26166272 -rw-------    1 root     root     26793463808 May 13 18:47 ShoreTel-WLA-DVS1-000001-delta.vmdk

700463364      0 -rw-------    1 root     root           338 Nov  5  2019 ShoreTel-WLA-DVS1-000001.vmdk

692074756 8388608 -rw-------    1 root     root     8589934592 Nov  4  2016 ShoreTel-WLA-DVS1-Snapshot1.vmem

687880452   2048 -rw-------    1 root     root       1294078 Nov  4  2016 ShoreTel-WLA-DVS1-Snapshot1.vmsn

494942468 8388608 -rw-------    1 root     root     8589934592 Jul 21  2019 ShoreTel-WLA-DVS1-eaac7caa.vswp

16791812 125829120 -rw-------    1 root     root     128849018880 Nov  4  2016 ShoreTel-WLA-DVS1-flat.vmdk

50346244   1024 -rw-------    1 root     root          8684 May 12 19:25 ShoreTel-WLA-DVS1.nvram

20986116      0 -rw-------    1 root     root           507 Jul  9  2016 ShoreTel-WLA-DVS1.vmdk

25180420      0 -rw-r--r--    1 root     root           481 Nov  4  2016 ShoreTel-WLA-DVS1.vmsd

12597508      8 -rwxr-xr-x    1 root     root          2740 Nov  5  2019 ShoreTel-WLA-DVS1.vmx

482359556      0 -rw-------    1 root     root             0 Jul 21  2019 ShoreTel-WLA-DVS1.vmx.lck

75512068      8 -rw-------    1 root     root          3864 Jan 25  2016 ShoreTel-WLA-DVS1.vmxf

562051332      8 -rwxr-xr-x    1 root     root          2740 Nov  5  2019 ShoreTel-WLA-DVS1.vmx~

109066500   1024 -rw-r--r--    1 root     root        908206 Dec 15  2018 vmware-25.log

184563972   1024 -rw-r--r--    1 root     root        306515 Dec 24  2018 vmware-26.log

251672836   1024 -rw-r--r--    1 root     root        273116 Dec 24  2018 vmware-27.log

339753220   1024 -rw-r--r--    1 root     root        502491 Mar 25  2019 vmware-28.log

440416516   1024 -rw-r--r--    1 root     root        902540 Jul 20  2019 vmware-29.log

490748164   1024 -rw-r--r--    1 root     root        614289 Nov  4  2019 vmware-30.log

566245636   1024 -rw-r--r--    1 root     root        824368 May 13 14:32 vmware.log

478165252 195584 -rw-------    1 root     root     200278016 Jul 21  2019 vmx-ShoreTel-WLA-DVS1-3937172650-1.vswp

[root@ESXi-WLA:/vmfs/volumes/56a62e3a-f12aed25-f905-5820b1068e18/ShoreTel-WLA-DVS1] df -h

Filesystem   Size   Used Available Use% Mounted on

VMFS-5     271.8G 233.7G     38.0G  86% /vmfs/volumes/datastore1

vfat       285.8M 208.1M     77.7M  73% /vmfs/volumes/56a62e3a-c10256a5-dcda-5820b1068e18

vfat         4.0G  17.9M      4.0G   0% /vmfs/volumes/56a62e3a-fd7e600d-028f-5820b1068e18

vfat       249.7M 183.5M     66.2M  74% /vmfs/volumes/a37c227f-0ef7f916-59c1-edbe21b399ae

vfat       249.7M   8.0K    249.7M   0% /vmfs/volumes/11a27e75-aa4e1c8d-ffd8-2ec23eefaa07

[root@ESXi-WLA:/vmfs/volumes/56a62e3a-f12aed25-f905-5820b1068e18/ShoreTel-WLA-DVS1]

0 Kudos
a_p_
Leadership
Leadership

It looks like someone created snapshots in Nov 2016, and forgot about them!?

Fortunately the VMs' have thick provisioned virtual disks, and you've got free disk space on the datastores, so there shouldn't be any issues with deleting the snapshots. This may take quite some time due to their sizes, so please remain patient. Deleting the snapshots will free up ~50GB on ShoreTel-ENC-DVS1, and ~30GB on ShoreTel-WLA-DVS1, which gives you sufficient free disk space faor a new snapshot for the upcoming backup.

Two hints:

  1. Create the new snapshot with the VMs being powered off. This will safe disk space, because this way there's no need for a memory snapshot, and you'll have a defined state, in case the update doesn't work as expected, and you need to revert to the snapshot.
  2. Keep the snapshot only as long as needed, i.e. if the update is done, and everything works, delete it.

André

0 Kudos
IRIX201110141
Champion
Champion

Snapshot age is 3.5Y?!?  Congrats.... my personal new record holder Smiley Wink

Just press the "delete all" within Snapshot manager. Yes shutting down the VM when possible is a good idea.  Please keep notice that you cant start the VM until snapshot is deleted.

Regards,
Joerg

0 Kudos
DrorAmbar
Enthusiast
Enthusiast

Thank you Andre for your reply, I appreciate the tips.

It was our VoIP vendor who created it before a previous software upgrade.

The previous vendor set it up. Before that, we didn't use VMWare for our VoIP servers and switches and that vendor pushed for it, but it looks like they didn't know exactly what they were doing.

0 Kudos
DrorAmbar
Enthusiast
Enthusiast

LOL...Thank you?...:smileylaugh:

OK, so I take it that Delete All will consolidate the data from the snapshot and will actually bring the parent VM to the working state the snapshot was in.

I also understand that it is recommended to take snapshots when the VM is powered down. Is it also recommended to Delete all when the VM is shutdown as well, or it doesn't matter in this case?

0 Kudos