Shared hard drive in guest VMs

[mirumpf]

Hi,

I have absolutely no background in VMware technologies, so please bear with me. I'm just a user of a shared service inside the company where I work.

We have quite a few Linux boxes on which build jobs are executed from a central master (jenkins-ci.org).

Currently we are using a Samba share where each build job has its own workspace folder. There is absolutely no interaction between the folders, each job (VM) reads and writes from and to to its own directory structure.

Unfortunately Samba comes with a performance penalty of almost 30%. The idea was to provide a shared virtual disk from the host and make it accessible to the guest VMs. The KB article VMware KB:    Sharing a virtual disk between multiple virtual machines explains some of the issues with such an approach.

The question is, are there any other ways to achieve what I explained above without using Samba or NFS? Data corruption would not be an issue as the read/write operations would be on separate files. Even the fact that the changes in one VM are not visible to the other VM is OK as the jobs work on completely different folders and files.

Thanks,

    Michael

[schepp]

Hi,

welcome to the communities,

you shouldn't mount a single virtual disk to multiple VMs, as the filesystem inside the disk is probably not made for multi access.

I would go with a NFS share, as it's built for multiple access and it's faster than samba. (Was there a reason to choose samba  and not NFS when you use linux boxes?)

Tim

Edit: Discussion successfully moved from VMware vCloud Director to VMware vSphere™ Storage

[mirumpf]

We had a NFS share first, but the NFS files-system creates temporary hidden files like ".nfs0123123 " by which our build jobs got confused. When there is an inventory process running, getting a list of files, the .nfs0123123 files are included in the list. When the ZIP operation runs to pack all files together, the .nfs0123123  file was already gone, which led to an error in the ZIP operation.

We have Ubuntu 12.04 LTS running and did some comparison between NFS and Samba and found no big difference in terms of performance. With Ubuntu 14.04 LTS and Samba 4.1 we expect a performance improvement by using SMB3 instead of SMB1 protocol.

I cannot believe that a shared disk feature is not available on the VMWare server side

Thanks,

    Michael

[schepp]

Well, it's available on VMware side. VMware uses VMFS so that multiple hosts can access the the same datastore.

Your problem is not on VMware side, but on client side, as the Filesystem inside the  VM (ext3,ext4, etc.) ist not capable of multiple hosts accessing.

By the way, the .nfsXXXXXXX files are created when a file is removed, that is still opened by another process. So your build jobs may be removing temp files before they close them.

Tim

[mirumpf]

We did not analyze the failures when an error occurred with the temporary .nfsXXXXXXXXXX files, but it happened from time to time. The solution was to switch to Samba

So the baseline of what you are saying is: "No, there is no easy way to provision a shared virtual disk in read-write mode to multiple VMs."

But is it possible that such a feature will be added in the future or do we need to further pursue performance improvements inside the VM by using Samba 4.1 for example.

Thanks,

    Michael

[schepp]

Correct, there's no easy way to share a RW disk across multiple VMs.

And no, I don't think VMware will or can add such a feature, as it's not VMware's problem. It's the filesystem inside the guest OS that only allows one write access at a time and this has nothing to do with virtualization

Tim

[f10]

Hi,

Did you explore the multi-writer option for VMFS? This allows a virtual disk to be shared by multiple VMs, check VMware KB VMware KB: Disabling simultaneous write protection provided by VMFS using the multi-writer flag for more information.

-Arun

https://twitter.com/arunpande

http://highoncloud.blogspot.in/

About VMware Virtualization on NetApp

[schepp]

Arun Pandey that won't fix the problem though, as you still need cluster-aware applications. If you just mount this vmdk to multiple linux boxes, the VMs all can write to the ext3/ext4 filesystem, but it will probably break the filesystem and data loss/corruption will occur.

[mirumpf]

The build jobs are separated like this

• /export/workspaces   Mount point of shared disk
• /export/workspaces/workspace1 <- VM1 writes to this folder tree
• /export/workspaces/workspace2 <- VM2 writes to this folder tree
• ...

So the question is whether the ext3/ext4 file-system gets corrupted when each VM writes to different folder trees!? So this goes down to how the ext3/ext4 file-system works internally.

And if ext/3ext4 is not working, how about using another file-system which can cope with this scenario!?

[mirumpf]

What if Linux could access VMFS directly?

https://github.com/glandium/vmfs-tools

Unfortunately not very actively developed!?

[f10]

schepp my response was specific to the question that mirumpf had asked in comment # 4. I completely agree that a cluster aware app would ensure data consistency across shared disks. However IMO if the writes are well controlled and none of the tasks that lead to metadata changes are performed things should be fine. In anyway this is my personal opinion and I haven't tried this but I am sure linux geeks would have a better answer.

-Arun

https://twitter.com/arunpande

http://highoncloud.blogspot.in/

About VMware Virtualization on NetApp