1 2 Previous Next 15 Replies Latest reply on Feb 2, 2008 6:25 AM by devzero

    Project proposal: device-mapper target "dm-vmware" !?

    devzero Master

      Hello !


      i spent some thoughts on this and like to have some discussion about the pro`s and con`s about some (for now completely theoretical) Linux kernel feature which could probably be helpful regarding handling of physical/raw disk mappings and virtual disks.


      Currently, i know of the following two disk related "issues" with vmware hosted products on Linux:


      \- VMware Workstation/Server/Player can not handle devices besides /dev/hd... and /dev/sd...., so if you want to use raw disks on /dev/cciss/...., LVM, loopback, cryptoloop, SAN, iSCSI,SATA...whatever, the only reasonable way to go is using the vmware-bdwrapper "hack", which needs to be build separately and may be tricky to setup for the unexperienced user. This issue may be probably easy to fix in VMWare products, but nobody knows if or when VMware will do this. I don`t expect this happen in the near future since this may not only be a technical issue.


      \- vmware-mount.pl also looks a little bit like a "hack" and having reported stability issues. It`s closely tied  to NBD (Network Block Device). Sending all the data trough the network layer is rather weird method to map/mount a

        file on your local system. It`s like opening a shell on your local GUI by doing ssh-login via localhost. So maybe it`s just time for a (better) replacement.



      For some time, linux has a "device-mapper" subsystem which is a real mighty part of the kernel.


      The idea which came to my mind is, to combine both vmware-bdwrapper and vmware-mount.pl into a common kernel module,  probably some device-mapper related one called "dm-vmware", to provide a generic/all-purpose alternative to the tools mentioned above.


      Things could look like this (adding some "vmware-mode" to dmsetup, communicating with dm-vmware):


      "dmsetup -vmware -fakedisk /dev/sdvm\{##}|/dev/hdvm\{##} GenericBlockDeviceName|/path/to/RawDiskImageFilename" would create a device out of "GenericBlockDeviceName" or "RawDiskImageFilename" which VMware could use (and recognize) as a physical SCSI or IDE disk. That means, adding the necessary ioctls as bdwrapper does. Mind that VMWare built`in limitation where it checks for device names being named /dev/sd* or /dev/hd*.


      "dmsetup -vmware -mountvmdk /dev/vmdk\{##} /path/to/VMdir/VmwareVirtualDiskFileName" would create a linear mapping

      like vmware-mount.pl, so we could do "mount /dev/vmdk1p1 /mnt" and could access the contents of a virtual disk in a more perfomant and more stable way. We could even access the whole disk besides single partitions, which is not possible for now,AFAIK. Vmdk specification is available for the public and also being told to be GPL compatible.


      I didn`t yet disgrace myself by posting this to lkml or dm-devel ,so i`d like to hear your comments.




        • 1. Re: Project proposal: device-mapper target "dm-vmware" !?
          Preem Lurker



          I'm more of a newbie here, so really don't understand all the issues you mentioned here.


          However i am trying to put up a stable/efficient/safe server for production use with mail/web/db/... VM's on it.


          One thing I'd like to have is having each VM on separate raw device. I have 2 500GB SATA drives with RAID1 array. I created 1 primary partition for the host OS and would like to have separate partitions for VM's, which if i understand correctly, could be done with LVM's, but the problem remains, as u mentioned it VMware server doesn't recognize LVM's as separate devices.


          Now, my question is, is vmware-bdwrapper - the #1 reasonable solution for this in the matter of stability and safety?


          And the other thing, how are the pro's taking care of this?


          Hoping someone could point me in the right direction here.





          edit: sorry for hijacking your thread - i know it's not a discussion you were expecting, but never the less it is about same topic and not really worth opening another thread.


          Message was edited by:


          • 2. Re: Project proposal: device-mapper target "dm-vmware" !?
            devzero Master

            >put up a stable/efficient/safe server for production use


            i wouldn`t recommend vmware-bdwrapper for production use. it`s sort of a hack and there isn`t much feedback from the people who are using it. so, there is nobody who can guarantee , that there aren`t issues - and there is only one person who \_could_ provice support with this (i.e. the author of vmware-bdwrapper)

            for me, it`s more like some "demonstration" to show the people, what would be possible if vmware would.......


            maybe you better open a SR and ask vmware why they don`t support any raw-device besides hdX and sdX

            • 3. Re: Project proposal: device-mapper target "dm-vmware" !?
              oreeh Guru

              Very interesting idea.


              But personally I'm unable to contribute or verify.

              I'm not a kernel hacker and C is not a programming language


              But I'm an eager alpha and beta tester.

              • 4. Re: Project proposal: device-mapper target "dm-vmware" !?
                mahadri Novice

                Hello!  I like your ideas, and I wish there were more discussion, so here are my thoughts.


                I thought vmware-mount.pl is a kludge, so I've happily avoided it.  All my vmdk files are plaintext, referring to either block devices or sparse monolithicFlat files.  I can mount both through a loopback device.  For snapshots, I use device-mapper's snapshot target.  Creating tools for this process would be a lot easier than a vmdk device-mapper target, unless some existing source code could be readily adapted for one.  There is one downside, though: migrating a disk between Windows and Linux would be a hassle.


                A fakedisk target would be easy to write and could use a build process similar to the vmnet and vmmon kernel modules.  However, I would like to avoid creating a device-mapper wrapping device for every device to be used as a virutal disk.  It would be best if users could just specify the actual device in VMWare.  This is where I thought of vmgbd.  When compiled as a static binary, users can download, run it, and use actual device paths in VMWare without any external configuration.


                To add fuel to the "block device limitations not being a technical issue" fire,  vmgbd patches only 46-57 bytes, which is <= 10 lines of C code.  Yes, 10 lines of code can remove the limitation.

                • 5. Re: Project proposal: device-mapper target "dm-vmware" !?
                  devzero Master

                  i found some other person spending some thoughts on this (see below).


                  let`s continue to discuss the device-mapper vs. the fuse based approach - maybe some other people jump in with this! (and hopefully they are some more skilled programmers than me)









                  vmware-loop (and vmware-mount) really sucks!

                  Several times I tried to mount a .vmdk file on a linux host directly in order to run some backups. But the box crashed to hard, that I had to reset-finger it. I found a >>small vmdk-mount hack from a australian coder, who told me, what the architecutral problem with both of those utils is (his own as well as the one from vmware)


                  matthewc> when the system is low on memory, the Linux kernel instructs nbd to write out all its dirty buffers

                  matthewc> but in order to do that, nbd needs to talk to the userspace process

                  derjohn> hm, that sounds like the problem vmware-loop itself has …

                  matthewc> and you get this recursive wait situation where the userspace process is blocked waiting on a memory allocation, but nbd needs to talk to it to make memory


                  It's time to port that or something similar to FuSE.


                  UPDATE: I asked kernel-people of the Linux-VServer project about nbd and add here an excerpt of the chat, that might be interesting,too:


                  derjohn: the main question is 'used for what?' I assume buffers and caches

                  well, doesnt sound like a good idea to rely on nbd


                  • 6. Re: Project proposal: device-mapper target "dm-vmware" !?
                    derjohn Lurker


                    as author of the blog you are posting here is just want to add my presence here to enlarge this thread a little (to give it more weight ).


                    With the FuSE approach we can only achieve a mounted filesystem, as FuSE links into the VFS layer. Advantage would be the "no need to recompile the kernel" approach, disadvantage that we cant modify the partitions (fdisk-ing) on that disk.


                    By architectural perspective the DM approach is the right thing to do. And I think we could do it as Device-mapper "target", which could be compiled separate from the kernel - that's as good as compile a FuSE module/fs.


                    Do we need to call a bounty and raise soem funds for a geek-coder who can do the job? Or will vmware have a pity on us "crazy filesystem mounters" and fix their outdated and buggy stuff (nbd) ?


                    asks himself,


                    • 7. Re: Project proposal: device-mapper target "dm-vmware" !?
                      devzero Master

                      i`m not sure about the maturity level of dm-userspace, but i think this should be the best approach - and an "easy" one - at least it is more straightforward to implement such thing in userspace, first.







                      some vmdk userspace code, probably suitable for being recycled for vmdk plugin: http://fabrice.bellard.free.fr/qemu/qemu-0.9.0.tar.gz (qemu-img)


                      Message was edited by:



                      Message was edited by:


                      • 8. Re: Project proposal: device-mapper target "dm-vmware" !?
                        devzero Master

                        vmware-tools is opensource now.

                        if vmware will make vmware-mount opensource, too ?:)



                        anyway - just came across uloop:



                        looks like some promising "component" for a better vmware-mount.pl


                        Message was edited by:


                        • 11. Re: Project proposal: device-mapper target "dm-vmware" !?
                          devzero Master

                          another interesting approach which could provide some infrastructure for a vmware-mount replacement:




                          Linux SCSI target framework (tgt) project


                          Linux target framework (tgt) aims to simplify various SCSI target driver (iSCSI, Fibre Channel, SRP, etc) creation and maintenance. Our key goals are the clean integration into the scsi-mid layer and implementing a great portion of tgt in user space.


                          Target drivers

                          Virtual SCSI target driver for IBM pSeries

                          Virtual SCSI target driver for Xen

                          Qlogic qla2xxx FC target driver (in progress)

                          LSI logic FC target driver (not yet)

                          iSCSI software target driver for Ethernet NICs

                          iSER software target driver for Infiniband and RDMA NICs (not yet)

                          Qlogic qla4xxx iSCSI target driver (not yet)

                          • 12. Re: Project proposal: device-mapper target "dm-vmware" !?
                            devzero Master


                            nice to have a solution now.  



                            (take a look at vmware server 2 beta)



                            not nice, that nobody at vmware is talking about what`s going on and i was completely wasting my time with that stuff.

                            i really dislike that vmware is so secretive with their stuff.



                            anyway - thanks for implementation via fuse!






                            • 13. Re: Project proposal: device-mapper target "dm-vmware" !?
                              larstr Virtuoso

                              Where can we find some more info on how this is solved in Server 2.0? Normal searches on fuse and VMware are not revealing anything here....



                              • 14. Re: Project proposal: device-mapper target "dm-vmware" !?
                                devzero Master

                                Hi Lars,


                                just download the Server 2.0 beta.

                                If you don`t want to install, just unpack the .tar.gz and you will see.

                                iirc, the binary within can work without vmware being installed at all.


                                unfortunately, it`s only half of the solution, since you cannot map the whole disk or parts of it to be accessible with other tools. you can only mount partitions. should be easy for vmware to make this better, imho - so please sign my feature request at http://communities.vmware.com/thread/114629?tstart=0


                                >Normal searches on fuse and VMware are not revealing anything here....

                                i don`t wonder. since when is vmware "open" about what they do ?


                                merry christmas and a happy new year, btw!






                                thanks for the vmktree beta. i will give it a try when i`m back at work (will be around mid january)

                                1 2 Previous Next