ESXi

I'm a developer on a content inspecting web gateway product. One of the features of our product is that it will recursively unpack downloaded files, in order to perform in-depth content analysis such as malware scanning and binary file type detection.

Our product is currently having problems with the ESXi 4.0.0 ISO image. More specifically, the files "cim.vgz" and "sys.vgz" are causing errors when we try to unpack them. In most cases, this prevents users of our software from being able to download the file. This is a situation that we'd prefer to avoid, especially as how the download is a legitimate VMWare ISO image (checked via MD5).

As far as our code can tell, both of the problem files are standard GZip archives. We use 7-Zip to extract the contents of these files, and there are no problems in doing so. The extracted files then look like TAR archives, which we again run 7-Zip on in order to extract the contents. It is at this point that 7-Zip returns an error, which our code picks up and marks the file as "bad".

We have used a number of tools in order to attempt to extract these files. 7-Zip simply says it can't open the TAR file. WinRAR reports corruption errors, but is sometimes able to report a few files. GNU TAR does best and is able to extract several files, but still reports errors.

Is there something special about these files that prevents them from being unpacked by normal tools? If so, is there any way to extract the contents of these files, in order for our software to inspect them? If not (for legal, technical or any other reason), is there something we can look for that differentiates these files from normal TAR+GZ archives?

Thanks in advance.

Hi,

have you figured anything out yet? I'm trying to extract the pci.ids and simple.map file from the sys.vgz file in order to override them in the oem.tgz, but I can't find a way to unpack the .vtar file which I get after gunzip'ping the .vgz ... have the same errors as you mention in your post.

Best regards,

Philipp

I haven't found a lot yet.

So far, it looks like the files are a variant of the UStar TAR format. They appear to have the additional UStar-style data associated with them. However, that's about all I can tell so far.

I'll post back if I find out anything else.

You can't use a build ESXi 4 server to track down the location of the map and PCI files?

This will not help my problem. My company's product needs to be able to extract the files from the TAR archive for inspection. This happens at the time that ESXi is being downloaded, so it's no good installing the product to get at the files.

So far, the biggest difference in the file format is that the VMWare TAR files use "visor\x20\x20\x20" in the "magic" field. For a UStar TAR file this would be "ustar\x00\x00\x00".

I'm not sure quite what the other differences are yet, as I haven't had time to take a better look at the files.

vgz is a modified tgz format. It is to reduce memory consumption both physical and reservation for user apps.

You can use vmtar to convert between tgz <-> vgz. The usage:

Usage: vmtar {[-x vtar/vgz-file] -o destination} | -t < vtar/vgz-file

-Praveen

Thanks for the information Praveen. Unfortunately it does not help me.

I cannot find any information on the vmtar application. I cannot find it on the VMWare site, in a general Google search, or in the Synaptic package manager in Ubuntu.

I'm also not sure why vmtar would be better with memory than normal tar. The tarball appears to have been compressed with normal GZip compression, and the data looks quite normal after decompression. It's only the tarball itself that's different, and there should be nothing in there the raw untar process that uses large amounts of memory.

Normal tar format aligns files on 512 byte boundary. vmtar does it on a 4K (page size) boundary. This helps sharing pages for running apps directly with the vmtar file. Otherwise, we would need to allocate memory. So for example, if you have 10 apps sharing a page, the memory needed is 1 page. But if we were to allocate, then it would be 10 pages.

vmtar is internal only and the following is not officially supported:

You can get vmtar from /sbin/vmtar after you boot the machine. To login into the machine, type "unsupported" on the console (Alt-F1). Then for the login as root and no password.

Hope this helps.

-Praveen

I am trying to expand the sys.vgz file, and receive the screen of death (no ramdisk, image corrupt) while ESXi is loading. All I want to do is add 2 pci-ids to the etc/vmware/simple.map within the sys.vgz.

This is my maddness:

cp sys.vgz sys.gz

gunzip sys.gz

vmtar -x sys -o sys.tar

tar xvf sys.tar

edit my files

tar -cf sys.tar ./*

vmtar -c sys.tar -o sys

gzip -S .vgz sys

I place the new sys.vgz in my build directory and mkisofs, then burn the ISO. I have tried variour methods to untar the .vgz file with no luck.

Any suggestions?

David

Can you remove ./ when doing tar, i.e. "tar -cf sys.tar ./*" should be "tar -cf sys.tar *"and make sure if there are files starting ".", they are part of tar file.

Please let me know if it still does not work.

Regards,

Praveen

I struggled with this for a while - here is where the breakdown is:

if you are trying to include hidden files, "tar -cf sys.tar *" will not work - it is the wildcard that is killing you here... instead tar the directory "tar -cf sys.tar ./"

Hey!

I am confused now:

stillfly writes that tar cf sys.tar ./ works for him: I found that "./" still leads to the "boot image is corrupted" error on boot.

My soulution is tar cf sys.tar `ls` what give the right structure to the archiv.

Like:

# tar tf sys.tar

bin/

bin/busybox

bin/ipkg

Cheers

Did you ever find a solution or workaround to extract files from .vgz archives without the vmtar program?

You can grab a copy of the vmtar binary from the ESXi 'tech support shell', scp root@youresxihost:/sbin/vmtar .

Seems to work fine for me on debian 5.0.5, I've successfully injected a driver into sys.vgz so I can install ESXi 4.1 (Update 1) on some HP BL460c g7's.

How about a windows version of vmtar, at least the sources so I can build it myself.

PS Why on earth do they use a non standard tar file format?

I don't believe that vmtar is an opensource tool but you could check the source available from http://downloads.vmware.com/d/info/datacenter_downloads/vmware_vsphere_hypervisor_esxi/4_0#open_source

When you copied vmtar from the maintenance console on ESXi, where did you copy it to?   What's the command sequence to mount a usb drive under ESXi so I have a place to copy it to?

Thanks

It isn't possible to attach a USB drive to ESXi. Copy the file to a datastore and then download it from the datastore browser.

Is there an ftp client, of something close to that?

The datastore browser is part of the vSphere client.

How do you expect to use file when you download it?

May be relevant, here is a blog post I wrote about injecting .vib packages into the installer and installed ESX image.

http://www.stumptownvirtualization.com/2011/02/adding-drivers-to-esxi-installer-and.html

The objective is to try to run in on my linux system to edit the standard ESXi install to add the PCI id for an unsupported NIC to point to a driver that I know runs a NIC that is register and functionally identical to a supported NIC.   I've discovered that many of the unsupported NICs differ from supported NIC only by their PCI id.   It shouldn't be this hard, it a linux kernel, I should be able to mount a usb thumb drive just like on any other linux kernel, its already smart enough to support USB keyboards.

On a different question, on the ESXi "Welcome" webpage, there's a section with links to "vSphere Remote Command Line".  All those links point to the VMware Developer http://communities.vmware.com/community/vmtn/developer Webpage that has dozens of downloads.  Do you have the specific link to what the unWelcome page incorrectly calls "Download the Windows Installer"

ESXi is not Linux and it is not possible to mount a USB disk drive. You mention a linux system yet you earlier asked about Windows sources so I wanted to understand your need to use the vmtar app. I wouldn't try editing anything in Windows and especially text files. The post immediately before your last one should be a good help to doing what you want to try. I would suggest that there may be more differences than just the PCIID even though the NICs might seem to be the same otherwise.

The links to various components were just reorganized this week and unfortunately the vCLI links don't function as they should. I will forward that information to the website maintenance crew. Many thanks for bringing that to our attention.

The link should have gone to http://communities.vmware.com/community/vmtn/server/vsphere/automationtools/vsphere_cli

   Actually, I've implemented NIC device drivers.   Its a common practice for vendors to make different devices register compatible.  If you look at the details of the supported Intel NICs, you will find they are not only register compatible with other Intel NICs, but other vendors make NICs register compatible with Intel NICs.    Intel produces dozens and dozens of NICs with different model numbers, the number of different register sets is just a handful.

I understand. I have been doing this for a very long time. You sounded a little unsure and asking for a Windows version of vmtar isn't a good sign for what may be a bit of non standard work. If you understand what you are doing and have the command line skills you should have no problems. I certainly wish you well and do let us know how it goes.

I asked for a windows version because windows is much better desktop environment than linux.  This is my first adventure in VMware and my initial impressions are that there lots of things that are very hard that should be very easy.     Oh, editing in Windows, don't worry, you may be a great sysadmin, but I'm an experienced os kernel developer (Windows and Linux and Unix), the difference between windows and unix text files is a pretty trivial issue/

What are the precise steps to get \sbin\vmtar into a spot where I can place in on my linux system?  Any media will do.

you can use scp from the linux commandline.

I tried that, it appears scp either isn't supported or enabled in ESXi

Read the post. From the Linux command line not ESXi.

I read the post, ESXi doesn't enable ssh by default, I discovered its hidden under the term "tech support mode".      Sort of like that old tv series Dragnet "names have been changed to confuse the informed"