Hi,
I have a special cluster where the VMs have USB devices. Each VM has more than one device from the same manufacturer.
I need to identify dongles via the serial number but the vClient show me only the path USB (USB ID).
In ESXi 5.0 can obtain the desired information in usb.log as the example:
2014-02-12T19: 33:58 Z usbarb [2835]: USBArb: Device 6: name: WIBU-Systems \ CodeMeter-Stick vid: pid 064f: 03e9 path: 6/1/1 speed: full
family: storage version: 2 serialnum: 000002319999 id: 60006099f09e9 owner: VM001
The log of ESXi 5.1 or 5.5 does not display this information.
To view the log as shown above, you must add the parameter "--verbose" at service startup in /etc/init.d/usbarbitrator.
However, this change is not persistent.
How could I use the verbose mode on ESXi 5.5 usbarbitrator service persistently?
Thanks.
Why you don't check them from your virtual machine OS?
Good question.
Because the VM logs do not show the USB ID (path) as in vClient. And I have to manage devices that are not yet connected.
These dongles are used on license servers. When the VM has 2 or more dongles and I need to remove or replace one of them,
the usb.log helps me. Without these logs I could remove the wrong device and stop the license service.
Hi EduLima22 ,
Here is one answer to your question. I hope that an easier way exists...
First of all, thanks for your information about the --verbose option for usbarbitrator, I did stay on ESXI 5.0 because I dind't find out this information.
For the record, I use a free version of ESXi (for my home server) and, via ssh, from one of my guests, I map an unmap my external USB drive, for backup purpose. But to do that, I need to know some VMware inofrmation that I used to search in usb.log and were missing since version 5.1.
OK, back to your question.
VMware use volatile filesystems for the operating system, and at each boot, these volatile FS are populated from differents files stored in /bootbank. Problem is that these files are not easely manipulated. They seems to be in tar gz format, but they aren't.
I found more information here :
http://www.virtuallyghetto.com/2011/08/how-to-create-and-modify-vgz-vmtar.html
Disclaimer : during my tests, I had once to reboot my ESXi from a CD, so don't try that on a production server ! Also, as I modify a stock ESXi file, I suppose that at every patch application, I probably will have to run the whole process again ! Last but not least, I'm not sure that this solution is supported by VMware
So, what I did : The tests were made on a completely fresh install of esxi 5.5 (from a DELL CD, as I own a DELL server), the version is 1746974. I also use a running CentOS 5 server to manipulate the file. The server is a 32 bits version, that seems to be OK for tar.
I had to activate local shell and remote ssh for these operations.
On the ESXi :
mkdir /vmfs/volumes/datastore1/s
cp /bootbank/s.v00 /vmfs/volumes/datastore1/s
cd /vmfs/volumes/datastore1/s
zcat s.v00 >s.vmtar
vmtar -x s.vmtar -o s.tar -v
--- copy the s.tar on the linux machine. (maybe the other steps could be done on the esxi ?)
On the Linux box
Important : you need to be root, to be sure that the file are restored with the original owner/group.
mkdir s
cd s
tar xvf {path to the file}/s.tar
vi etc/init.d/usbarbitrator
(I did this change :
USBARBITRATOR_SCHED_PARAM="--verbose ++group=usbArbitrator"
)
touch -r etc/init.d/cdp etc/init.d/usbarbitrator
# it seems that the order of the file is critical
tar tf {path to the file}/s.tar >/tmp/s.filelist
tar -cv -T /tmp/s.filelist --no-recursion --same-order -f {path to the file}/s1.tar
------------ send s1.tar back to the ESXi in the same folder than s.tar
On the ESXi :
vmtar -c s1.tar -o s1.vmtar -v
gzip -9 -c s1.vmtar >s1.v00
Now, a little check can be useful !
vmtar -t <s.v00 >s.txt
vmtar -t <s1.v00 >s1.txt
Back to the unix machine
retrieve the two files
We have to suppress the date and time :
cat s.txt | awk ' { $4="" ; $5="" ; print } ' >sc.txt
cat s1.txt | awk ' { $4="" ; $5="" ; print } ' >s1c.txt
diff sc.txt s1c.txt
183c183
< -r-xr-xr-x 201/201 4483 etc/init.d/usbarbitrator
---
> -r-xr-xr-x 201/201 4493 etc/init.d/usbarbitrator
OK, only one change, hat is what we expected
and now, tadaa !!!
cp s1.v00 /bootbank/s.v00
reboot (and cross your fingers !)
It works for me !
Thanks,
Pascal
Thanks Pascal,
I'll try this solution as soon as possible!
However, will this change may be undone after an update?
The main point is the lack of information about the USB devices on vCenter.
I bypassed this limitation through the script below. I'm not sure it will work in all scenarios.
Not my best Perl script but it has worked very well.
--x--
#!/usr/bin/perl use strict; use warnings; use VMware::VILib; use VMware::VIRuntime; use Net::SSH::Expect; #$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0; ############################################################################ sub host_ssh_manager($$) { my $h = shift; my $state = shift; if ($state) { $state = "starting"; } else { $state = "stopping"; } my $servicemanager = Vim::get_view( mo_ref => $h->get_property('configManager.serviceSystem') ); my $hostservices = $servicemanager->serviceInfo->service; foreach my $s (@$hostservices) { if ($s->label eq 'SSH') { if (! $s->running) { eval { if ($state eq 'starting') { $servicemanager->StartService(id => $s->key); print "SSH started.\n"; } }; } else { eval { if ($state eq 'stopping') { $servicemanager->StopService(id => $s->key); print "SSH stopped.\n"; } }; } if ($@) { print "Error $state SSH service.\n"; } last; } } } ############################################################################ sub main() { my %opts = ( host => { type => "=s", help => "Target Host", required => 0, }, cluster => { type => "=s", help => "Target Cluster", required => 0, }, ); Opts::add_options(%opts); Opts::parse(); Opts::validate(); Util::connect(); my $host_username; my $host_password; my $my_host = Opts::get_option('host'); $my_host =~ s/\%(..)/chr(hex($1))/gie if (defined($my_host)); my $my_cluster = Opts::get_option('cluster'); $my_cluster =~ s/\%(..)/chr(hex($1))/gie if (defined($my_cluster)); print "Host username: "; chop($host_username = ); print "Host password: "; system("stty -echo"); chop($host_password = ); print "\n"; system("stty echo"); my $list_host; my $host_obj; # Host name is specified on command line if (defined($my_host)) { $host_obj = Vim::find_entity_view( view_type => 'HostSystem', properties => [ 'name', 'runtime.powerState', 'vm', 'configManager.serviceSystem' ], filter => {'name' => "$my_host"}, ); if (defined($host_obj)) { if ($host_obj->get_property('runtime.powerState')->val eq 'poweredOn') { push @$list_host, $host_obj; } } } # Cluster name is specified on command line if (defined($my_cluster)) { my $cluster_obj = Vim::find_entity_view( view_type => 'ClusterComputeResource', properties => [ 'name', 'host', ], filter => {'name' => "$my_cluster"}, ); if (defined($cluster_obj)) { print "Cluster: $my_cluster\n"; my $cluster_hosts = $cluster_obj->host; foreach my $h (@$cluster_hosts) { $host_obj = Vim::get_view( mo_ref => $h, properties => [ 'configManager.serviceSystem', 'name', 'runtime.powerState', 'vm', ] ); if ($host_obj->get_property('runtime.powerState')->val eq 'poweredOn') { push @$list_host, $host_obj; } } } } # Search for USB devices foreach my $h (@$list_host) { my $hostname = $h->name; print "Host: " . $hostname . "\n"; # Search VMs my $list_vm = $h->vm; my %vms; my %paths; my %dongles; foreach my $vm_ref (@$list_vm) { my $vm_obj = Vim::get_view( mo_ref => $vm_ref, properties => [ 'config.hardware.device', 'name', ] ); my $vm_name = $vm_obj->name; my $vm_devices = $vm_obj->{'config.hardware.device'}; # Search USB devices foreach my $device (@$vm_devices) { if($device->isa('VirtualUSB')) { my $usbdevice_name = $device->backing->deviceName; # Unfortunately, USB path (?) is the "primary key" for vCenter. my $usbdevice_path = ($usbdevice_name =~ /(path:\S+)/)[0]; if ($usbdevice_path =~ /path:(\d+)\/\d+\/(\d+)/) { $usbdevice_path = "path:" . $1 . "/X/" . $2; } my $usbdevice_info = $device->deviceInfo->summary; if (defined($usbdevice_path) && defined($usbdevice_info)) { if (! $device->connected) { $vms{$vm_name}{$usbdevice_path} = "DEVICE NOT PRESENT!"; } else { $vms{$vm_name}{$usbdevice_path} = $usbdevice_info; } $paths{$usbdevice_path} = $vm_name; } } } } # Start sshd on host host_ssh_manager($h,1); # Search information on the host's logs by ssh command lines my $ssh = Net::SSH::Expect->new ( host => "$hostname", password=> "$host_password", user => "$host_username", ); my $login_output = $ssh->login(); my ($exec,$line,$date,$vm,$device,$available); my ($serial,$bus,$level,$port,$path); my $usblog = 1; my $usbdevices = 1; # Look at /dev/usbdevices. if ($usbdevices) { $exec = $ssh->send("cat /dev/usbdevices"); $level = 0; while (defined($line = $ssh->read_line())) { if ($line =~ /^T:\s+Bus=\s*(\d+)\sLev=\s*(\d+)\sPrnt=\s*(\d+)\sPort=\s*(\d+)\sCnt=\s*(\d+)\sDev#=\s*(\S+)\s/) { $bus = $1; $level = $2; $port = $4; if ($level > 1) { $path = sprintf("path:%d/X/%d",$bus,$port); } elsif ($level == 1) { $path = sprintf("path:%d/%d",$bus,$port); } $device = $serial = ""; $available = 0; } elsif ($line =~ /V:\s+Available for Passthrough/) { $available = 1; } elsif ($line =~ /Product=(.*)/) { $device = $1; } elsif ($line =~ /SerialNumber=(.*)/) { $serial = $1; } elsif ($line =~ /^E:/ && $path && $device && $available) { $dongles{$path}{device} = $device; $dongles{$path}{serial} = $serial; $path = $device = $serial = $port = $bus = ""; $level = $available = 0; } } } # Look at /var/log/usb.log (ESXi 5.0). if ($usblog) { $exec = $ssh->send("grep 'USBArb: Device' /var/log/usb.log"); while (defined($line = $ssh->read_line())) { $date = $vm = $device = $serial = $path = 0; if ($line =~ /^([^\ ]+)/) { $date = $1; } if ($line =~ /:name:(.*?) vid/) { $device = $1; } if ($line =~ /(path:[^\ ]+) /) { $path = $1; if ($path =~ /path:(\d+)\/\d+\/(\d+)/) { $path = "path:" . $1 . "/X/" . $2; } } if ($line =~ /serialnum:([^\ ]+) /) { $serial = $1; } if ($line =~ /owner:(\S+)/) { $vm = $1; } if ($device && $path && $vm && defined($dongles{$path}) ) { if ($vm !~ /\(null\)/) { $dongles{$path}{vm} = $vm; } else { undef $dongles{$path}{vm}; } $dongles{$path}{date} = $date; $dongles{$path}{device} = $device; $dongles{$path}{serial} = $serial; } } } if (!%dongles) { print "\n"; print "\tOBS: No dongles found in the host logs!"; print "\n"; } # Stop sshd host_ssh_manager($h,0); # Show the list of VMs with USB dongles foreach my $vm_name (sort keys %vms) { print "\tVM: " . $vm_name . "\n"; foreach my $path (sort keys %{$vms{$vm_name}}) { if (defined($dongles{$path}{vm})) { if ($dongles{$path}{vm} eq $vm_name) { print "\t\tPath: " . $path; print "\n"; print "\t\tDongle: " . $dongles{$path}{device}; print "\n"; print "\t\tSerial: " . $dongles{$path}{serial}; print "\n"; print "\t\tConnected in: " . $dongles{$path}{date}; print "\n"; print "\n"; } else { print "\t\t>>> Dongle not found in the host logs."; print "\n"; print "\t\tPath: " . $path; print "\n"; print "\t\tDongle: " . $vms{$vm_name}{$path}; print "\n"; print "\n"; } } else { print "\t\t>>> Dongle not found in the host logs."; print "\n"; print "\t\tPath: " . $path; print "\n"; print "\t\tDongle: " . $vms{$vm_name}{$path}; print "\n"; if (defined($dongles{$path}{serial})) { print "\t\tSerial: " . $dongles{$path}{serial}; print "\n"; } print "\n"; } } } print "\n"; print "\tDongles connected to the host but not associated with VMs:\n"; foreach my $path (sort keys %dongles) { if (! defined($dongles{$path}{vm}) && ! defined($paths{$path})) { print "\t\tPath: " . $path; print "\n"; print "\t\tDongle: " . $dongles{$path}{device}; print "\n"; print "\t\tSerial: " . $dongles{$path}{serial}; print "\n"; if (defined($dongles{$path}{date})) { print "\t\tConnected in: " . $dongles{$path}{date}; print "\n"; } print "\n"; } } print "\n"; } Util::disconnect(); } main(); #==eof==
Hi EduLima22,
As I was on my run test (The whole is done at home, and, to have a test ESXI, I change the drive of one of my PC ), I tried that also. I installed the latest patch for ESXi 5.5, and after update, I had to rerun the complete process to have again the log verbosity increased.
The whole process took me around a quarter of an hour, and could be scripted. Moreover, if you need to do the change on many ESXi, most of the work is to be done once.
I completely agree with you, the biggest problem is the lack of informations. For example, we can see that the usbarbotrator search for customisation informations in two files at boot. If we only know the possible options... Maybe increasing the log verbosity is just a matter of creating a configuration file!
About your script : as far as I know, the free version of ESXi only give a read only access to the API. So it's not a solution for me, at least at home (at work, this is another story! I would probably write a bit of orchestrator to deal with that!)
Regards,
Pascal
Hi Edulima22,
Just for the record :
I did all my tests on my personal desktop. I did store my modified S.V00 files, of course. Last friday, I reinstall my 'personal production' server in VMware 5.5 (I didn't upgrade as my boot disk just died, hence the version change).
I just copied back the modified S.V00 file on this server, and since, everything is OK.
So, I would say that for me, this is a working solution.
Regards,
Pascal