VMware Cloud Community
EduLima22
Contributor
Contributor

How to change verbosity of usb.log?

Hi,

I have a special cluster where the VMs have USB devices. Each VM has more than one device from the same manufacturer.

I need to identify dongles via the serial number but the vClient show me only the path USB (USB ID).

In ESXi 5.0 can obtain the desired information in usb.log as the example:

2014-02-12T19: 33:58 Z usbarb [2835]: USBArb: Device 6: name: WIBU-Systems \ CodeMeter-Stick vid: pid 064f: 03e9 path: 6/1/1 speed: full

family: storage version: 2 serialnum: 000002319999 id: 60006099f09e9 owner: VM001

The log of ESXi 5.1 or 5.5 does not display this information.

To view the log as shown above, you must add the parameter "--verbose" at service startup in /etc/init.d/usbarbitrator.

However, this change is not persistent.

How could I use the verbose mode on ESXi 5.5 usbarbitrator service persistently?

Thanks.

6 Replies
DavoudTeimouri
Virtuoso
Virtuoso

Why you don't check them from your virtual machine OS?

-------------------------------------------------------------------------------------
Davoud Teimouri - https://www.teimouri.net - Twitter: @davoud_teimouri Facebook: https://www.facebook.com/teimouri.net/
Reply
0 Kudos
EduLima22
Contributor
Contributor

Good question.

Because the VM logs do not show the USB ID (path) as in vClient. And I have to manage devices that are not yet connected.

These dongles are used on license servers. When the VM has 2 or more dongles and I need to remove or replace one of them,

the usb.log helps me. Without these logs I could remove the wrong device and stop the license service.

Reply
0 Kudos
sibsib
Contributor
Contributor

Hi ,

Here is one answer to your question. I hope that an easier way exists...

First of all, thanks for your information about the --verbose option for usbarbitrator, I did stay on ESXI  5.0 because I dind't find out this information.

For the record, I use a free version of ESXi (for my home server) and, via ssh, from one of my guests, I map an unmap my external USB drive, for backup purpose. But to do that, I need to know some VMware inofrmation that I used to search in usb.log and were missing since version 5.1.

OK, back to your question.

VMware use volatile filesystems for the operating system, and at each boot, these volatile FS are populated from differents files stored in /bootbank. Problem is that these files are not easely manipulated. They seems to be in tar gz format, but they aren't.

I found more information here :

http://www.virtuallyghetto.com/2011/08/how-to-create-and-modify-vgz-vmtar.html

Disclaimer : during my tests, I had once to reboot my ESXi from a CD, so don't try that on a production server ! Also, as I modify a stock ESXi file, I suppose that at every patch application, I probably will have to run the whole process again ! Last but not least, I'm not sure that this solution is supported by VMware Smiley Wink

So, what I did : The tests were made on a completely fresh install of esxi 5.5 (from a DELL CD, as I own a DELL server), the version is 1746974. I also use a running CentOS 5 server to manipulate the file. The server is a 32 bits version, that seems to be OK for tar.

I had to activate local shell and remote ssh for these operations.

On the ESXi :

mkdir /vmfs/volumes/datastore1/s

cp /bootbank/s.v00 /vmfs/volumes/datastore1/s

cd /vmfs/volumes/datastore1/s

zcat s.v00 >s.vmtar

vmtar -x s.vmtar -o s.tar -v

--- copy the s.tar on the linux machine. (maybe the other steps could be done on the esxi ?)

On the Linux box

Important : you need to be root, to be sure that the file are restored with the original owner/group.

mkdir s

cd s

tar xvf {path to the file}/s.tar

vi etc/init.d/usbarbitrator

(I did this change :

USBARBITRATOR_SCHED_PARAM="--verbose ++group=usbArbitrator"

)

touch -r etc/init.d/cdp etc/init.d/usbarbitrator
# it seems that the order of the file is critical

tar tf {path to the file}/s.tar >/tmp/s.filelist

tar -cv -T /tmp/s.filelist  --no-recursion --same-order -f {path to the file}/s1.tar

------------ send s1.tar back to the ESXi in the same folder than s.tar

On the ESXi :

vmtar -c s1.tar -o s1.vmtar -v

gzip -9 -c s1.vmtar >s1.v00

Now, a little check can be useful !

vmtar -t <s.v00 >s.txt

vmtar -t <s1.v00 >s1.txt

Back to the unix machine

retrieve the two files

We have to suppress the date and time :

cat s.txt | awk ' { $4="" ; $5="" ; print } ' >sc.txt

cat s1.txt | awk ' { $4="" ; $5="" ; print } ' >s1c.txt

diff sc.txt s1c.txt

183c183

< -r-xr-xr-x 201/201 4483   etc/init.d/usbarbitrator

---

> -r-xr-xr-x 201/201 4493   etc/init.d/usbarbitrator

OK, only one change, hat is what we expected Smiley Happy

and now, tadaa !!!

cp s1.v00 /bootbank/s.v00

reboot (and cross your fingers !)

It works for me !

Thanks,

Pascal

EduLima22
Contributor
Contributor

Thanks Pascal,

I'll try this solution as soon as possible!

However, will this change may be undone after an update?

The main point is the lack of information about the USB devices on vCenter.

I bypassed this limitation through the script below. I'm not sure it will work in all scenarios.

Not my best Perl script but it has worked very well.

--x--

#!/usr/bin/perl

use strict;
use warnings;
use VMware::VILib;
use VMware::VIRuntime;
use Net::SSH::Expect;
#$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;

############################################################################
sub host_ssh_manager($$) {
     my $h = shift;
     my $state = shift;
     if ($state) {
          $state = "starting";
     } else {
          $state = "stopping";
     }
     my $servicemanager = Vim::get_view(
          mo_ref => $h->get_property('configManager.serviceSystem')
     );
     my $hostservices = $servicemanager->serviceInfo->service;
     foreach my $s (@$hostservices) {
          if ($s->label eq 'SSH') {
               if (! $s->running) {
                    eval {
                         if ($state eq 'starting') {
                              $servicemanager->StartService(id => $s->key);
                              print "SSH started.\n";
                         }
                    };
               } else {
                    eval {
                         if ($state eq 'stopping') {
                              $servicemanager->StopService(id => $s->key);
                              print "SSH stopped.\n";
                         }
                    };
               }
               if ($@) {
                    print "Error $state SSH service.\n";
               }
               last;
          }
     }
}

############################################################################
sub main() {
     my %opts = (
          host => {
               type => "=s",
               help => "Target Host",
               required => 0,
          },
          cluster => {
               type => "=s",
               help => "Target Cluster",
               required => 0,
          },
     );

     Opts::add_options(%opts);
     Opts::parse();
     Opts::validate();
     Util::connect();

     my $host_username;
     my $host_password;
     my $my_host  = Opts::get_option('host');
     $my_host =~ s/\%(..)/chr(hex($1))/gie if (defined($my_host));
     my $my_cluster  = Opts::get_option('cluster');
     $my_cluster =~ s/\%(..)/chr(hex($1))/gie if (defined($my_cluster));

     print "Host username: ";
     chop($host_username = );
     print "Host password: ";
     system("stty -echo");
     chop($host_password = );
     print "\n";
     system("stty echo");

     my $list_host;
     my $host_obj;

     # Host name is specified on command line
     if (defined($my_host)) {
          $host_obj = Vim::find_entity_view(
               view_type => 'HostSystem',
               properties => [
                    'name',
                    'runtime.powerState',
                    'vm',
                    'configManager.serviceSystem'
               ],
               filter => {'name' => "$my_host"},
          );
          if (defined($host_obj)) {
               if ($host_obj->get_property('runtime.powerState')->val
                    eq 'poweredOn') {
                    push @$list_host, $host_obj;
               }
          }
     }

     # Cluster name is specified on command line
     if (defined($my_cluster)) {
          my $cluster_obj = Vim::find_entity_view(
               view_type => 'ClusterComputeResource',
               properties => [
                    'name',
                    'host',
               ],
               filter => {'name' => "$my_cluster"},
          );
          if (defined($cluster_obj)) {
               print "Cluster: $my_cluster\n";
               my $cluster_hosts = $cluster_obj->host;
               foreach my $h (@$cluster_hosts) {
                    $host_obj = Vim::get_view(
                         mo_ref => $h,
                         properties => [
                              'configManager.serviceSystem',
                              'name',
                              'runtime.powerState',
                              'vm',
                         ]
                    );
                    if ($host_obj->get_property('runtime.powerState')->val
                         eq 'poweredOn') {
                         push @$list_host, $host_obj;
                    }
               }
          }
     }

     # Search for USB devices
     foreach my $h (@$list_host) {
          my $hostname = $h->name;
          print "Host: " . $hostname . "\n";

          # Search VMs
          my $list_vm = $h->vm;
          my %vms;
          my %paths;
          my %dongles;
          foreach my $vm_ref (@$list_vm) {
               my $vm_obj = Vim::get_view(
                    mo_ref => $vm_ref,
                    properties => [
                         'config.hardware.device',
                         'name',
                    ]
               );
               my $vm_name = $vm_obj->name;
               my $vm_devices = $vm_obj->{'config.hardware.device'};
               # Search USB devices
               foreach my $device (@$vm_devices) {
                    if($device->isa('VirtualUSB')) {
                         my $usbdevice_name = $device->backing->deviceName;
                         # Unfortunately, USB path (?) is the "primary key" for vCenter.
                         my $usbdevice_path =
                              ($usbdevice_name =~ /(path:\S+)/)[0];
                         if ($usbdevice_path =~ /path:(\d+)\/\d+\/(\d+)/) {
                              $usbdevice_path = "path:" . $1 . "/X/" . $2;
                         }
                         my $usbdevice_info = $device->deviceInfo->summary;
                         if (defined($usbdevice_path) &&
                              defined($usbdevice_info)) {
                              if (! $device->connected) {
                                   $vms{$vm_name}{$usbdevice_path} =
                                        "DEVICE NOT PRESENT!";
                              } else {
                                   $vms{$vm_name}{$usbdevice_path} = $usbdevice_info;
                              }
                              $paths{$usbdevice_path} = $vm_name;
                         }
                    }
               }
          }

          # Start sshd on host
          host_ssh_manager($h,1);

          # Search information on the host's logs by ssh command lines
          my $ssh = Net::SSH::Expect->new (
               host => "$hostname", 
               password=> "$host_password", 
               user => "$host_username", 
          );
          my $login_output = $ssh->login();
          my ($exec,$line,$date,$vm,$device,$available);
          my ($serial,$bus,$level,$port,$path);

          my $usblog = 1;
          my $usbdevices = 1;

          # Look at /dev/usbdevices.
          if ($usbdevices) {
               $exec = $ssh->send("cat /dev/usbdevices");
               $level = 0;
               while (defined($line = $ssh->read_line())) {
                    if ($line =~ /^T:\s+Bus=\s*(\d+)\sLev=\s*(\d+)\sPrnt=\s*(\d+)\sPort=\s*(\d+)\sCnt=\s*(\d+)\sDev#=\s*(\S+)\s/) {
                         $bus = $1;
                         $level = $2;
                         $port = $4;
                         if ($level > 1) {
                              $path = sprintf("path:%d/X/%d",$bus,$port);
                         } elsif ($level == 1) {
                              $path = sprintf("path:%d/%d",$bus,$port);
                         }
                         $device = $serial = "";
                         $available = 0;
                    }
                    elsif ($line =~ /V:\s+Available for Passthrough/) {
                         $available = 1;
                    }
                    elsif ($line =~ /Product=(.*)/) {
                         $device = $1;
                    }
                    elsif ($line =~ /SerialNumber=(.*)/) {
                         $serial = $1;
                    }
                    elsif ($line =~ /^E:/ && $path && $device && $available) {
                         $dongles{$path}{device} = $device;
                         $dongles{$path}{serial} = $serial;
                         $path = $device = $serial = $port = $bus = "";
                         $level = $available = 0;
                    }
               }
          }

          # Look at /var/log/usb.log (ESXi 5.0).
          if ($usblog) {
               $exec = $ssh->send("grep 'USBArb: Device' /var/log/usb.log");
               while (defined($line = $ssh->read_line())) {
                    $date = $vm = $device = $serial = $path = 0;
                    if ($line =~ /^([^\ ]+)/) {
                         $date = $1;
                    }
                    if ($line =~ /:name:(.*?) vid/) {
                         $device = $1;
                    }
                    if ($line =~ /(path:[^\ ]+) /) {
                         $path = $1;
                         if ($path =~ /path:(\d+)\/\d+\/(\d+)/) {
                              $path = "path:" . $1 . "/X/" . $2;
                         }
                    }
                    if ($line =~ /serialnum:([^\ ]+) /) {
                         $serial = $1;
                    }
                    if ($line =~ /owner:(\S+)/) {
                         $vm = $1;
                    }
                    if ($device && $path && $vm &&
                         defined($dongles{$path})
                    ) {
                              if ($vm !~ /\(null\)/) {
                                   $dongles{$path}{vm} = $vm;
                              } else {
                                   undef $dongles{$path}{vm};
                              }
                              $dongles{$path}{date} = $date;
                              $dongles{$path}{device} = $device;
                              $dongles{$path}{serial} = $serial;
                    }
               }
          }

          if (!%dongles) {
               print "\n";
               print "\tOBS: No dongles found in the host logs!";
               print "\n";
          }

          # Stop sshd
          host_ssh_manager($h,0);

          # Show the list of VMs with USB dongles
          foreach my $vm_name (sort keys %vms) {
               print "\tVM: " . $vm_name . "\n";
               foreach my $path (sort keys %{$vms{$vm_name}}) {
                    if (defined($dongles{$path}{vm})) {
                         if ($dongles{$path}{vm} eq $vm_name) {
                              print "\t\tPath:   " . $path;
                              print "\n";
                              print "\t\tDongle: " . $dongles{$path}{device};
                              print "\n";
                              print "\t\tSerial: " . $dongles{$path}{serial};
                              print "\n";
                              print "\t\tConnected in: " . $dongles{$path}{date};
                              print "\n";
                              print "\n";
                         } else {
                              print "\t\t>>> Dongle not found in the host logs.";
                              print "\n";
                              print "\t\tPath:   " . $path;
                              print "\n";
                              print "\t\tDongle: " . $vms{$vm_name}{$path};
                              print "\n";
                              print "\n";
                         }
                    } else {
                         print "\t\t>>> Dongle not found in the host logs.";
                         print "\n";
                         print "\t\tPath:   " . $path;
                         print "\n";
                         print "\t\tDongle: " . $vms{$vm_name}{$path};
                         print "\n";
                         if (defined($dongles{$path}{serial})) {
                              print "\t\tSerial: " . $dongles{$path}{serial};
                              print "\n";
                         }
                         print "\n";
                    }
               }
          }

          print "\n";
          print "\tDongles connected to the host but not associated with VMs:\n";
          foreach my $path (sort keys %dongles) {
               if (! defined($dongles{$path}{vm}) &&
                    ! defined($paths{$path})) {
                    print "\t\tPath:   " . $path;
                    print "\n";
                    print "\t\tDongle: " . $dongles{$path}{device};
                    print "\n";
                    print "\t\tSerial: " . $dongles{$path}{serial};
                    print "\n";
                    if (defined($dongles{$path}{date})) {
                         print "\t\tConnected in: " . $dongles{$path}{date};
                         print "\n";
                    }
                    print "\n";
               }
          }
          print "\n";
     }

     Util::disconnect();
}

main();
#==eof==
Reply
0 Kudos
sibsib
Contributor
Contributor

Hi EduLima22,

As I was on my run test (The whole is done at home, and, to have a test ESXI, I change the drive of one of my PC Smiley Happy ), I tried that also. I installed the latest patch for ESXi 5.5, and after update, I had to rerun the complete process to have again the log verbosity increased.

The whole process took me around a quarter of an hour, and could be scripted. Moreover, if you need to do the change on many ESXi, most of the work is to be done once.

I completely agree with you, the biggest problem is the lack of informations. For example, we can see that the usbarbotrator search for customisation informations in two files at boot. If we only know the possible options... Maybe increasing the log verbosity is just a matter of creating a configuration file!

About your script : as far as I know, the free version of ESXi only give a read only access to the API. So it's not a solution for me, at least at home (at work, this is another story! I would probably write a bit of orchestrator to deal with that!)

Regards,

Pascal

Reply
0 Kudos
sibsib
Contributor
Contributor

Hi Edulima22,

Just for the record :

I did all my tests on my personal desktop. I did store my modified S.V00 files, of course. Last friday, I reinstall my 'personal production' server in VMware 5.5 (I didn't upgrade as my boot disk just died, hence the version change).

I just copied back the modified S.V00 file on this server, and since, everything is OK.

So, I would say that for me, this is a working solution.

Regards,

Pascal

Reply
0 Kudos