nimo1983
Contributor
Contributor
‎07-04-2023 08:52 AM

VMmark test failing with Trust level 5 required for the PROCESS service's START request

Hello Team,

Running VMMark 3.1.1 against a 4 host setup.

Num of hosts: 4
ESXi version: VMware ESXi, 8.0.1, 21495797
Processor Type: Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz
Logical Processors: 64 (per host)
Prime client running with a single NIC. The client VMs and VMmark VMs are running on the same cluster.
No of tiles: 2
Storage : vSAN

When running the test, it fails with the below error:

The process failed to start. RC: 25, STAFResult: Trust level 5 required for the PROCESS service's START request

The host file is updated correctly and VMs are reachable. Trust level is 5 on all VMs.

Attachments: Results folder including the VMMark properties files.

Looking forward for expert inputs to get over this and complete the testing.

Regards,

Niyaz

 

Results_20230704153518_2Tile-20230704.zip
Reply
0 Kudos
jpschnee
VMware Employee
VMware Employee
‎07-05-2023 09:25 AM

I'm not really seeing a lot in the results you uploaded.  Here are a few suggestions to see if we can make some progress.

  1. Reboot all VMs (including your prime client and client0) and try again.
  2. If you are still seeing an issue: Although you said you only have one NIC on your primeclient I see two IP addresses.  You may want to review your networking configuration.  Additionally, it would help if you went ahead and posted your Standby0's /etc/hosts file and it's /usr/local/staf/bin/STAF.cfg file for us to review.
  3. You could also add the alias "primeclient" to the primeclient's /etc/hosts file (with IP 127.0.0.1)
-Joshua
Reply
0 Kudos
BhojaRavi
Contributor
Contributor
‎07-10-2023 07:03 AM

Thanks Joshua for your help reviewing this issue.


Reboot all VMs (including your prime client and client0) and try again.

Reboot done on all VMs including PrimeClient & Client0

If you are still seeing an issue: Although you said you only have one NIC on your primeclient I see two IP addresses. You may want to review your networking configuration. Additionally, it would help if you went ahead and posted your Standby0's /etc/hosts file and it's /usr/local/staf/bin/STAF.cfg file for us to review.

PrimeClient has 2 NICs. One of them on the networks connected to the subnet hosting all the VMmark VMs.

Here is the STAF.cfg file(standby0):

[root@primeclient ~]# ssh standby0 "cat /usr/local/staf/bin/STAF.cfg"
# Turn on tracing of internal errors and deprecated options
trace enable tracepoints "error deprecated"

# Enable TCP/IP connections
interface ssl library STAFTCP option Secure=Yes option Port=6550
interface tcp library STAFTCP option Secure=No option Port=6500

# Set default local trust
trust machine local://local level 5
trust machine primeclient level 5

# Add default service loader
serviceloader library STAFDSLS
trust machine 172.21.169.22 level 5
trust machine 172.21.169.17 level 5
trust machine 172.21.169.11 level 5
trust machine 172.21.169.21 level 5
trust machine 172.21.169.12 level 5
trust machine 172.21.169.26 level 5
trust machine 172.21.169.24 level 5
trust machine 172.21.169.18 level 5
trust machine 172.21.169.19 level 5
trust machine 172.21.169.28 level 5
trust machine 172.21.169.23 level 5
trust machine 172.21.169.16 level 5
trust machine 172.21.169.13 level 5
trust machine 172.21.169.27 level 5
trust machine 172.21.169.15 level 5
trust machine 172.21.169.29 level 5
trust machine 172.21.169.25 level 5
trust machine 172.21.169.20 level 5
trust machine 172.21.169.14 level 5
trust machine 172.21.169.30 level 5
trust machine 172.21.166.* level 5

You could also add the alias "primeclient" to the primeclient's /etc/hosts file (with IP 127.0.0.1)

This is done as well.

 

Reply
0 Kudos
jpschnee
VMware Employee
VMware Employee
‎07-10-2023 08:50 AM

Great, can you please set 'DebugLevel=1' in your properties file then zip up and share your latest results directory so I can review?

-Joshua
Reply
0 Kudos
BhojaRavi
Contributor
Contributor
‎07-10-2023 10:35 AM

Attaching the results file after setting Debug level to 1.

Thanks in advance.

Results_20230710173158_2Tile-Test-20230710.zip
Reply
0 Kudos
jpschnee
VMware Employee
VMware Employee
‎07-10-2023 11:03 AM

I believe the issue is because of that second network on the prime client that I mentioned:

20230710-17:32:17 Info Info: FS Copy Staf.cfg File : Standby0 Returned: RC = 0, STAFResult = Trust level 4 required for the FS service's COPY request Requester has trust level 3 on machineStandby0.hmcdc.local Requesting machine: ssl://172.21.169.225 (ssl://172.21.169.225) Requesting user : none://anonymous
 
Your prime client has both 172.21.166.225 and 172.21.169.225 IP addresses but Standby's Staf.cfg doesn't have an appropriate trust machine line for 172.21.169.225.  As you can see above, the requestor is listed as 172.21.169.225 so that would seem to be the issue.  You either need to have the workload VMs trust 172.21.169.225 or you need to have the prime client communicate to the workload VMs via 172.21.166.225 since that IP is trusted.
-Joshua
Reply
0 Kudos