VMware Performance Community
krishnadc
Enthusiast
Enthusiast
Jump to solution

STAX Job stuck at "Tile 0:AuctionLB:Restore for 9500,1500 Users" for 3 hours

Hi Team,

I've started the VMmark run (with only WorkloadList) and the STAX job has been stuck at "AuctionLB: Restore" step for about 3 hours.

VMmark Initialization and Set-up steps worked fine and they were shown as "Pass" in the STAX console.

Attaching the STAX screen-shot here:

STAX-Console.JPG

Also, attaching the archive of the complete Results output folder from the PrimeClient machine here.

Any help here is highly appreciated.

Thank you,

Krishna

1 Solution

Accepted Solutions
dmorse
VMware Employee
VMware Employee
Jump to solution

On a seperate note, if I want to run only "Infrastructure Operations", Can I set the "Workload List" to empty value and start the run?

I wasn't sure about this myself, so I tried a run with no workloads and just infrastructure operations, and it failed when trying to produce a score.  So no, you need to run 1 or more workloads for a valid score to be produced.

View solution in original post

Reply
0 Kudos
22 Replies
krishnadc
Enthusiast
Enthusiast
Jump to solution

The run has been stuck in this step forever.

Can someone let me know if I can send any other logs/info for this issue?

Thanks,

Krishna

Reply
0 Kudos
dmorse
VMware Employee
VMware Employee
Jump to solution

Hi Krishna,

Please run this command from the PrimeClient VM:

ssh client0 cat /Client0_restore.txt > /root/client0_restore_stuck.txt

Then attach the /root/client0_restore_stuck.txt to this thread.

Thanks,

David

krishnadc
Enthusiast
Enthusiast
Jump to solution

Thanks dmorse​ for your response and Sorry for the delay in sending the log file.

Surprisingly, the VMmark run has been completed and it gave the Score file now. But I'm sure it took atleast 40 hours to get past the "Tile 0:AuctionLB:Restore for 9500,1500 Users" step.

I've got the Client0_restore.txt file and attaching it here. It would be very helpful for me to understand any reason on why it took so long for this step (or is it normal for this step?).

Thank you,

Krishna

Reply
0 Kudos
dmorse
VMware Employee
VMware Employee
Jump to solution

Hi Krishna,

No, this is definitely not typical.  I'd like to get some more information about your environment:

  • Can you run this command from the prime client VM and send the output:
    ssh AuctionLB0 cat /root/version.txt
  • Can you zip up the contents of the Results_2020... directory under /root/VMmark3/results on the prime client VM that finally finished after 40+ hours, and attach that as well?

Thanks,

David

Reply
0 Kudos
krishnadc
Enthusiast
Enthusiast
Jump to solution

Thanks David for your response.

I've bring down the System-Under-Test set-up to run a different test, so could not get the previous /root/version.txt

I've started another run in the same environment and it looks like it has stuck in the same step for about 11 hours now.

Please let me know if I can I send the required logs now? or once the test run is completed.

Best,

Krishna

Reply
0 Kudos
dmorse
VMware Employee
VMware Employee
Jump to solution

Hi Krishna,

OK, then yes please rerun this command from the prime client:

ssh AuctionLB0 cat /root/version.txt

Also, can you verify the filename of your .ova template?  Is it vmmark3.1.1-template-031420.ova?

Thanks,

David

Reply
0 Kudos
krishnadc
Enthusiast
Enthusiast
Jump to solution

Hi David,

Please find the output of ssh AuctionLB0 cat /root/version.txt below:

053117

yum update run 102616

weathervane 1.0.13

VMmark3 05312017-001

The VMmark template name is : vmmark3-template-053117.ova

Thanks,

Krishna

Reply
0 Kudos
dmorse
VMware Employee
VMware Employee
Jump to solution

Hi Krishna,

Okay, this is the root cause of this issue.  You're using VMmark 3.0, which was released over 3 years ago.

Please download the latest version of VMmark, version 3.1.1, here:

https://www.vmware.com/go/download-vmmark

The latest/correct template to use is named vmmark3.1.1-template-031420.ova.  You'll need to delete all the old VMs and recreate your environment.

Thanks,

David

krishnadc
Enthusiast
Enthusiast
Jump to solution

Thanks a lot David for looking into this.

I'll provision the Clients and VMs with this new template from my next test run and update this thread. (the current test is about to complete, so I'm letting it to continue).

On a seperate note, if I want to run only "Infrastructure Operations", Can I set the "Workload List" to empty value and start the run?

Best,

Krishna

Reply
0 Kudos
dmorse
VMware Employee
VMware Employee
Jump to solution

On a seperate note, if I want to run only "Infrastructure Operations", Can I set the "Workload List" to empty value and start the run?

I wasn't sure about this myself, so I tried a run with no workloads and just infrastructure operations, and it failed when trying to produce a score.  So no, you need to run 1 or more workloads for a valid score to be produced.

Reply
0 Kudos
krishnadc
Enthusiast
Enthusiast
Jump to solution

Thanks David for your response.

Reply
0 Kudos
Avaakaford
Enthusiast
Enthusiast
Jump to solution

@dmorse,

 

I'm facing similar issue on vmmark 3.1.1 , May i know workaround or resolution for this one . I'm able to run tile0 to tile5 2 dyas back,somehow i struck with this issue now.

 

Thanks & Regards

Ajay

Reply
0 Kudos
Avaakaford
Enthusiast
Enthusiast
Jump to solution

@dmorse,

 

Here required files.

 

Thanks & Regards

Ajay

 

Reply
0 Kudos
fredab2
VMware Employee
VMware Employee
Jump to solution

More information is in file "CleanData_W1I1.log". Please upload that file or just .tgz the entire run dir and upload it.

Reply
0 Kudos
Avaakaford
Enthusiast
Enthusiast
Jump to solution

@fredab2 

 

can you please guide me, where this file is located (CleanData_W1I1.log)

 

Thanks & Regards

Ajay

Reply
0 Kudos
fredab2
VMware Employee
VMware Employee
Jump to solution

Per VMmark document:

VMmark Results Files

After a VMmark benchmark test run has completed, the VMmark harness automatically collects the results data from the client system (or systems) and places those results in the results directory on the prime client, then processes the results to score the test.

Look under /root/VMmark3 for results directories created after each run.

Reply
0 Kudos
Avaakaford
Enthusiast
Enthusiast
Jump to solution

didn't find it . Here one of my successfull run results

 

Avaakaford_0-1663096507448.png

 

Thanks & Regards

Ajay

 

Reply
0 Kudos
fredab2
VMware Employee
VMware Employee
Jump to solution

Have you tried restarting your clients and VM's for tile0 and then try a turbo run and see if it is fine.  I will look further for where (probably on client) CleanData_W1I1.log file is located.

 

Reply
0 Kudos
Avaakaford
Enthusiast
Enthusiast
Jump to solution

Yes, i restarted client and all the tile0 vm's . Still struck at same state

Screen Shot 2022-09-14 at 1.56.01 PM.png

 

Thanks & Regards

Ajay

Reply
0 Kudos