nimo1983's Posts

Thanks James. Had issues with storage, which is sorted. Have spread the workload VMs now. The run is still failing. Attached is the log. Anything obvious in this one. Service Requested Stop: ExceptionFound: SVMotion TargetDatastore Exception : Review Log and Configuration : :Target and Source Datastores are the same

Thanks James. The output I attached is for the failed run for DS3DB0 and it gets stuck at ds3 everytime. Here is the df output. [root@DS3DB0 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos-root 14G 5.4G 8.5G 39% / devtmpfs 24G 0 24G 0% /dev tmpfs 24G 0 24G 0% /dev/shm tmpfs 24G 8.8M 24G 1% /run tmpfs 24G 0 24G 0% /sys/fs/cgroup /dev/sdc1 99G 89G 4.5G 96% /ds3 /dev/sdb1 246G 196G 38G 84% /var/lib/mysql /dev/sda1 497M 349M 149M 71% /boot tmpfs 4.8G 0 4.8G 0% /run/user/0 [root@DS3DB0 ~]# it runs fine when it is 1 tile, but when we add 2 or 3 tiles to the run, it throws a timeout. And always happen with DS3DB VMs when tiles number are increased. Attaching results for 2 recent runs with 1 tile (guestinfo working) and 2 tiles (guestinfo failing for DS3DB0) Any pointers before hitting the weekend. Regards, Niyaz

Thanks James. Ran it with 1 tile and it did work. Redeploying the tiles 1 & 2 now and rerunning. hit similar issue here with MiscInfotimeout too (I have updated another thread for another setup similar issue for MiscInfotimeout). It gets stuck at collecting info for ds3. /root/ds3/: --------------------------------------------------------------------------------------------------------------- /ds3/: ============================================================================== In both the cases, I have used database backup and restore script. is there any issue with restoring DB from snapshot. Regards, Niyaz  

Thanks Rebecca and happy New year. The MiscInfoTimeOut did help, however in subsequent attempts it failed. So I tried running it directly without STAX client It gets stuck at collecting info for ds3. /root/ds3/: --------------------------------------------------------------------------------------------------------------- /ds3/: ============================================================================== I tried running the same from prime client directly (ssh ds3db0 perl "/root/VMmark3/tools/VMmark3-GuestInfo.pl /root/DS3DB0-GuestInfo.txt") than through STAX UI, it succeeds afte4 16 mins, but with STAX it fails even after 30 mins. Anything specific we shd look at.

Thanks James. MTU is set on all hosts and time is synced for standby* VMs. Ran again with 3 hosts and 4 tiles this time and still it fails with the same error. Could not Run the following 8 Wklds: ['DS3WebA Tile0 fa iled to start/complete process', 'DS3WebA Tile3 failed to start/complete process', 'DS3WebC Tile0 failed to st art/complete process', 'DS3WebB Tile0 failed to start/c omplete process', 'DS3WebC Tile1 failed to start/comple te process', 'DS3WebB Tile3 failed to start/complete pr ocess', 'DS3WebC Tile3 failed to start/complete process ', 'deploy Failed Run Phase'] Checking the wrf files, it has too many failures. Error during browse for reviews returned by web server: The request timed out Thread 15: Error in Login for User newuser48069702, failure 1, retrying Error during browse for reviews returned by web server: The request timed out Thread 18: Error in Login for User newuser166974341, failure 1, retrying Error during browse for reviews returned by web server: The request timed out Thread 21: Error in Login for User user23566746, failure 1, retrying Error during browse for reviews returned by web server: The request timed out Thread 17: Error in Login for User user133715143, failure 1, retrying Error during browse for reviews returned by web server: The request timed out Thread 13: Error in Login for User user152089543, failure 1, retrying Error during browse for reviews returned by web server: The request timed out Thread 4: Error in Login for User newuser185687334, failure 1, retrying Thread 0: exiting Thread 11: error in parsing response from browse reviews request Thread 11: Error in Login for User user51878161, failure 2, retrying Unhandled Exception: System.ObjectDisposedException: Cannot access a disposed object. Object name: 'System.Net.Sockets.NetworkStream'. at System.Net.WebConnection.BeginRead (System.Net.HttpWebRequest request, System.Byte[] buffer, System.Int32 offset, System.Int32 size, System.AsyncCallback cb, System.Object state) [0x0002e] in <59be416de143456b88b9988284f43350>:0 at System.Net.WebConnectionStream.BeginRead (System.Byte[] buffer, System.Int32 offset, System.Int32 size, System.AsyncCallback cb, System.Object state) [0x001b2] in <59be416de143456b88b9988284f43350>:0 at System.Net.WebConnectionStream.Read (System.Byte[] buffer, System.Int32 offset, System.Int32 size) [0x00007] in <59be416de143456b88b9988284f43350>:0 at System.IO.StreamReader.ReadBuffer (System.Char[] userBuffer, System.Int32 userOffset, System.Int32 desiredChars, System.Boolean& readToUserBuffer) [0x0003c] in <dca3b561b8ad4f9fb10141d81b39ff45>:0 at System.IO.StreamReader.Read (System.Char[] buffer, System.Int32 index, System.Int32 count) [0x0009d] in <dca3b561b8ad4f9fb10141d81b39ff45>:0 at ds2xdriver.ds2Interface.ds2browsereview (System.String browse_review_type_in, System.String get_review_category_in, System.String get_review_actor_in, System.String get_review_title_in, System.Int32 batch_size_in, System.Int32 customerid_out, System.Int32& rows_returned, System.Int32[]& prod_id_out, System.String[]& title_out, System.String[]& actor_out, System.Int32[]& review_id_out, System.String[]& review_date_out, System.Int32[]& review_stars_out, System.Int32[]& review_customerid_out, System.String[]& review_summary_out, System.String[]& review_text_out, System.Int32[]& review_helpfulness_sum_out, System.Double& rt) [0x00139] in <d329ccf9df0d44c493533f20c2907579>:0 at ds2xdriver.User.Emulate () [0x00abb] in <d329ccf9df0d44c493533f20c2907579>:0 at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) [0x00017] in <dca3b561b8ad4f9fb10141d81b39ff45>:0 at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x0008d] in <dca3b561b8ad4f9fb10141d81b39ff45>:0 at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <dca3b561b8ad4f9fb10141d81b39ff45>:0 at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) [0x00031] in <dca3b561b8ad4f9fb10141d81b39ff45>:0 at System.Threading.ThreadHelper.ThreadStart () [0x0000b] in <dca3b561b8ad4f9fb10141d81b39ff45>:0 [ERROR] FATAL UNHANDLED EXCEPTION: System.ObjectDisposedException: Cannot access a disposed object. Object name: 'System.Net.Sockets.NetworkStream'. at System.Net.WebConnection.BeginRead (System.Net.HttpWebRequest request, System.Byte[] buffer, System.Int32 offset, System.Int32 size, System.AsyncCallback cb, System.Object state) [0x0002e] in <59be416de143456b88b9988284f43350>:0 at System.Net.WebConnectionStream.BeginRead (System.Byte[] buffer, System.Int32 offset, System.Int32 size, System.AsyncCallback cb, System.Object state) [0x001b2] in <59be416de143456b88b9988284f43350>:0 at System.Net.WebConnectionStream.Read (System.Byte[] buffer, System.Int32 offset, System.Int32 size) [0x00007] in <59be416de143456b88b9988284f43350>:0 at System.IO.StreamReader.ReadBuffer (System.Char[] userBuffer, System.Int32 userOffset, System.Int32 desiredChars, System.Boolean& readToUserBuffer) [0x0003c] in <dca3b561b8ad4f9fb10141d81b39ff45>:0 at System.IO.StreamReader.Read (System.Char[] buffer, System.Int32 index, System.Int32 count) [0x0009d] in <dca3b561b8ad4f9fb10141d81b39ff45>:0 at ds2xdriver.ds2Interface.ds2browsereview (System.String browse_review_type_in, System.String get_review_category_in, System.String get_review_actor_in, System.String get_review_title_in, System.Int32 batch_size_in, System.Int32 customerid_out, System.Int32& rows_returned, System.Int32[]& prod_id_out, System.String[]& title_out, System.String[]& actor_out, System.Int32[]& review_id_out, System.String[]& review_date_out, System.Int32[]& review_stars_out, System.Int32[]& review_customerid_out, System.String[]& review_summary_out, System.String[]& review_text_out, System.Int32[]& review_helpfulness_sum_out, System.Double& rt) [0x00139] in <d329ccf9df0d44c493533f20c2907579>:0 at ds2xdriver.User.Emulate () [0x00abb] in <d329ccf9df0d44c493533f20c2907579>:0 at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) [0x00017] in <dca3b561b8ad4f9fb10141d81b39ff45>:0 at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x0008d] in <dca3b561b8ad4f9fb10141d81b39ff45>:0 at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <dca3b561b8ad4f9fb10141d81b39ff45>:0 at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) [0x00031] in <dca3b561b8ad4f9fb10141d81b39ff45>:0 at System.Threading.ThreadHelper.ThreadStart () [0x0000b] in <dca3b561b8ad4f9fb10141d81b39ff45>:0 DVDstore_End_Iteration_Number 0 Any tip on how to troubleshoot this issue further and get a successful run? Or could this be a problem with the environment? Your help is greatly appreciated. Regards, Niyaz

Thanks Rebecca. I shall try with Misc timeout value. Here is the results folder for latest run and failure for deployVM0. It fails with CPU or RAM. This time the error was - Unable to HotAddMemoryDeployVM0 to sue new memory settings. (result folder attached). Looking forward to hearing from you soon so as to get a valid complete 3 hour run. Regards, NiMo  

Hello Rebecca and all, We have re-created the env with private IPs. A turbo run succeeded just fine, but fails for 3 hour run. 20211214-15:47:48 Info Guest Info failures for the following machines: 4 machines : DS3DB0 DS3DB3 DS3DB1 DS3DB2 Error Messages DS3DB0= :DS3DB3= :DS3DB1= :DS3DB2= : Primeclient to DS3DB* connectivity is all working well. If I trigger it with 3 tiles, it will run further, but then failed with another error - 20211214-13:31:43 Info Service Requested Stop: ExceptionFound: Unable to Update DeployVM0 to use new CPUS:CPUS: 8 20211214-13:31:43 Info VMmark3: Service Requested Stop: ExceptionFound: Unableto Update DeployVM0 to use new CPUS:CPUS: 8 : 3630.41(secs) Note: I have attached the logs for Guest info failure run. Please provide inputs as this is holding a project rollout. Regards, NiMo

Thanks Rebecca. I had started deployment using private IPs and Prime Client is now assigned with 2 IPs. Shall keep you posted about the progress once tile0 is progressed. In the meantime, I have one more q - How can I run the job with workload Ops only and no infraOps. Regards, NiMo