VMware Performance Community
mathieugont
Enthusiast
Enthusiast

Does load the data into the DS3DB0 database need access to the Internet?

Dear community.

I have provisioned my tile0. Among others, the virtual machine DS3DB0 is created on the cluster referenced by vCServerVMmark3Cluster.

The User's guide says "The provisioning process initiates creation of the DS3DB0 virtual machine, which can take 12 hours or more while data is loaded into the DS3DB0 database. This procedure is not considered part of the provisioning process.", and some threads on vmware.community mention more than 36h.

Does load the data into the DS3DB0 database need access to the Internet?

If yes... My ESXi hosts and thus my VMs have access to the Internet only if I set proxies. So, once DS3DB0 is provisioned and I have set up my proxies, how could I load the data into the database: is there a script on DS3DB0 or a procedure from VMmark3.properties?

If no... I must be patient but I am wondering how can I check the progress? When is hosts-stub.txt generated (I guess after loading the data because I have never seen this file)?

Thanks a lot for your support

Mat.

Reply
0 Kudos
15 Replies
fredab2
VMware Employee
VMware Employee

You do not need an internet connection.  You can monitor the progress of DS3 DB creation by following instructions in VMmark3 Users Guide.

Chapter 3, Preparing the Infrastructure for VMmark 3.1 Benchmark Tests, "Provision VMmark Tiles", "Creating the First VMmark Tile (Tile 0)" , paragraph number 6 (on page 50).

6.  Wait for the DS3DB0 creation process to complete; depending on your environment, this can take 12 hours or more.To determine if it’s complete, follow these steps:

a. In a terminal window on the prime client, run the following command:

    ssh  ‘tail -f /ds3/VMmark3-DS3installoutput.txt’

    (where  is the IP address of the DS3DB0 VM, which can be found in the hosts-stub.txt file referenced above).

b. Watch for the following text: VMmark3 DS3DB Setup Completed : Rebooting

Fred

Reply
0 Kudos
mathieugont
Enthusiast
Enthusiast

Hi Fred

Thanks for your answer.

I am thus puzzled.

It is not the first time I am provisioning the VMs running java -jar tools/VMmark3Service.jar -d 9 -c VMmark3.tile0.properties

but even after 72 hours, /des3/ is empty on DS3DB0 and the file hosts-stub.txt have not been generated.

Find in attachment my files VMmark3-ProvisioningService.log and VMmark3.tile0.properties.

Can you please let me know if you see something wrong?

This time, I can see there is a problem with thne VM Standby0. I must still work on that.

Mat.

Reply
0 Kudos
mathieugont
Enthusiast
Enthusiast

Here more info ...

Login on DS3DB0, I can run

>>> cd /root/VMmark3/client/DS3DB

>>> ./DS3_checker.sh

Installing checker files

ds3db_100GB.sh

tmp_tables/

tmp_tables/customers.txt

tmp_tables/inventory.txt

tmp_tables/categories.txt

tmp_tables/custhist.txt

tmp_tables/orderlines.txt

tmp_tables/reviews.txt

tmp_tables/reviews_helpfulness.txt

tmp_tables/membership.txt

tmp_tables/products.txt

tmp_tables/orders.txt

Starting DS3 checker

Mon May  6 11:48:58 CEST 2019

============================================================================

============================================================================

====       DS3 database chacker must be run from DS3 host node          ====

====            Currently running on host = DS3DB0

============================================================================

PING DS3DB0.vmmark3.eng.vmware.com (127.0.0.1) 56(84) bytes of data.

64 bytes from DS3DB0.vmmark3.eng.vmware.com (127.0.0.1): icmp_seq=1 ttl=64 time=0.058 ms

--- DS3DB0.vmmark3.eng.vmware.com ping statistics ---

1 packets transmitted, 1 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 0.058/0.058/0.058/0.000 ms

DS3DB0 is alive

mysqlshow: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)

./ds3db_100GB.sh: line 32: [: ==: unary operator expected

ERROR: Database  doesn't exist.

============================================================================

>>>

>>>   ./ConfigServer_DS3DB.sh 100

mysqlshow: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)

./ds3db_100GB.sh: line 32: [: ==: unary operator expected

And S3DB0, I can run

>>> service mysqld start

Redirecting to /bin/systemctl start  mysqld.service

never ends...

Reply
0 Kudos
mathieugont
Enthusiast
Enthusiast

Going a step further...

(please read up to the end to see the question)

I run

>>> cd /root/VMmark3/tools/

>>> ./make-ds3db.sh 100

And could see:

>>> service mysqld status

Redirecting to /bin/systemctl status  mysqld.service

● mysqld.service - MySQL Community Server

   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)

   Active: active (running) since Mon 2019-05-06 13:25:19 CEST; 8min ago

  Process: 3416 ExecStartPost=/usr/bin/mysql-systemd-start post (code=exited, status=0/SUCCESS)

  Process: 3399 ExecStartPre=/usr/bin/mysql-systemd-start pre (code=exited, status=0/SUCCESS)

Main PID: 3415 (mysqld_safe)

   CGroup: /system.slice/mysqld.service

           ├─3415 /bin/sh /usr/bin/mysqld_safe

           └─3750 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mysqld.log --open-files-limit=65535 --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysq...

May 06 13:25:17 DS3DB0 systemd[1]: Starting MySQL Community Server...

May 06 13:25:17 DS3DB0 mysqld_safe[3415]: 190506 13:25:17 mysqld_safe Logging to '/var/log/mysqld.log'.

May 06 13:25:17 DS3DB0 mysqld_safe[3415]: 190506 13:25:17 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql

May 06 13:25:19 DS3DB0 systemd[1]: Started MySQL Community Server.

Which is already an improvement.

Then I run:

>>> /root/VMmark3/client/DS3DB/

>>> ./ConfigServer_DS3DB.sh

And I could read in /DS3DP0_restore.txt

01:25:12 PM: Starting DS3DB0 restore v. 03022017, writing to /DS3DB0_restore.txt

[...]

Starting DS3 checker

^[[3;J^[[H^[[2JMon May  6 13:25:13 CEST 2019

============================================================================

============================================================================

====       DS3 database chacker must be run from DS3 host node          ====

====            Currently running on host = DS3DB0

============================================================================

PING DS3DB0.vmmark3.eng.vmware.com (127.0.0.1) 56(84) bytes of data.

64 bytes from DS3DB0.vmmark3.eng.vmware.com (127.0.0.1): icmp_seq=1 ttl=64 time=0.043 ms

--- DS3DB0.vmmark3.eng.vmware.com ping statistics ---

1 packets transmitted, 1 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 0.043/0.043/0.043/0.000 ms

DS3DB0 is alive

ERROR: Database #mysql50#lost+found doesn't exist.

============================================================================

01:25:15 PM: DS3 checker script finished..

01:25:15 PM: Stopping mysqld.

01:25:17 PM: Deleted logs.

01:25:19 PM: Successfully started mysqldn01:25:59 PM: Unable to ping DS3WebA0. ping: unknown host DS3WebA0

01:26:39 PM: Unable to ping DS3WebB0. ping: unknown host DS3WebB0

01:27:19 PM: Unable to ping DS3WebC0. ping: unknown host DS3WebC0

01:27:19 PM: DS3DB0 restore completed Mon May  6 13:27:19 CEST 2019

Errors:3, Errors:DS3DB0 unable to ping DS3WebA0. DS3DB0 unable to ping DS3WebB0. DS3DB0 unable to ping DS3WebC0. , Warnings:0, Warnings:None

I thus set /etc/hosts manually to map hostnames and ip adresses:

>>> cat /etc/hosts

127.0.0.1       DS3DB0.vmmark3.eng.vmware.com localhost localhost4 localhost4.localdomain4 DS3DB0

::1         localhost6 localhost6.localdomain6

172.22.197.***  DS3WebA0.vmmark3.eng.vmware.com DS3WebA0

172.22.197.***  DS3WebB0.vmmark3.eng.vmware.com DS3WebB0

172.22.197.***  DS3WebC0.vmmark3.eng.vmware.com DS3WebC0

172.22.197.***  DS3DB0.vmmark3.eng.vmware.com DS3DB0

And rerun the creation script:

>>> ./ConfigServer_DS3DB.sh

>>> cat /DS3DP0_restore.txt

[...]

DS3DB0 is alive

ERROR: Database #mysql50#lost+found doesn't exist.

============================================================================

01:47:51 PM: DS3 checker script finished..

01:47:51 PM: Stopping mysqld.

01:47:53 PM: Deleted logs.

01:47:57 PM: Successfully started mysqldn01:47:57 PM: Ping connectivity established with DS3WebA0.

01:47:57 PM: Ping connectivity established with DS3WebB0.

01:47:57 PM: Ping connectivity established with DS3WebC0.

01:47:57 PM: DS3DB0 restore completed Mon May  6 13:47:57 CEST 2019

Errors:0, Errors:None, Warnings:0, Warnings:None

I thus set /etc/hosts manually to map hostnames and ip addresses on all the VMs and the prime client.

From the prime client,

>>> ssh DS3DB0

>>> tail -f /ds3/VMmark3-DS3installoutput.txt

ERROR 1045 (28000): Access denied for user 'web'@'localhost' (using password: YES)

Loading Reviews data

Warning: Using a password on the command line interface can be insecure.

ERROR 1045 (28000): Access denied for user 'web'@'localhost' (using password: YES)

Loading Reviews Helpfulness Ratings Data

Warning: Using a password on the command line interface can be insecure.

ERROR 1045 (28000): Access denied for user 'web'@'localhost' (using password: YES)

Removing data files

Mon May  6 12:54:50 CEST 2019

VMmark3 DS3DB Setup Completed : Rebooting

It seems now ok, doesn't it?

Is there anything else to do manually on DS3DB0? Or any other VMs?

Reply
0 Kudos
fredab2
VMware Employee
VMware Employee

Based on that transcript it looks like you did get DS3DB0 VM ds3 db built doing manually:

>>> cd /root/VMmark3/tools/

>>> ./make-ds3db.sh 100

I don't understand what went wrong in provisioning script that it didn't do it.

You can now try a single tile run (I would do with turbo flag set in VMmark3.properties file - see documentation on Turbo).

Fred

Reply
0 Kudos
mathieugont
Enthusiast
Enthusiast

Hi Fred

I do not know neither.

But the most important is you seem to confirm I figured out how to finalize the procedure manually. So I can continue.

I will continue feeding this post to let the community know if at the end it works (or not).

To be continued...

Reply
0 Kudos
mathieugont
Enthusiast
Enthusiast

(no question here, thus no answer expected)

In the User's Guide, the following command is given to generate the file hosts-stub.txt.

java -jar tools/VMmark3Service.jar -c VMmark3.properties -m hostsFile -tiles 1

It completes the manual procedure.

Reply
0 Kudos
dmorse
VMware Employee
VMware Employee

mathieugont​ Every provisioning output folder should have a hosts-stub.txt, per this section in the User's Guide, so I am surprised you had to run that "-m hostsFile" command:

pastedImage_0.png

Looking back at your specific provisioning / properties file, you should have this file:

/root/VMmark3/provisioning-output/vmmark3-output-tile0-rerun1/hosts-stub.txt

Therefore, you'd run a command like this to populate your hosts file:

cat ~/VMmark3/provisioning-output/vmmark3-output-tile0-rerun1/hosts-stub.txt >> /etc/hosts

Please let us know if you don't see that file on your Prime Client.

Thanks,

David

Reply
0 Kudos
mathieugont
Enthusiast
Enthusiast

Dear community

Do you have remarks/ideas to finalize loading the DB on DS3DB0? Below the status...

In my previous message, I missed an error highlighting the DB was not properly loaded:

>>> ./ConfigServer_DS3DB.sh

>>> cat /DS3DP0_restore.txt

[...]

DS3DB0 is alive

ERROR: Database #mysql50#lost+found doesn't exist.

============================================================================

01:47:51 PM: DS3 checker script finished..

01:47:51 PM: Stopping mysqld.

01:47:53 PM: Deleted logs.

01:47:57 PM: Successfully started mysqldn01:47:57 PM: Ping connectivity established with DS3WebA0.

01:47:57 PM: Ping connectivity established with DS3WebB0.

01:47:57 PM: Ping connectivity established with DS3WebC0.

01:47:57 PM: DS3DB0 restore completed Mon May  6 13:47:57 CEST 2019

Errors:0, Errors:None, Warnings:0, Warnings:None

I thus logged on DS3DB0 and run it manually:

>>> cd /ds3

>>> nohup perl ./Install_VMmark_DVDStore.pl &

It works well, except it is now spending more than 24h loading the Customer History Data (even if DS3_cherker.sh tells me I am in the right direction):

>>> tail -n 50 /ds3/VMmark3-DS3installoutput.txt

Warning messages about using password via command line are normal......

Creating database

Warning: Using a password on the command line interface can be insecure.

Creating Indexes

Warning: Using a password on the command line interface can be insecure.

Creating Stored Procedures

Warning: Using a password on the command line interface can be insecure.

Loading Customer Data

Warning: Using a password on the command line interface can be insecure.

ERROR 1317 (70100) at line 7: Query execution was interrupted

Terminal close -- sending "KILL QUERY 26" to server ...

Terminal close -- query aborted.

Loading Orders Data

Warning: Using a password on the command line interface can be insecure.

Loading Orderlines Data

Warning: Using a password on the command line interface can be insecure.

Loading Customer History Data

Warning: Using a password on the command line interface can be insecure.

>>> /root/VMmark3/client/DS3DB

>>> ./DS3_checker.sh

Installing checker files

ds3db_100GB.sh

tmp_tables/

tmp_tables/customers.txt

tmp_tables/inventory.txt

tmp_tables/categories.txt

tmp_tables/custhist.txt

tmp_tables/orderlines.txt

tmp_tables/reviews.txt

tmp_tables/reviews_helpfulness.txt

tmp_tables/membership.txt

tmp_tables/products.txt

tmp_tables/orders.txt

Starting DS3 checker

Fri May 10 09:24:49 CEST 2019

============================================================================

============================================================================

====       DS3 database chacker must be run from DS3 host node          ====

====            Currently running on host = DS3DB0

============================================================================

PING DS3DB0 (172.22.197.204) 56(84) bytes of data.

64 bytes from DS3DB0 (172.22.197.204): icmp_seq=1 ttl=64 time=0.063 ms

--- DS3DB0 ping statistics ---

1 packets transmitted, 1 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 0.063/0.063/0.063/0.000 ms

DS3DB0 is alive

Database DS3 exists. Proceeding with DS3 table checks ...

============================================================================

Table CATEGORIES existance check passed

Table CUSTOMERS existance check passed

Table CUST_HIST existance check passed

Table INVENTORY existance check passed

Table MEMBERSHIP existance check passed

Table ORDERLINES existance check passed

Table ORDERS existance check passed

Table PRODUCTS existance check passed

Table REORDER existance check passed

Table REVIEWS existance check passed

Table REVIEWS_HELPFULNESS existance check passed

PASSED: All DS3 Tables were found to exist. Proceeding with DS3 checks ...

============================================================================

PASSED: DS3 Tables prelim count passed. Proceeding with DS3 checks ...

============================================================================

PASSED: DS3 appears to have web privileges granted. Proceeding with DS3 checks ...

============================================================================

Checking the contents of Tables = category

................All checks passwd checking Table CATEGORIES

Checking the contents of Tables = orders

........................All checks passwd checking Table ORDERS

Checking the contents of Tables = custhist

........ERROR:  DS3 value check failed.

============================================================================

--------------------------------------------------------------------------

FAIL:  DS3 is either not built completely or not in initialized state.

--------------------------------------------------------------------------

Any advised, idea?

Reply
0 Kudos
dmorse
VMware Employee
VMware Employee

Hi mathieugont

This (during database creation) is not a good sign:

Loading Customer Data

Warning: Using a password on the command line interface can be insecure.

ERROR 1317 (70100) at line 7: Query execution was interrupted

Terminal close -- sending "KILL QUERY 26" to server ...

Terminal close -- query aborted.

This error must be why the DS3 database checker failed while checking the customer history table:

Checking the contents of Tables = custhist

........ERROR:  DS3 value check failed.

============================================================================

--------------------------------------------------------------------------

FAIL:  DS3 is either not built completely or not in initialized state.

--------------------------------------------------------------------------

Unfortunately, this tells us your MySQL DS3DB0 database is "dirty", and must be rebuilt.

I just spoke with fredab2, and we both agree that your best bet is to rebuild the entire tile 0 from scratch, since creating the non-DS3DB VMs does not really

take very long.  Most of the tile0 creation time is in the DS3DB.

To do this, power off and delete-from-disk all [workloadname]0 VMs, and go back to "Creating the First VMmark Tile (Tile 0)" in the User's Guide.

Good luck, and let Fred and I know how it goes.

Thanks, David

Reply
0 Kudos
mathieugont
Enthusiast
Enthusiast

David, Fred

I totally agree with you: the best would be reinstall the tile 0.

But I have already done it 3 times and after each installation, DSDB0 did not loaded the data properly (DS3 does not exit in mysql).

I attached my properties fie in case I made a mistake.

Any other idea?

Reply
0 Kudos
mathieugont
Enthusiast
Enthusiast

I rerun the procedure to reinstall the tile0.

But I had to run manually the script make-ds3db.sh from /root/VMmark3/tools/@DS3DB0 to load the DB. 

Good new, after 24h, it has reached

Loading Customer History Data

Warning: Using a password on the command line interface can be insecure.

without the error mentioned earlier. It continue running.

Any comment?

Reply
0 Kudos
dmorse
VMware Employee
VMware Employee

Hi mathieugont

I rerun the procedure to reinstall the tile0.

But I had to run manually the script make-ds3db.sh from /root/VMmark3/tools/@DS3DB0 to load the DB.

Good new, after 24h, it has reached

Loading Customer History Data

Warning: Using a password on the command line interface can be insecure.

without the error mentioned earlier. It continue running.

Any comment?

I have asked about this issue internally with our VMmark engineering team.  This is very unusual, i.e. not something we have seen before.  The DS3DB0 creation should always start automatically, and the fact that you had to start it manually indicates something is wrong.  It's just not obvious to me what.

Can you elaborate on:

  • Server make/model (ESXi hosts)
  • Storage make/model (including types of vendor/capacity of your drives, whether they're HDDs [and if so what speed] or SSDs, and what RAID/LUN configurations they are in?  Are you using vSAN?
  • ESXi and vCenter version, including exact build numbers

Thanks, and apologies for all the trouble!

David

Reply
0 Kudos
dmorse
VMware Employee
VMware Employee

Hi mathieugont​ -- just checking in -- could you provide these details?

Can you elaborate on:

  • Server make/model (ESXi hosts)
  • Storage make/model (including types of vendor/capacity of your drives, whether they're HDDs [and if so what speed] or SSDs, and what RAID/LUN configurations they are in?  Are you using vSAN?
  • ESXi and vCenter version, including exact build numbers

Thanks, David

Reply
0 Kudos
mathieugont
Enthusiast
Enthusiast

Hi David

Like you already know, I run the load procedure manually.

It finally ends successfully after almost 4 days.

The only think I cannot explain is why this procedure was not automatically launched after the provisioning (/ds3 was empty).

Here my system:

Server make/model (ESXi hosts)

- Dual socket BDW E5-2699v4 servers

- 780GB DRAM per server

- 1x local ATA disk (1.8TB HDD) per server (Western Digital?), partition format GPT

- ESXi-6.7.0.13006603

- Each server's HDD is a datastore based on VMFS-6.82

- no vSan (i guess)

- vSphere Client 6.7.0.30000

- All my clients are on a hardware server (ESXi001), except the tile and the prime clients hosted on another one (ESXi005).

This thread is not my priority anymore, but if it can help the community, we can continue figuring out the why.

The other one

VMmark: staf returns RC = 17 (file-system problem)'

is because I would like to run weathervane asap.

Reply
0 Kudos