Dear community.
I have provisioned my tile0. Among others, the virtual machine DS3DB0 is created on the cluster referenced by vCServerVMmark3Cluster.
The User's guide says "The provisioning process initiates creation of the DS3DB0 virtual machine, which can take 12 hours or more while data is loaded into the DS3DB0 database. This procedure is not considered part of the provisioning process.", and some threads on vmware.community mention more than 36h.
Does load the data into the DS3DB0 database need access to the Internet?
If yes... My ESXi hosts and thus my VMs have access to the Internet only if I set proxies. So, once DS3DB0 is provisioned and I have set up my proxies, how could I load the data into the database: is there a script on DS3DB0 or a procedure from VMmark3.properties?
If no... I must be patient but I am wondering how can I check the progress? When is hosts-stub.txt generated (I guess after loading the data because I have never seen this file)?
Thanks a lot for your support
Mat.
You do not need an internet connection. You can monitor the progress of DS3 DB creation by following instructions in VMmark3 Users Guide.
Chapter 3, Preparing the Infrastructure for VMmark 3.1 Benchmark Tests, "Provision VMmark Tiles", "Creating the First VMmark Tile (Tile 0)" , paragraph number 6 (on page 50).
6. Wait for the DS3DB0 creation process to complete; depending on your environment, this can take 12 hours or more.To determine if it’s complete, follow these steps:
a. In a terminal window on the prime client, run the following command:
ssh ‘tail -f /ds3/VMmark3-DS3installoutput.txt’
(where is the IP address of the DS3DB0 VM, which can be found in the hosts-stub.txt file referenced above).
b. Watch for the following text: VMmark3 DS3DB Setup Completed : Rebooting
Fred
Hi Fred
Thanks for your answer.
I am thus puzzled.
It is not the first time I am provisioning the VMs running java -jar tools/VMmark3Service.jar -d 9 -c VMmark3.tile0.properties
but even after 72 hours, /des3/ is empty on DS3DB0 and the file hosts-stub.txt have not been generated.
Find in attachment my files VMmark3-ProvisioningService.log and VMmark3.tile0.properties.
Can you please let me know if you see something wrong?
This time, I can see there is a problem with thne VM Standby0. I must still work on that.
Mat.
Here more info ...
Login on DS3DB0, I can run
>>> cd /root/VMmark3/client/DS3DB
>>> ./DS3_checker.sh
Installing checker files
ds3db_100GB.sh
tmp_tables/
tmp_tables/customers.txt
tmp_tables/inventory.txt
tmp_tables/categories.txt
tmp_tables/custhist.txt
tmp_tables/orderlines.txt
tmp_tables/reviews.txt
tmp_tables/reviews_helpfulness.txt
tmp_tables/membership.txt
tmp_tables/products.txt
tmp_tables/orders.txt
Starting DS3 checker
Mon May 6 11:48:58 CEST 2019
============================================================================
============================================================================
==== DS3 database chacker must be run from DS3 host node ====
==== Currently running on host = DS3DB0
============================================================================
PING DS3DB0.vmmark3.eng.vmware.com (127.0.0.1) 56(84) bytes of data.
64 bytes from DS3DB0.vmmark3.eng.vmware.com (127.0.0.1): icmp_seq=1 ttl=64 time=0.058 ms
--- DS3DB0.vmmark3.eng.vmware.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.058/0.058/0.058/0.000 ms
DS3DB0 is alive
mysqlshow: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
./ds3db_100GB.sh: line 32: [: ==: unary operator expected
ERROR: Database doesn't exist.
============================================================================
>>>
>>> ./ConfigServer_DS3DB.sh 100
mysqlshow: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
./ds3db_100GB.sh: line 32: [: ==: unary operator expected
And S3DB0, I can run
>>> service mysqld start
Redirecting to /bin/systemctl start mysqld.service
never ends...
Going a step further...
(please read up to the end to see the question)
I run
>>> cd /root/VMmark3/tools/
>>> ./make-ds3db.sh 100
And could see:
>>> service mysqld status
Redirecting to /bin/systemctl status mysqld.service
● mysqld.service - MySQL Community Server
Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2019-05-06 13:25:19 CEST; 8min ago
Process: 3416 ExecStartPost=/usr/bin/mysql-systemd-start post (code=exited, status=0/SUCCESS)
Process: 3399 ExecStartPre=/usr/bin/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Main PID: 3415 (mysqld_safe)
CGroup: /system.slice/mysqld.service
├─3415 /bin/sh /usr/bin/mysqld_safe
└─3750 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mysqld.log --open-files-limit=65535 --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysq...
May 06 13:25:17 DS3DB0 systemd[1]: Starting MySQL Community Server...
May 06 13:25:17 DS3DB0 mysqld_safe[3415]: 190506 13:25:17 mysqld_safe Logging to '/var/log/mysqld.log'.
May 06 13:25:17 DS3DB0 mysqld_safe[3415]: 190506 13:25:17 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
May 06 13:25:19 DS3DB0 systemd[1]: Started MySQL Community Server.
Which is already an improvement.
Then I run:
>>> /root/VMmark3/client/DS3DB/
>>> ./ConfigServer_DS3DB.sh
And I could read in /DS3DP0_restore.txt
01:25:12 PM: Starting DS3DB0 restore v. 03022017, writing to /DS3DB0_restore.txt
[...]
Starting DS3 checker
^[[3;J^[[H^[[2JMon May 6 13:25:13 CEST 2019
============================================================================
============================================================================
==== DS3 database chacker must be run from DS3 host node ====
==== Currently running on host = DS3DB0
============================================================================
PING DS3DB0.vmmark3.eng.vmware.com (127.0.0.1) 56(84) bytes of data.
64 bytes from DS3DB0.vmmark3.eng.vmware.com (127.0.0.1): icmp_seq=1 ttl=64 time=0.043 ms
--- DS3DB0.vmmark3.eng.vmware.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.043/0.043/0.043/0.000 ms
DS3DB0 is alive
ERROR: Database #mysql50#lost+found doesn't exist.
============================================================================
01:25:15 PM: DS3 checker script finished..
01:25:15 PM: Stopping mysqld.
01:25:17 PM: Deleted logs.
01:25:19 PM: Successfully started mysqldn01:25:59 PM: Unable to ping DS3WebA0. ping: unknown host DS3WebA0
01:26:39 PM: Unable to ping DS3WebB0. ping: unknown host DS3WebB0
01:27:19 PM: Unable to ping DS3WebC0. ping: unknown host DS3WebC0
01:27:19 PM: DS3DB0 restore completed Mon May 6 13:27:19 CEST 2019
Errors:3, Errors:DS3DB0 unable to ping DS3WebA0. DS3DB0 unable to ping DS3WebB0. DS3DB0 unable to ping DS3WebC0. , Warnings:0, Warnings:None
I thus set /etc/hosts manually to map hostnames and ip adresses:
>>> cat /etc/hosts
127.0.0.1 DS3DB0.vmmark3.eng.vmware.com localhost localhost4 localhost4.localdomain4 DS3DB0
::1 localhost6 localhost6.localdomain6
172.22.197.*** DS3WebA0.vmmark3.eng.vmware.com DS3WebA0
172.22.197.*** DS3WebB0.vmmark3.eng.vmware.com DS3WebB0
172.22.197.*** DS3WebC0.vmmark3.eng.vmware.com DS3WebC0
172.22.197.*** DS3DB0.vmmark3.eng.vmware.com DS3DB0
And rerun the creation script:
>>> ./ConfigServer_DS3DB.sh
>>> cat /DS3DP0_restore.txt
[...]
DS3DB0 is alive
ERROR: Database #mysql50#lost+found doesn't exist.
============================================================================
01:47:51 PM: DS3 checker script finished..
01:47:51 PM: Stopping mysqld.
01:47:53 PM: Deleted logs.
01:47:57 PM: Successfully started mysqldn01:47:57 PM: Ping connectivity established with DS3WebA0.
01:47:57 PM: Ping connectivity established with DS3WebB0.
01:47:57 PM: Ping connectivity established with DS3WebC0.
01:47:57 PM: DS3DB0 restore completed Mon May 6 13:47:57 CEST 2019
Errors:0, Errors:None, Warnings:0, Warnings:None
I thus set /etc/hosts manually to map hostnames and ip addresses on all the VMs and the prime client.
From the prime client,
>>> ssh DS3DB0
>>> tail -f /ds3/VMmark3-DS3installoutput.txt
ERROR 1045 (28000): Access denied for user 'web'@'localhost' (using password: YES)
Loading Reviews data
Warning: Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'web'@'localhost' (using password: YES)
Loading Reviews Helpfulness Ratings Data
Warning: Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'web'@'localhost' (using password: YES)
Removing data files
Mon May 6 12:54:50 CEST 2019
VMmark3 DS3DB Setup Completed : Rebooting
It seems now ok, doesn't it?
Is there anything else to do manually on DS3DB0? Or any other VMs?
Based on that transcript it looks like you did get DS3DB0 VM ds3 db built doing manually:
>>> cd /root/VMmark3/tools/
>>> ./make-ds3db.sh 100
I don't understand what went wrong in provisioning script that it didn't do it.
You can now try a single tile run (I would do with turbo flag set in VMmark3.properties file - see documentation on Turbo).
Fred
Hi Fred
I do not know neither.
But the most important is you seem to confirm I figured out how to finalize the procedure manually. So I can continue.
I will continue feeding this post to let the community know if at the end it works (or not).
To be continued...
(no question here, thus no answer expected)
In the User's Guide, the following command is given to generate the file hosts-stub.txt.
java -jar tools/VMmark3Service.jar -c VMmark3.properties -m hostsFile -tiles 1
It completes the manual procedure.
mathieugont Every provisioning output folder should have a hosts-stub.txt, per this section in the User's Guide, so I am surprised you had to run that "-m hostsFile" command:
Looking back at your specific provisioning / properties file, you should have this file:
/root/VMmark3/provisioning-output/vmmark3-output-tile0-rerun1/hosts-stub.txt
Therefore, you'd run a command like this to populate your hosts file:
cat ~/VMmark3/provisioning-output/vmmark3-output-tile0-rerun1/hosts-stub.txt >> /etc/hosts
Please let us know if you don't see that file on your Prime Client.
Thanks,
David
Dear community
Do you have remarks/ideas to finalize loading the DB on DS3DB0? Below the status...
In my previous message, I missed an error highlighting the DB was not properly loaded:
>>> ./ConfigServer_DS3DB.sh
>>> cat /DS3DP0_restore.txt
[...]
DS3DB0 is alive
ERROR: Database #mysql50#lost+found doesn't exist.
============================================================================
01:47:51 PM: DS3 checker script finished..
01:47:51 PM: Stopping mysqld.
01:47:53 PM: Deleted logs.
01:47:57 PM: Successfully started mysqldn01:47:57 PM: Ping connectivity established with DS3WebA0.
01:47:57 PM: Ping connectivity established with DS3WebB0.
01:47:57 PM: Ping connectivity established with DS3WebC0.
01:47:57 PM: DS3DB0 restore completed Mon May 6 13:47:57 CEST 2019
Errors:0, Errors:None, Warnings:0, Warnings:None
I thus logged on DS3DB0 and run it manually:
>>> cd /ds3
>>> nohup perl ./Install_VMmark_DVDStore.pl &
It works well, except it is now spending more than 24h loading the Customer History Data (even if DS3_cherker.sh tells me I am in the right direction):
>>> tail -n 50 /ds3/VMmark3-DS3installoutput.txt
Warning messages about using password via command line are normal......
Creating database
Warning: Using a password on the command line interface can be insecure.
Creating Indexes
Warning: Using a password on the command line interface can be insecure.
Creating Stored Procedures
Warning: Using a password on the command line interface can be insecure.
Loading Customer Data
Warning: Using a password on the command line interface can be insecure.
ERROR 1317 (70100) at line 7: Query execution was interrupted
Terminal close -- sending "KILL QUERY 26" to server ...
Terminal close -- query aborted.
Loading Orders Data
Warning: Using a password on the command line interface can be insecure.
Loading Orderlines Data
Warning: Using a password on the command line interface can be insecure.
Loading Customer History Data
Warning: Using a password on the command line interface can be insecure.
>>> /root/VMmark3/client/DS3DB
>>> ./DS3_checker.sh
Installing checker files
ds3db_100GB.sh
tmp_tables/
tmp_tables/customers.txt
tmp_tables/inventory.txt
tmp_tables/categories.txt
tmp_tables/custhist.txt
tmp_tables/orderlines.txt
tmp_tables/reviews.txt
tmp_tables/reviews_helpfulness.txt
tmp_tables/membership.txt
tmp_tables/products.txt
tmp_tables/orders.txt
Starting DS3 checker
Fri May 10 09:24:49 CEST 2019
============================================================================
============================================================================
==== DS3 database chacker must be run from DS3 host node ====
==== Currently running on host = DS3DB0
============================================================================
PING DS3DB0 (172.22.197.204) 56(84) bytes of data.
64 bytes from DS3DB0 (172.22.197.204): icmp_seq=1 ttl=64 time=0.063 ms
--- DS3DB0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.063/0.063/0.063/0.000 ms
DS3DB0 is alive
Database DS3 exists. Proceeding with DS3 table checks ...
============================================================================
Table CATEGORIES existance check passed
Table CUSTOMERS existance check passed
Table CUST_HIST existance check passed
Table INVENTORY existance check passed
Table MEMBERSHIP existance check passed
Table ORDERLINES existance check passed
Table ORDERS existance check passed
Table PRODUCTS existance check passed
Table REORDER existance check passed
Table REVIEWS existance check passed
Table REVIEWS_HELPFULNESS existance check passed
PASSED: All DS3 Tables were found to exist. Proceeding with DS3 checks ...
============================================================================
PASSED: DS3 Tables prelim count passed. Proceeding with DS3 checks ...
============================================================================
PASSED: DS3 appears to have web privileges granted. Proceeding with DS3 checks ...
============================================================================
Checking the contents of Tables = category
................All checks passwd checking Table CATEGORIES
Checking the contents of Tables = orders
........................All checks passwd checking Table ORDERS
Checking the contents of Tables = custhist
........ERROR: DS3 value check failed.
============================================================================
--------------------------------------------------------------------------
FAIL: DS3 is either not built completely or not in initialized state.
--------------------------------------------------------------------------
Any advised, idea?
Hi mathieugont
This (during database creation) is not a good sign:
Loading Customer Data
Warning: Using a password on the command line interface can be insecure.
ERROR 1317 (70100) at line 7: Query execution was interrupted
Terminal close -- sending "KILL QUERY 26" to server ...
Terminal close -- query aborted.
This error must be why the DS3 database checker failed while checking the customer history table:
Checking the contents of Tables = custhist
........ERROR: DS3 value check failed.
============================================================================
--------------------------------------------------------------------------
FAIL: DS3 is either not built completely or not in initialized state.
--------------------------------------------------------------------------
Unfortunately, this tells us your MySQL DS3DB0 database is "dirty", and must be rebuilt.
I just spoke with fredab2, and we both agree that your best bet is to rebuild the entire tile 0 from scratch, since creating the non-DS3DB VMs does not really
take very long. Most of the tile0 creation time is in the DS3DB.
To do this, power off and delete-from-disk all [workloadname]0 VMs, and go back to "Creating the First VMmark Tile (Tile 0)" in the User's Guide.
Good luck, and let Fred and I know how it goes.
Thanks, David
David, Fred
I totally agree with you: the best would be reinstall the tile 0.
But I have already done it 3 times and after each installation, DSDB0 did not loaded the data properly (DS3 does not exit in mysql).
I attached my properties fie in case I made a mistake.
Any other idea?
I rerun the procedure to reinstall the tile0.
But I had to run manually the script make-ds3db.sh from /root/VMmark3/tools/@DS3DB0 to load the DB.
Good new, after 24h, it has reached
Loading Customer History Data
Warning: Using a password on the command line interface can be insecure.
without the error mentioned earlier. It continue running.
Any comment?
Hi mathieugont
I rerun the procedure to reinstall the tile0.
But I had to run manually the script make-ds3db.sh from /root/VMmark3/tools/@DS3DB0 to load the DB.
Good new, after 24h, it has reached
Loading Customer History Data
Warning: Using a password on the command line interface can be insecure.
without the error mentioned earlier. It continue running.
Any comment?
I have asked about this issue internally with our VMmark engineering team. This is very unusual, i.e. not something we have seen before. The DS3DB0 creation should always start automatically, and the fact that you had to start it manually indicates something is wrong. It's just not obvious to me what.
Can you elaborate on:
Thanks, and apologies for all the trouble!
David
Hi mathieugont -- just checking in -- could you provide these details?
Can you elaborate on:
- Server make/model (ESXi hosts)
- Storage make/model (including types of vendor/capacity of your drives, whether they're HDDs [and if so what speed] or SSDs, and what RAID/LUN configurations they are in? Are you using vSAN?
- ESXi and vCenter version, including exact build numbers
Thanks, David
Hi David
Like you already know, I run the load procedure manually.
It finally ends successfully after almost 4 days.
The only think I cannot explain is why this procedure was not automatically launched after the provisioning (/ds3 was empty).
Here my system:
Server make/model (ESXi hosts)
- Dual socket BDW E5-2699v4 servers
- 780GB DRAM per server
- 1x local ATA disk (1.8TB HDD) per server (Western Digital?), partition format GPT
- ESXi-6.7.0.13006603
- Each server's HDD is a datastore based on VMFS-6.82
- no vSan (i guess)
- vSphere Client 6.7.0.30000
- All my clients are on a hardware server (ESXi001), except the tile and the prime clients hosted on another one (ESXi005).
This thread is not my priority anymore, but if it can help the community, we can continue figuring out the why.
The other one
is because I would like to run weathervane asap.