1 2 Previous Next 15 Replies Latest reply on May 16, 2019 11:16 AM by mathieugont

    Does load the data into the DS3DB0 database need access to the Internet?

    mathieugont Novice

      Dear community.

       

      I have provisioned my tile0. Among others, the virtual machine DS3DB0 is created on the cluster referenced by vCServerVMmark3Cluster.

       

      The User's guide says "The provisioning process initiates creation of the DS3DB0 virtual machine, which can take 12 hours or more while data is loaded into the DS3DB0 database. This procedure is not considered part of the provisioning process.", and some threads on vmware.community mention more than 36h.

       

      Does load the data into the DS3DB0 database need access to the Internet?

       

      If yes... My ESXi hosts and thus my VMs have access to the Internet only if I set proxies. So, once DS3DB0 is provisioned and I have set up my proxies, how could I load the data into the database: is there a script on DS3DB0 or a procedure from VMmark3.properties?

       

      If no... I must be patient but I am wondering how can I check the progress? When is hosts-stub.txt generated (I guess after loading the data because I have never seen this file)?

       

      Thanks a lot for your support

      Mat.

        • 1. Re: Does load the data into the DS3DB0 database need access to the Internet?
          fredab2 Novice
          VMware Employees

          You do not need an internet connection.  You can monitor the progress of DS3 DB creation by following instructions in VMmark3 Users Guide.

           

          Chapter 3, Preparing the Infrastructure for VMmark 3.1 Benchmark Tests, "Provision VMmark Tiles", "Creating the First VMmark Tile (Tile 0)" , paragraph number 6 (on page 50).

           

          6.  Wait for the DS3DB0 creation process to complete; depending on your environment, this can take 12 hours or more.To determine if it’s complete, follow these steps:

          a. In a terminal window on the prime client, run the following command:

              ssh  ‘tail -f /ds3/VMmark3-DS3installoutput.txt’

              (where  is the IP address of the DS3DB0 VM, which can be found in the hosts-stub.txt file referenced above).

          b. Watch for the following text: VMmark3 DS3DB Setup Completed : Rebooting

           

          Fred

          • 2. Re: Does load the data into the DS3DB0 database need access to the Internet?
            mathieugont Novice

            Hi Fred

             

            Thanks for your answer.

             

            I am thus puzzled.

            It is not the first time I am provisioning the VMs running java -jar tools/VMmark3Service.jar -d 9 -c VMmark3.tile0.properties

            but even after 72 hours, /des3/ is empty on DS3DB0 and the file hosts-stub.txt have not been generated.

             

            Find in attachment my files VMmark3-ProvisioningService.log and VMmark3.tile0.properties.

            Can you please let me know if you see something wrong?

             

            This time, I can see there is a problem with thne VM Standby0. I must still work on that.

             

            Mat.

            • 3. Re: Does load the data into the DS3DB0 database need access to the Internet?
              mathieugont Novice

              Here more info ...

               

              Login on DS3DB0, I can run

              >>> cd /root/VMmark3/client/DS3DB

              >>> ./DS3_checker.sh

              Installing checker files

              ds3db_100GB.sh

              tmp_tables/

              tmp_tables/customers.txt

              tmp_tables/inventory.txt

              tmp_tables/categories.txt

              tmp_tables/custhist.txt

              tmp_tables/orderlines.txt

              tmp_tables/reviews.txt

              tmp_tables/reviews_helpfulness.txt

              tmp_tables/membership.txt

              tmp_tables/products.txt

              tmp_tables/orders.txt

              Starting DS3 checker

              Mon May  6 11:48:58 CEST 2019

              ============================================================================

              ============================================================================

              ====       DS3 database chacker must be run from DS3 host node          ====

              ====            Currently running on host = DS3DB0

              ============================================================================

              PING DS3DB0.vmmark3.eng.vmware.com (127.0.0.1) 56(84) bytes of data.

              64 bytes from DS3DB0.vmmark3.eng.vmware.com (127.0.0.1): icmp_seq=1 ttl=64 time=0.058 ms

               

               

              --- DS3DB0.vmmark3.eng.vmware.com ping statistics ---

              1 packets transmitted, 1 received, 0% packet loss, time 0ms

              rtt min/avg/max/mdev = 0.058/0.058/0.058/0.000 ms

              DS3DB0 is alive

              mysqlshow: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)

              ./ds3db_100GB.sh: line 32: [: ==: unary operator expected

              ERROR: Database  doesn't exist.

              ============================================================================

              >>>

              >>>   ./ConfigServer_DS3DB.sh 100

              mysqlshow: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)

              ./ds3db_100GB.sh: line 32: [: ==: unary operator expected

               

              And S3DB0, I can run

              >>> service mysqld start

              Redirecting to /bin/systemctl start  mysqld.service

               

              never ends...

              • 4. Re: Does load the data into the DS3DB0 database need access to the Internet?
                mathieugont Novice

                Going a step further...

                (please read up to the end to see the question)

                 

                I run

                >>> cd /root/VMmark3/tools/

                >>> ./make-ds3db.sh 100

                 

                And could see:

                >>> service mysqld status

                Redirecting to /bin/systemctl status  mysqld.service

                ● mysqld.service - MySQL Community Server

                   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)

                   Active: active (running) since Mon 2019-05-06 13:25:19 CEST; 8min ago

                  Process: 3416 ExecStartPost=/usr/bin/mysql-systemd-start post (code=exited, status=0/SUCCESS)

                  Process: 3399 ExecStartPre=/usr/bin/mysql-systemd-start pre (code=exited, status=0/SUCCESS)

                Main PID: 3415 (mysqld_safe)

                   CGroup: /system.slice/mysqld.service

                           ├─3415 /bin/sh /usr/bin/mysqld_safe

                           └─3750 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mysqld.log --open-files-limit=65535 --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysq...

                 

                May 06 13:25:17 DS3DB0 systemd[1]: Starting MySQL Community Server...

                May 06 13:25:17 DS3DB0 mysqld_safe[3415]: 190506 13:25:17 mysqld_safe Logging to '/var/log/mysqld.log'.

                May 06 13:25:17 DS3DB0 mysqld_safe[3415]: 190506 13:25:17 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql

                May 06 13:25:19 DS3DB0 systemd[1]: Started MySQL Community Server.

                 

                Which is already an improvement.

                Then I run:

                >>> /root/VMmark3/client/DS3DB/

                >>> ./ConfigServer_DS3DB.sh

                 

                And I could read in /DS3DP0_restore.txt

                01:25:12 PM: Starting DS3DB0 restore v. 03022017, writing to /DS3DB0_restore.txt

                [...]

                Starting DS3 checker

                ^[[3;J^[[H^[[2JMon May  6 13:25:13 CEST 2019

                ============================================================================

                ============================================================================

                ====       DS3 database chacker must be run from DS3 host node          ====

                ====            Currently running on host = DS3DB0

                ============================================================================

                PING DS3DB0.vmmark3.eng.vmware.com (127.0.0.1) 56(84) bytes of data.

                64 bytes from DS3DB0.vmmark3.eng.vmware.com (127.0.0.1): icmp_seq=1 ttl=64 time=0.043 ms

                 

                 

                --- DS3DB0.vmmark3.eng.vmware.com ping statistics ---

                1 packets transmitted, 1 received, 0% packet loss, time 0ms

                rtt min/avg/max/mdev = 0.043/0.043/0.043/0.000 ms

                DS3DB0 is alive

                ERROR: Database #mysql50#lost+found doesn't exist.

                ============================================================================

                01:25:15 PM: DS3 checker script finished..

                01:25:15 PM: Stopping mysqld.

                01:25:17 PM: Deleted logs.

                01:25:19 PM: Successfully started mysqldn01:25:59 PM: Unable to ping DS3WebA0. ping: unknown host DS3WebA0

                01:26:39 PM: Unable to ping DS3WebB0. ping: unknown host DS3WebB0

                01:27:19 PM: Unable to ping DS3WebC0. ping: unknown host DS3WebC0

                01:27:19 PM: DS3DB0 restore completed Mon May  6 13:27:19 CEST 2019

                Errors:3, Errors:DS3DB0 unable to ping DS3WebA0. DS3DB0 unable to ping DS3WebB0. DS3DB0 unable to ping DS3WebC0. , Warnings:0, Warnings:None

                 

                I thus set /etc/hosts manually to map hostnames and ip adresses:

                >>> cat /etc/hosts

                127.0.0.1       DS3DB0.vmmark3.eng.vmware.com localhost localhost4 localhost4.localdomain4 DS3DB0

                ::1         localhost6 localhost6.localdomain6

                 

                172.22.197.***  DS3WebA0.vmmark3.eng.vmware.com DS3WebA0

                172.22.197.***  DS3WebB0.vmmark3.eng.vmware.com DS3WebB0

                172.22.197.***  DS3WebC0.vmmark3.eng.vmware.com DS3WebC0

                172.22.197.***  DS3DB0.vmmark3.eng.vmware.com DS3DB0

                 

                And rerun the creation script:

                >>> ./ConfigServer_DS3DB.sh

                >>> cat /DS3DP0_restore.txt

                [...]

                DS3DB0 is alive

                ERROR: Database #mysql50#lost+found doesn't exist.

                ============================================================================

                01:47:51 PM: DS3 checker script finished..

                01:47:51 PM: Stopping mysqld.

                01:47:53 PM: Deleted logs.

                01:47:57 PM: Successfully started mysqldn01:47:57 PM: Ping connectivity established with DS3WebA0.

                01:47:57 PM: Ping connectivity established with DS3WebB0.

                01:47:57 PM: Ping connectivity established with DS3WebC0.

                01:47:57 PM: DS3DB0 restore completed Mon May  6 13:47:57 CEST 2019

                Errors:0, Errors:None, Warnings:0, Warnings:None

                 

                I thus set /etc/hosts manually to map hostnames and ip addresses on all the VMs and the prime client.

                 

                From the prime client,

                >>> ssh DS3DB0

                >>> tail -f /ds3/VMmark3-DS3installoutput.txt

                ERROR 1045 (28000): Access denied for user 'web'@'localhost' (using password: YES)

                Loading Reviews data

                Warning: Using a password on the command line interface can be insecure.

                ERROR 1045 (28000): Access denied for user 'web'@'localhost' (using password: YES)

                Loading Reviews Helpfulness Ratings Data

                Warning: Using a password on the command line interface can be insecure.

                ERROR 1045 (28000): Access denied for user 'web'@'localhost' (using password: YES)

                Removing data files

                Mon May  6 12:54:50 CEST 2019

                VMmark3 DS3DB Setup Completed : Rebooting

                 

                It seems now ok, doesn't it?

                Is there anything else to do manually on DS3DB0? Or any other VMs?

                • 5. Re: Does load the data into the DS3DB0 database need access to the Internet?
                  fredab2 Novice
                  VMware Employees

                  Based on that transcript it looks like you did get DS3DB0 VM ds3 db built doing manually:

                   

                  >>> cd /root/VMmark3/tools/

                  >>> ./make-ds3db.sh 100

                   

                  I don't understand what went wrong in provisioning script that it didn't do it.

                   

                  You can now try a single tile run (I would do with turbo flag set in VMmark3.properties file - see documentation on Turbo).

                   

                  Fred

                  • 6. Re: Does load the data into the DS3DB0 database need access to the Internet?
                    mathieugont Novice

                    Hi Fred

                     

                    I do not know neither.

                     

                    But the most important is you seem to confirm I figured out how to finalize the procedure manually. So I can continue.

                    I will continue feeding this post to let the community know if at the end it works (or not).

                     

                    To be continued...

                    • 7. Re: Does load the data into the DS3DB0 database need access to the Internet?
                      mathieugont Novice

                      (no question here, thus no answer expected)

                       

                      In the User's Guide, the following command is given to generate the file hosts-stub.txt.

                      java -jar tools/VMmark3Service.jar -c VMmark3.properties -m hostsFile -tiles 1

                       

                      It completes the manual procedure.

                      • 8. Re: Does load the data into the DS3DB0 database need access to the Internet?
                        dmorse Enthusiast
                        VMware Employees

                        mathieugont Every provisioning output folder should have a hosts-stub.txt, per this section in the User's Guide, so I am surprised you had to run that "-m hostsFile" command:

                        Looking back at your specific provisioning / properties file, you should have this file:

                        /root/VMmark3/provisioning-output/vmmark3-output-tile0-rerun1/hosts-stub.txt

                         

                        Therefore, you'd run a command like this to populate your hosts file:

                        cat ~/VMmark3/provisioning-output/vmmark3-output-tile0-rerun1/hosts-stub.txt >> /etc/hosts

                        Please let us know if you don't see that file on your Prime Client.

                         

                        Thanks,

                        David

                        • 9. Re: Does load the data into the DS3DB0 database need access to the Internet?
                          mathieugont Novice

                          Dear community

                           

                          Do you have remarks/ideas to finalize loading the DB on DS3DB0? Below the status...

                           

                          In my previous message, I missed an error highlighting the DB was not properly loaded:

                          >>> ./ConfigServer_DS3DB.sh

                          >>> cat /DS3DP0_restore.txt

                          [...]

                          DS3DB0 is alive

                          ERROR: Database #mysql50#lost+found doesn't exist.

                          ============================================================================

                          01:47:51 PM: DS3 checker script finished..

                          01:47:51 PM: Stopping mysqld.

                          01:47:53 PM: Deleted logs.

                          01:47:57 PM: Successfully started mysqldn01:47:57 PM: Ping connectivity established with DS3WebA0.

                          01:47:57 PM: Ping connectivity established with DS3WebB0.

                          01:47:57 PM: Ping connectivity established with DS3WebC0.

                          01:47:57 PM: DS3DB0 restore completed Mon May  6 13:47:57 CEST 2019

                          Errors:0, Errors:None, Warnings:0, Warnings:None

                           

                          I thus logged on DS3DB0 and run it manually:

                          >>> cd /ds3

                          >>> nohup perl ./Install_VMmark_DVDStore.pl &

                           

                          It works well, except it is now spending more than 24h loading the Customer History Data (even if DS3_cherker.sh tells me I am in the right direction):

                          >>> tail -n 50 /ds3/VMmark3-DS3installoutput.txt

                          Warning messages about using password via command line are normal......

                          Creating database

                          Warning: Using a password on the command line interface can be insecure.

                          Creating Indexes

                          Warning: Using a password on the command line interface can be insecure.

                          Creating Stored Procedures

                          Warning: Using a password on the command line interface can be insecure.

                          Loading Customer Data

                          Warning: Using a password on the command line interface can be insecure.

                          ERROR 1317 (70100) at line 7: Query execution was interrupted

                          Terminal close -- sending "KILL QUERY 26" to server ...

                          Terminal close -- query aborted.

                          Loading Orders Data

                          Warning: Using a password on the command line interface can be insecure.

                          Loading Orderlines Data

                          Warning: Using a password on the command line interface can be insecure.

                          Loading Customer History Data

                          Warning: Using a password on the command line interface can be insecure.

                          >>> /root/VMmark3/client/DS3DB

                          >>> ./DS3_checker.sh

                          Installing checker files

                          ds3db_100GB.sh

                          tmp_tables/

                          tmp_tables/customers.txt

                          tmp_tables/inventory.txt

                          tmp_tables/categories.txt

                          tmp_tables/custhist.txt

                          tmp_tables/orderlines.txt

                          tmp_tables/reviews.txt

                          tmp_tables/reviews_helpfulness.txt

                          tmp_tables/membership.txt

                          tmp_tables/products.txt

                          tmp_tables/orders.txt

                          Starting DS3 checker

                          Fri May 10 09:24:49 CEST 2019

                          ============================================================================

                          ============================================================================

                          ====       DS3 database chacker must be run from DS3 host node          ====

                          ====            Currently running on host = DS3DB0

                          ============================================================================

                          PING DS3DB0 (172.22.197.204) 56(84) bytes of data.

                          64 bytes from DS3DB0 (172.22.197.204): icmp_seq=1 ttl=64 time=0.063 ms

                           

                          --- DS3DB0 ping statistics ---

                          1 packets transmitted, 1 received, 0% packet loss, time 0ms

                          rtt min/avg/max/mdev = 0.063/0.063/0.063/0.000 ms

                          DS3DB0 is alive

                          Database DS3 exists. Proceeding with DS3 table checks ...

                          ============================================================================

                          Table CATEGORIES existance check passed

                          Table CUSTOMERS existance check passed

                          Table CUST_HIST existance check passed

                          Table INVENTORY existance check passed

                          Table MEMBERSHIP existance check passed

                          Table ORDERLINES existance check passed

                          Table ORDERS existance check passed

                          Table PRODUCTS existance check passed

                          Table REORDER existance check passed

                          Table REVIEWS existance check passed

                          Table REVIEWS_HELPFULNESS existance check passed

                          PASSED: All DS3 Tables were found to exist. Proceeding with DS3 checks ...

                          ============================================================================

                          PASSED: DS3 Tables prelim count passed. Proceeding with DS3 checks ...

                          ============================================================================

                          PASSED: DS3 appears to have web privileges granted. Proceeding with DS3 checks ...

                          ============================================================================

                          Checking the contents of Tables = category

                          ................All checks passwd checking Table CATEGORIES

                          Checking the contents of Tables = orders

                          ........................All checks passwd checking Table ORDERS

                          Checking the contents of Tables = custhist

                          ........ERROR:  DS3 value check failed.

                          ============================================================================

                          --------------------------------------------------------------------------

                          FAIL:  DS3 is either not built completely or not in initialized state.

                          --------------------------------------------------------------------------

                           

                          Any advised, idea?

                          • 10. Re: Does load the data into the DS3DB0 database need access to the Internet?
                            dmorse Enthusiast
                            VMware Employees

                            Hi mathieugont

                             

                            This (during database creation) is not a good sign:

                            Loading Customer Data

                            Warning: Using a password on the command line interface can be insecure.

                            ERROR 1317 (70100) at line 7: Query execution was interrupted

                            Terminal close -- sending "KILL QUERY 26" to server ...

                            Terminal close -- query aborted.

                             

                            This error must be why the DS3 database checker failed while checking the customer history table:

                            Checking the contents of Tables = custhist

                            ........ERROR:  DS3 value check failed.

                            ============================================================================

                            --------------------------------------------------------------------------

                            FAIL:  DS3 is either not built completely or not in initialized state.

                            --------------------------------------------------------------------------

                            Unfortunately, this tells us your MySQL DS3DB0 database is "dirty", and must be rebuilt.

                             

                            I just spoke with fredab2, and we both agree that your best bet is to rebuild the entire tile 0 from scratch, since creating the non-DS3DB VMs does not really

                            take very long.  Most of the tile0 creation time is in the DS3DB.

                             

                            To do this, power off and delete-from-disk all [workloadname]0 VMs, and go back to "Creating the First VMmark Tile (Tile 0)" in the User's Guide.

                             

                            Good luck, and let Fred and I know how it goes.

                             

                            Thanks, David

                            • 11. Re: Does load the data into the DS3DB0 database need access to the Internet?
                              mathieugont Novice

                              David, Fred

                               

                              I totally agree with you: the best would be reinstall the tile 0.

                              But I have already done it 3 times and after each installation, DSDB0 did not loaded the data properly (DS3 does not exit in mysql).

                              I attached my properties fie in case I made a mistake.

                               

                              Any other idea?

                              • 12. Re: Does load the data into the DS3DB0 database need access to the Internet?
                                mathieugont Novice

                                I rerun the procedure to reinstall the tile0.

                                But I had to run manually the script make-ds3db.sh from /root/VMmark3/tools/@DS3DB0 to load the DB. 

                                Good new, after 24h, it has reached

                                Loading Customer History Data

                                Warning: Using a password on the command line interface can be insecure.

                                without the error mentioned earlier. It continue running.

                                 

                                Any comment?

                                • 13. Re: Does load the data into the DS3DB0 database need access to the Internet?
                                  dmorse Enthusiast
                                  VMware Employees

                                  Hi mathieugont

                                  I rerun the procedure to reinstall the tile0.

                                  But I had to run manually the script make-ds3db.sh from /root/VMmark3/tools/@DS3DB0 to load the DB.

                                  Good new, after 24h, it has reached

                                  Loading Customer History Data

                                  Warning: Using a password on the command line interface can be insecure.

                                  without the error mentioned earlier. It continue running.

                                   

                                  Any comment?

                                  I have asked about this issue internally with our VMmark engineering team.  This is very unusual, i.e. not something we have seen before.  The DS3DB0 creation should always start automatically, and the fact that you had to start it manually indicates something is wrong.  It's just not obvious to me what.

                                   

                                  Can you elaborate on:

                                  • Server make/model (ESXi hosts)
                                  • Storage make/model (including types of vendor/capacity of your drives, whether they're HDDs [and if so what speed] or SSDs, and what RAID/LUN configurations they are in?  Are you using vSAN?
                                  • ESXi and vCenter version, including exact build numbers

                                   

                                  Thanks, and apologies for all the trouble!

                                  David

                                  • 14. Re: Does load the data into the DS3DB0 database need access to the Internet?
                                    dmorse Enthusiast
                                    VMware Employees

                                    Hi mathieugont -- just checking in -- could you provide these details?

                                     

                                    Can you elaborate on:

                                    • Server make/model (ESXi hosts)
                                    • Storage make/model (including types of vendor/capacity of your drives, whether they're HDDs [and if so what speed] or SSDs, and what RAID/LUN configurations they are in?  Are you using vSAN?
                                    • ESXi and vCenter version, including exact build numbers

                                    Thanks, David

                                    1 2 Previous Next