1 Reply Latest reply on Oct 19, 2018 4:27 AM by SebastianGrugel

    vCD 9.1.0 adding to Cassandra DB additional metrics to collect performance data

    SebastianGrugel Enthusiast
    vExpert

      Hello.

       

      Im using vCD 9.1.0 and few days ago configured Cassandra 3 in our LAB.

       

      I used for installation and integration:

      http://www.vstellar.com/2017/01/08/learning-apache-cassandra-part-3-installing-cassandra-on-rhel6/

      http://www.vstellar.com/2017/10/05/configuring-vcd-9-0-to-send-metric-data-to-cassandra-db/

      https://anthonyspiteri.net/enabling-configuring-and-viewing-metrics-in-vcloud-director-9-0/

      https://docs.vmware.com/en/vCloud-Director/9.0/com.vmware.vcloud.admin.doc/GUID-55D8360A-B4F9-4CDC-8BDA-972C4F387D74.html#GUID-55D8360A-B4F9-4CDC-8BDA-972C4F387D74

      https://docs.vmware.com/en/vCloud-Director/9.1/com.vmware.vcloud.admin.doc/GUID-983C566A-7DE5-4F06-9044-C979F8AE7C99.html

       

      Combined those all docs and blogs and deployed 1 Centos node with cassandra.

       

      After integration by default we have access only to 8 metrics (we can confirm this checking table in vCD DB and Cassandra DB or simpler login to tenant portal and check metrics):

       

      cpu.usage.average

      cpu.usage.maximum

      cpu.usagemhz.average

      disk.provisioned.latest

      disk.read.average

      disk.used.latest

      disk.write.average

      mem.usage.average

       

       

      This is fine but what if we want to collect some additional metrics? For example, those metrics what have Tomas included in his article:

      https://fojta.wordpress.com/2017/11/24/how-to-configure-additional-vm-metrics-in-vcloud-director/

       

      mem.granted.average

      mem.granted.maximum

       

      I tried adding those metrics without success.

       

       

      What i did:

       

      1. Created file /tmp/metrics.groovy  with content:

       

      configuration {

      metric("mem.granted.average")

      metric("mem.granted.maximum")

      }

       

      2. Used command with success because new metric appeared in vCD DB (MSSQL)

       

      $VCLOUD_HOME/bin/cell-management-tool configure-metrics –metrics-config /tmp/metrics.groovy

       

      3. Used command (without success):

      /opt/vmware/vcloud-director/bin/cell-management-tool cassandra –configure –cluster-nodes 192.x.x.x –username root –password "PASSWORD" –port 9042 –ttl 15 –update-schema

       

      Verifying Cassandra settings...

      Cassandra setting valid for node: 10.3.5.50

      Cassandra configuration settings verified successfully

      ERROR: Unable to update Cassandra schema

      Unexpected error, exiting...

       

      in cell-management-tool.log logs i found only:

       

      2018-09-19 10:17:37,614 | DEBUG    | main                      | EncryptionManager              | EncryptionManager configured with system key. |

      2018-09-19 10:17:37,704 | DEBUG    | main                      | EncryptionManager              | EncryptionManager has been loaded successfully. |

      2018-09-19 10:17:42,991 | ERROR    | main                      | ConfigureCassandraCommand      | Cassandra configuration settings verified successfully |

      2018-09-19 10:17:45,525 | ERROR    | main                      | ConfigureCassandraCommand      | Unexpected error, exiting... |

      java.lang.Exception: Cannot find configured metrics in cassandra schema

              at com.vmware.vcloud.cassandra.cmt.ConfigureCassandraCommand.getSchemaColumns(ConfigureCassandraCommand.java:333)

              at com.vmware.vcloud.cassandra.cmt.ConfigureCassandraCommand.updateSchema(ConfigureCassandraCommand.java:266)

              at com.vmware.vcloud.cassandra.cmt.ConfigureCassandraCommand.configure(ConfigureCassandraCommand.java:217)

              at com.vmware.vcloud.cassandra.cmt.ConfigureCassandraCommand.executeCommand(ConfigureCassandraCommand.java:178)

              at com.vmware.vcloud.cassandra.cmt.ConfigureCassandraCommand.executeCommand(ConfigureCassandraCommand.java:51)

              at com.vmware.vcloud.common.cmt.HibernateBasedToolCommand.doCommandInternal(HibernateBasedToolCommand.java:43)

              at com.vmware.vcloud.common.cmt.AbstractDatabaseToolCommand.doCommand(AbstractDatabaseToolCommand.java:87)

              at com.vmware.vcloud.cell.management.internal.AbstractToolCommand.call(AbstractToolCommand.java:177)

              at com.vmware.vcloud.cell.management.internal.AbstractToolCommand.call(AbstractToolCommand.java:37)

              at com.vmware.vcloud.cell.management.CellManagementTool.start(CellManagementTool.java:299)

              at com.vmware.vcloud.cell.management.CellManagementTool.main(CellManagementTool.java:215)

              at com.vmware.vcloud.cell.management.CellManagementTool.main(CellManagementTool.java:228)

      2018-09-19 10:17:45,540 | DEBUG    | main                      | CellManagementTool             | Exiting with completion code: OTHER |

       

       

      Next step, in my troubleshoot I logged to Cassandra and logged direct to DB and used command "DESCRIBE SCHEMA" in tool "cqlsh":

       

      Interesting output for us is list of existing in Cassandra metrics (what only confirmed error: "Cannot find configured metrics in cassandra schema" :

       

      CREATE TABLE vcloud_metrics.vm_metrics (

          vm_id text,

          sample_time timestamp,

          "cpu.usage.average" bigint,

          "cpu.usage.average.instance" text,

          "cpu.usage.average.unit" int,

          "cpu.usage.maximum" bigint,

          "cpu.usage.maximum.instance" text,

          "cpu.usage.maximum.unit" int,

          "cpu.usagemhz.average" bigint,

          "cpu.usagemhz.average.instance" text,

          "cpu.usagemhz.average.unit" int,

          "disk.provisioned.latest" bigint,

          "disk.provisioned.latest.instance" text,

          "disk.provisioned.latest.unit" int,

          "disk.read.average" bigint,

          "disk.read.average.instance" text,

          "disk.read.average.unit" int,

          "disk.used.latest" bigint,

          "disk.used.latest.instance" text,

          "disk.used.latest.unit" int,

          "disk.write.average" bigint,

          "disk.write.average.instance" text,

          "disk.write.average.unit" int,

          "mem.usage.average" bigint,

          "mem.usage.average.instance" text,

          "mem.usage.average.unit" int,

          org_id text,

          vapp_id text,

          vdc_id text,

          PRIMARY KEY (vm_id, sample_time)

      ) WITH CLUSTERING ORDER BY (sample_time ASC)

          AND bloom_filter_fp_chance = 0.01

          AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}

          AND comment = ''

          AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}

          AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}

          AND crc_check_chance = 1.0

          AND dclocal_read_repair_chance = 0.1

          AND default_time_to_live = 0

          AND gc_grace_seconds = 864000

          AND max_index_interval = 2048

          AND memtable_flush_period_in_ms = 0

          AND min_index_interval = 128

          AND read_repair_chance = 0.0

          AND speculative_retry = '99PERCENTILE';

       

       

      Unfortunatelly i dont have nothing more. I performed some workaround which is looks like working but i think is destructive to historical data:

       

       

      WORKAROUND:

      Inspired by command from this article:

      https://blogs.vmware.com/vcat/2017/11/virtual-machine-performance-metrics-vmware-vcloud-director-9-0.html

       

       

      1.Use that same file /tmp/metrics.groovy  with content

       

      configuration {

      metric("mem.granted.average")

      metric("mem.granted.maximum")

      }

       

      2. Remove keyspace in Cassandra

      "drop keyspace vcloud_metrics"

      3. Used command which was used during integration CELLs with Cassandra (on begining) which create schema again:

      "/opt/vmware/vcloud-director/bin/cell-management-tool cassandra -configure -create-schema -cluster-nodes 192.x.x.x -username root -password "xxxxxxxx" -port 9042 -ttl 15"

       

       

      Results (additional metrics looks like are imported from file):

       

      "Verifying Cassandra settings...

      Cassandra setting valid for node: 10.3.5.50

      Cassandra configuration settings verified successfully

      vcloud_metrics keyspace created...

      vm_metrics table created...

      adding configured metrics to the schema...

      adding counter: mem.granted.average

      adding counter: mem.granted.maximum

      adding counter: cpu.usage.average

      adding counter: cpu.usage.maximum

      adding counter: cpu.usagemhz.average

      adding counter: disk.provisioned.latest

      adding counter: disk.read.average

      adding counter: disk.used.latest

      adding counter: disk.write.average

      adding counter: mem.usage.average

      vapp_metrics table created...

      vdc_metrics table created...

      org_metrics table created...

      Persisting Cassandra settings...

       

       

      Success. The monitoring service is now configured to persist data into cassandra nodes(10.3.5.50), vCD cell(s) must be restarted if they are already running."

       

      After restart CELLs looks OK:

       

      In my opinion this is workaround which is not safe.... i think during removing KEYSPACE i removed some historical data...

       

      Some suggestion why i cant update schema with new metrics ? by command:

      /opt/vmware/vcloud-director/bin/cell-management-tool cassandra –configure –cluster-nodes 192.x.x.x –username root –password "PASSWORD" –port 9042 –ttl 15 –update-schema

       

      or maybe im doing something wrong

       

      Sebastian

      www.vSebastian.net

       

      Message was edited by: SebastianGrugel

        • 1. Re: vCD 9.1.0 adding to Cassandra DB additional metrics to collect performance data
          SebastianGrugel Enthusiast
          vExpert

          OK i have update:

           

          1. My SR case in VMware and resolve which i received:

           

          "- Running the statement "drop keyspace vcloud_metrics" in the database it is expected to drop what historical data it has in that table, it would be the same as dropping tables inside SQL.
          - Regarding the update schema issue it is an issue with the cell-management-tool and it will be resolved in the next release of vCD, vCD 9.5. The release of vCD 9.5 was released yesterday
          - Asking Engineering was your workaround supported/good etc, the answer was: Seeing as we have a working workaround available and your method gets rid of existing historical data, it wouldnt be officially supported by us and would not be recommended.

          I have included the workaround Engineering provided below, so you can see how it would have being resolved in a supported manner, please note the metrics will be different as the case was opened for a slightly different issue

          Workaround:
          Workaround would be to manually go into one of the Cassandra nodes and update the DB columns manually. Then Cassandra can replicate the new columns to the other nodes as designed.

          1) We want to double check that the new values are present in VCD SQL “metric_configuration” table

          2) Connect to one of the Cassandra nodes and run the following:

          ALTER TABLE vcloud_metrics.vm_metrics ADD "virtualDisk.numberReadAveraged" bigint;
          ALTER TABLE vcloud_metrics.vm_metrics ADD "virtualDisk.numberReadAveraged.instance" text;
          ALTER TABLE vcloud_metrics.vm_metrics ADD "virtualDisk.numberReadAveraged.unit" int;

          ALTER TABLE vcloud_metrics.vm_metrics ADD "virtualDisk.numberWriteAveraged" bigint;
          ALTER TABLE vcloud_metrics.vm_metrics ADD "virtualDisk.numberWriteAveraged.instance" text;
          ALTER TABLE vcloud_metrics.vm_metrics ADD "virtualDisk.numberWriteAveraged.unit" int;

          3) Upon restarting the vCD cells, navigating back to the metrics view for VMs should show virtualDisk.numberReadAveraged and virtualDisk.numberWriteAveraged."

           

          2. This problem was resolved in vCD 9.5 - Tested this today in our test environment with vCD 9.5 and Cassandra 3.11

           

           

          The problem was really resolved in vCD 9.5

           

          If you will have some doubts contact with me directly.

           

          Thanks

          Sebastian

          www.vSebastian.Net