Hello.
Im using vCD 9.1.0 and few days ago configured Cassandra 3 in our LAB.
I used for installation and integration:
http://www.vstellar.com/2017/01/08/learning-apache-cassandra-part-3-installing-cassandra-on-rhel6/
http://www.vstellar.com/2017/10/05/configuring-vcd-9-0-to-send-metric-data-to-cassandra-db/
https://anthonyspiteri.net/enabling-configuring-and-viewing-metrics-in-vcloud-director-9-0/
Combined those all docs and blogs and deployed 1 Centos node with cassandra.
After integration by default we have access only to 8 metrics (we can confirm this checking table in vCD DB and Cassandra DB or simpler login to tenant portal and check metrics):
cpu.usage.average
cpu.usage.maximum
cpu.usagemhz.average
disk.provisioned.latest
disk.read.average
disk.used.latest
disk.write.average
mem.usage.average
This is fine but what if we want to collect some additional metrics? For example, those metrics what have Tomas included in his article:
https://fojta.wordpress.com/2017/11/24/how-to-configure-additional-vm-metrics-in-vcloud-director/
mem.granted.average
mem.granted.maximum
I tried adding those metrics without success.
What i did:
1. Created file /tmp/metrics.groovy with content:
configuration {
metric("mem.granted.average")
metric("mem.granted.maximum")
}
2. Used command with success because new metric appeared in vCD DB (MSSQL)
$VCLOUD_HOME/bin/cell-management-tool configure-metrics –metrics-config /tmp/metrics.groovy
3. Used command (without success):
/opt/vmware/vcloud-director/bin/cell-management-tool cassandra –configure –cluster-nodes 192.x.x.x –username root –password "PASSWORD" –port 9042 –ttl 15 –update-schema
Verifying Cassandra settings...
Cassandra setting valid for node: 10.3.5.50
Cassandra configuration settings verified successfully
ERROR: Unable to update Cassandra schema
Unexpected error, exiting...
in cell-management-tool.log logs i found only:
2018-09-19 10:17:37,614 | DEBUG | main | EncryptionManager | EncryptionManager configured with system key. |
2018-09-19 10:17:37,704 | DEBUG | main | EncryptionManager | EncryptionManager has been loaded successfully. |
2018-09-19 10:17:42,991 | ERROR | main | ConfigureCassandraCommand | Cassandra configuration settings verified successfully |
2018-09-19 10:17:45,525 | ERROR | main | ConfigureCassandraCommand | Unexpected error, exiting... |
java.lang.Exception: Cannot find configured metrics in cassandra schema
at com.vmware.vcloud.cassandra.cmt.ConfigureCassandraCommand.getSchemaColumns(ConfigureCassandraCommand.java:333)
at com.vmware.vcloud.cassandra.cmt.ConfigureCassandraCommand.updateSchema(ConfigureCassandraCommand.java:266)
at com.vmware.vcloud.cassandra.cmt.ConfigureCassandraCommand.configure(ConfigureCassandraCommand.java:217)
at com.vmware.vcloud.cassandra.cmt.ConfigureCassandraCommand.executeCommand(ConfigureCassandraCommand.java:178)
at com.vmware.vcloud.cassandra.cmt.ConfigureCassandraCommand.executeCommand(ConfigureCassandraCommand.java:51)
at com.vmware.vcloud.common.cmt.HibernateBasedToolCommand.doCommandInternal(HibernateBasedToolCommand.java:43)
at com.vmware.vcloud.common.cmt.AbstractDatabaseToolCommand.doCommand(AbstractDatabaseToolCommand.java:87)
at com.vmware.vcloud.cell.management.internal.AbstractToolCommand.call(AbstractToolCommand.java:177)
at com.vmware.vcloud.cell.management.internal.AbstractToolCommand.call(AbstractToolCommand.java:37)
at com.vmware.vcloud.cell.management.CellManagementTool.start(CellManagementTool.java:299)
at com.vmware.vcloud.cell.management.CellManagementTool.main(CellManagementTool.java:215)
at com.vmware.vcloud.cell.management.CellManagementTool.main(CellManagementTool.java:228)
2018-09-19 10:17:45,540 | DEBUG | main | CellManagementTool | Exiting with completion code: OTHER |
Next step, in my troubleshoot I logged to Cassandra and logged direct to DB and used command "DESCRIBE SCHEMA" in tool "cqlsh":
Interesting output for us is list of existing in Cassandra metrics (what only confirmed error: "Cannot find configured metrics in cassandra schema" :
CREATE TABLE vcloud_metrics.vm_metrics (
vm_id text,
sample_time timestamp,
"cpu.usage.average" bigint,
"cpu.usage.average.instance" text,
"cpu.usage.average.unit" int,
"cpu.usage.maximum" bigint,
"cpu.usage.maximum.instance" text,
"cpu.usage.maximum.unit" int,
"cpu.usagemhz.average" bigint,
"cpu.usagemhz.average.instance" text,
"cpu.usagemhz.average.unit" int,
"disk.provisioned.latest" bigint,
"disk.provisioned.latest.instance" text,
"disk.provisioned.latest.unit" int,
"disk.read.average" bigint,
"disk.read.average.instance" text,
"disk.read.average.unit" int,
"disk.used.latest" bigint,
"disk.used.latest.instance" text,
"disk.used.latest.unit" int,
"disk.write.average" bigint,
"disk.write.average.instance" text,
"disk.write.average.unit" int,
"mem.usage.average" bigint,
"mem.usage.average.instance" text,
"mem.usage.average.unit" int,
org_id text,
vapp_id text,
vdc_id text,
PRIMARY KEY (vm_id, sample_time)
) WITH CLUSTERING ORDER BY (sample_time ASC)
AND bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';
Unfortunatelly i dont have nothing more. I performed some workaround which is looks like working but i think is destructive to historical data:
WORKAROUND:
Inspired by command from this article:
1.Use that same file /tmp/metrics.groovy with content
configuration {
metric("mem.granted.average")
metric("mem.granted.maximum")
}
2. Remove keyspace in Cassandra
"drop keyspace vcloud_metrics"
3. Used command which was used during integration CELLs with Cassandra (on begining) which create schema again:
"/opt/vmware/vcloud-director/bin/cell-management-tool cassandra -configure -create-schema -cluster-nodes 192.x.x.x -username root -password "xxxxxxxx" -port 9042 -ttl 15"
Results (additional metrics looks like are imported from file):
"Verifying Cassandra settings...
Cassandra setting valid for node: 10.3.5.50
Cassandra configuration settings verified successfully
vcloud_metrics keyspace created...
vm_metrics table created...
adding configured metrics to the schema...
adding counter: mem.granted.average
adding counter: mem.granted.maximum
adding counter: cpu.usage.average
adding counter: cpu.usage.maximum
adding counter: cpu.usagemhz.average
adding counter: disk.provisioned.latest
adding counter: disk.read.average
adding counter: disk.used.latest
adding counter: disk.write.average
adding counter: mem.usage.average
vapp_metrics table created...
vdc_metrics table created...
org_metrics table created...
Persisting Cassandra settings...
Success. The monitoring service is now configured to persist data into cassandra nodes(10.3.5.50), vCD cell(s) must be restarted if they are already running."
After restart CELLs looks OK:
In my opinion this is workaround which is not safe.... i think during removing KEYSPACE i removed some historical data...
Some suggestion why i cant update schema with new metrics ? by command:
/opt/vmware/vcloud-director/bin/cell-management-tool cassandra –configure –cluster-nodes 192.x.x.x –username root –password "PASSWORD" –port 9042 –ttl 15 –update-schema
or maybe im doing something wrong 
Sebastian
Message was edited by: SebastianGrugel
Accepted Solutions
OK i have update:
1. My SR case in VMware and resolve which i received:
"- Running the statement "drop keyspace vcloud_metrics" in the database it is expected to drop what historical data it has in that table, it would be the same as dropping tables inside SQL.
- Regarding the update schema issue it is an issue with the cell-management-tool and it will be resolved in the next release of vCD, vCD 9.5. The release of vCD 9.5 was released yesterday
- Asking Engineering was your workaround supported/good etc, the answer was: Seeing as we have a working workaround available and your method gets rid of existing historical data, it wouldnt be officially supported by us and would not be recommended.
I have included the workaround Engineering provided below, so you can see how it would have being resolved in a supported manner, please note the metrics will be different as the case was opened for a slightly different issue
Workaround:
Workaround would be to manually go into one of the Cassandra nodes and update the DB columns manually. Then Cassandra can replicate the new columns to the other nodes as designed.
1) We want to double check that the new values are present in VCD SQL “metric_configuration” table
2) Connect to one of the Cassandra nodes and run the following:
ALTER TABLE vcloud_metrics.vm_metrics ADD "virtualDisk.numberReadAveraged" bigint;
ALTER TABLE vcloud_metrics.vm_metrics ADD "virtualDisk.numberReadAveraged.instance" text;
ALTER TABLE vcloud_metrics.vm_metrics ADD "virtualDisk.numberReadAveraged.unit" int;
ALTER TABLE vcloud_metrics.vm_metrics ADD "virtualDisk.numberWriteAveraged" bigint;
ALTER TABLE vcloud_metrics.vm_metrics ADD "virtualDisk.numberWriteAveraged.instance" text;
ALTER TABLE vcloud_metrics.vm_metrics ADD "virtualDisk.numberWriteAveraged.unit" int;
3) Upon restarting the vCD cells, navigating back to the metrics view for VMs should show virtualDisk.numberReadAveraged and virtualDisk.numberWriteAveraged."
2. This problem was resolved in vCD 9.5 - Tested this today in our test environment with vCD 9.5 and Cassandra 3.11
The problem was really resolved in vCD 9.5
If you will have some doubts contact with me directly.
Thanks
Sebastian
OK i have update:
1. My SR case in VMware and resolve which i received:
"- Running the statement "drop keyspace vcloud_metrics" in the database it is expected to drop what historical data it has in that table, it would be the same as dropping tables inside SQL.
- Regarding the update schema issue it is an issue with the cell-management-tool and it will be resolved in the next release of vCD, vCD 9.5. The release of vCD 9.5 was released yesterday
- Asking Engineering was your workaround supported/good etc, the answer was: Seeing as we have a working workaround available and your method gets rid of existing historical data, it wouldnt be officially supported by us and would not be recommended.
I have included the workaround Engineering provided below, so you can see how it would have being resolved in a supported manner, please note the metrics will be different as the case was opened for a slightly different issue
Workaround:
Workaround would be to manually go into one of the Cassandra nodes and update the DB columns manually. Then Cassandra can replicate the new columns to the other nodes as designed.
1) We want to double check that the new values are present in VCD SQL “metric_configuration” table
2) Connect to one of the Cassandra nodes and run the following:
ALTER TABLE vcloud_metrics.vm_metrics ADD "virtualDisk.numberReadAveraged" bigint;
ALTER TABLE vcloud_metrics.vm_metrics ADD "virtualDisk.numberReadAveraged.instance" text;
ALTER TABLE vcloud_metrics.vm_metrics ADD "virtualDisk.numberReadAveraged.unit" int;
ALTER TABLE vcloud_metrics.vm_metrics ADD "virtualDisk.numberWriteAveraged" bigint;
ALTER TABLE vcloud_metrics.vm_metrics ADD "virtualDisk.numberWriteAveraged.instance" text;
ALTER TABLE vcloud_metrics.vm_metrics ADD "virtualDisk.numberWriteAveraged.unit" int;
3) Upon restarting the vCD cells, navigating back to the metrics view for VMs should show virtualDisk.numberReadAveraged and virtualDisk.numberWriteAveraged."
2. This problem was resolved in vCD 9.5 - Tested this today in our test environment with vCD 9.5 and Cassandra 3.11
The problem was really resolved in vCD 9.5
If you will have some doubts contact with me directly.
Thanks
Sebastian