VMware Cloud Community
EGV
Contributor
Contributor

VIO upgrade 1.03 to 2.0.1 failed.

  when upgrading vio cluster through the GUI we get the error message "Failed to upgrade Openstack. Failed to calculate provisioning plan during upgrade"

the log file /var/log/oms/oms.log on the management node contains the following error message:

Caused by: com.vmware.openstack.exception.BddException: Network dvS Cluster_VIO not found.

  at com.vmware.openstack.exception.BddException.NOT_FOUND(BddException.java:106)

  at com.vmware.openstack.exception.BddException.NOT_FOUND(BddException.java:110)

  at com.vmware.openstack.manager.ClusterPlanManager.allocatNetworkIp(ClusterPlanManager.java:388)

  at com.vmware.openstack.manager.ClusterPlanManager.expandNetworkConfig(ClusterPlanManager.java:302)

  at com.vmware.openstack.manager.ClusterPlanManager.createClusterPlan(ClusterPlanManager.java:129)

  at com.vmware.openstack.manager.ClusterPlanManager$$FastClassBySpringCGLIB$$8c34e68.invoke()

  at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)

  at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:640)

  at com.vmware.openstack.manager.ClusterPlanManager$$EnhancerBySpringCGLIB$$41638c30.createClusterPlan()

  at com.vmware.openstack.manager.ClusterManager.prepareGreenPanelSpec(ClusterManager.java:569)

  ... 75 more

We are using VIO in combination with the Distributed Switch plugin. This network "dvS Cluster_VIO", which he complains about in the error message exists in the networks list in the openstack plugin for vCenter, and it is a portgroup on our Distributed Switch. So I don't get why the installer can't see it.

We are pretty handy in managing Linux systems, but I could not find any configuration fault in conf files or the vpostgres database. Can anybody give me a hint where this fault is coming from?

Tags (1)
Reply
0 Kudos
9 Replies
admin
Immortal
Immortal

EGV,

    could you please attach the whole oms.log? thanks.

Reply
0 Kudos
ZhangAdam
VMware Employee
VMware Employee

Did you do some operation regarding the "dvs Cluster_VIO" port group after deployed vio 1.0, such as renaming, deleting/recreating etc?

It seems VIO can not find the port group.

Reply
0 Kudos
EGV
Contributor
Contributor

Okay the complete story is:

When we rolled out the 1.0.3 VIO openstack cluster, there was an existing port group 'dvs Cluster_VIO', which I forgot to delete. Therefore the installer created a portroup 'dvs Cluster_VIO1' on which it put the dhcpservers. The installed icehouse cluster worked fine, but when we tried to upgrade it to VIO 2.0 we got the error message "Network dvS Cluster_VIO1 has not been defined". Then I updated the vpostgres database tables and removed the '1' for the portgroup name, and did the same for the portgroup in vCenter. The error message was exactly the same, but with the 1 missing of course.

I attached my oms.log file.

Reply
0 Kudos
ZhangAdam
VMware Employee
VMware Employee

That explains.

what table did u touched?  Update will use table cluster_profile, the deployment profile for 1.0 is saved there as a json text, you need to update it too to make upgrade work.

Reply
0 Kudos
EGV
Contributor
Contributor

I did a database export to psql file and searched for all tables which contained this value and updated them. In the 'cluster' table were some rows which contained the old value, which I've updated. The cluster_profile table doesn't contain any rows with the value 'dvS Cluster_VIO'. I've done some testing with updating the JSON value in the cluster_profile table, but without any results. Do you have an example how this value should look like?

Reply
0 Kudos
ZhangAdam
VMware Employee
VMware Employee

In Cluster_Profile.profile column, it is json representation of cluster_profile used to provision cluster 1.0, i think it should contain the network information there, it looks like this in my environment:

It's located at attributes of Controller node group:

{"name":"VIO","nodeGroups":[{"name":"LoadBalancer","role":"LoadBalancer","instanceNum":2,"cpuNum":2,"memCapacityMB":4096,"swapRatio":1.0,"haFlag":"on","attributes":{"public_vip":"10.155.20.246","public_hostname":""}},{"name":"Controller","role":"Controller","instanceNu

m":2,"cpuNum":8,"memCapacityMB":16384,"swapRatio":1.0,"haFlag":"on","attributes":{"vcenter_user":"root","syslog_server":"10.117.171.108","token_expiration_time":"7200","nsxv_vdn_scope_moref":"vdnscope-1","vcenter_password":"******","nsxv_dvs_moref":"dvs-68","syslog_serv

er_protocol":"udp","cinder_folder":"Volumes","nsxv_external_network_moref":"PG-4001","vcenter_insecure":"false","syslog_server_tag":"VIO","admin_password":"******","glance_datastores":"OpenstackDC:ds21-01:100","nsxv_edge_cluster_moref":"domain-c27","nsxv_username":"admi

n","neutron_backend":"nsxv","keystone_backend":"sql","vcenter_ip":"10.155.21.180","syslog_server_port":"514","glance_folder":"/images","nsxv_manager":"10.155.21.184","admin_user":"admin","nsxv_edge_cluster_name":"edgeCluster2","admin_tenant_name":"admin","nsxv_password"

:"******"}},{"name":"DB","role":"Database","instanceNum":3,"cpuNum":4,"memCapacityMB":16384,"swapRatio":1.0,"haFlag":"on"},{"name":"RabbitMQ","role":"MessageQueue","instanceNum":2,"cpuNum":4,"memCapacityMB":16384,"swapRatio":1.0,"haFlag":"on"},{"name":"ObjectStorage","r

ole":"ObjectStorage","instanceNum":1,"cpuNum":2,"memCapacityMB":4096,"swapRatio":1.0,"haFlag":"on"},{"name":"Memcache","role":"MemoryCache","instanceNum":2,"cpuNum":2,"memCapacityMB":16384,"swapRatio":1.0,"haFlag":"on"},{"name":"ComputeDriver","role":"Compute","instance

Num":1,"cpuNum":2,"memCapacityMB":4096,"swapRatio":1.0,"haFlag":"on","attributes":{"region_name":"nova"},"nodeAttributes":[{"cluster_name":"computeCluster3","datastore_regex":"dscomp_31","cluster_moid":"domain-c29"}]}],"vcClusters":[{"name":"mgmtCluster","moid":"domain-

c7"}],"networkConfig":{"METADATA_NETWORK":["PG-1902"],"DATA_NETWORK":["PG-1902"],"EXTERNAL_NETWORK":["PG-2004"],"REPL_NETWORK":["PG-1902"],"MGT_NETWORK":["PG-1902"]},"networkings":[{"name":"PG-2004","portGroup":"PG-2004","isDhcp":false,"type":"static","dns1":"10.132.71.

1","dns2":"10.132.71.1","ipBlocks":[{"beginIp":"10.155.20.249","endIp":"10.155.20.250"}],"gateway":"10.155.20.253","netmask":"255.255.255.0","dnsList":["10.132.71.1","10.132.71.1"]},{"name":"PG-1902","portGroup":"PG-1902","isDhcp":false,"type":"static","dns1":"10.132.71

.1","dns2":"10.132.71.1","ipBlocks":[{"beginIp":"10.155.21.190","endIp":"10.155.21.210"}],"gateway":"10.155.21.253","netmask":"255.255.255.0","dnsList":["10.132.71.1","10.132.71.1"]}],"specFile":true,"attributes":{"plan":"[{\"name\":\"VIO-LoadBalancer-0\",\"moId\":null,

\"rack\":null,\"hostName\":\"10.155.19.33\",\"ipConfigs\":{\"EXTERNAL_NETWORK\":[{\"portGroupName\":\"PG-2004\",\"networkName\":\"PG-2004\",\"trafficType\":\"EXTERNAL_NETWORK\",\"ipAddress\":\"10.155.20.249\"}],\"MGT_NETWORK\":[{\"portGroupName\":\"PG-1902\",\"networkNa

me\":\"PG-1902\",\"trafficType\":\"MGT_NETWORK\",\"ipAddress\":\"10.155.21.200\"}]},\"status\":null,\"action\":null,\"role\":\"LoadBalancer\",\"memory\":4096,\"volumes\":null,\"cpuNumber\":2,\"attributes\":null,\"actionFailed\":false,\"errMessage\":null,\"version\":\"Ea

rlier\",\"targetSystemDs\":\"ds33-03\"},{\"name\":\"VIO-LoadBalancer-1\",\"moId\":null,\"rack\":null,\"hostName\":\"10.155.19.33\",\"ipConfigs\":{\"EXTERNAL_NETWORK\":[{\"portGroupName\":\"PG-2004\",\"networkName\":\"PG-2004\",\"trafficType\":\"EXTERNAL_NETWORK\",\"ipAd

dress\":\"10.155.20.250\"}],\"MGT_NETWORK\":[{\"portGroupName\":\"PG-1902\",\"networkName\":\"PG-1902\",\"trafficType\":\"MGT_NETWORK\",\"ipAddress\":\"10.155.21.201\"}]},\"status\":null,\"action\":null,\"role\":\"LoadBalancer\",\"memory\":4096,\"volumes\":null,\"cpuNum

ber\":2,\"attributes\":null,\"actionFailed\":false,\"errMessage\":null,\"version\":\"Earlier\",\"targetSystemDs\":\"ds33-02\"},{\"name\":\"VIO-Controller-0\",\"moId\":null,\"rack\":null,\"hostName\":\"10.155.19.33\",\"ipConfigs\":{\"MGT_NETWORK\":[{\"portGroupName\":\"P

G-1902\",\"networkName\":\"PG-1902\",\"trafficType\":\"MGT_NETWORK\",\"ipAddress\":\"10.155.21.196\"}]},\"status\":null,\"action\":null,\"role\":\"Controller\",\"memory\":16384,\"volumes\":null,\"cpuNumber\":8,\"attributes\":null,\"actionFailed\":false,\"errMessage\":nu

ll,\"version\":\"Earlier\",\"targetSystemDs\":\"ds33-05 (1)\"},{\"name\":\"VIO-Controller-1\",\"moId\":null,\"rack\":null,\"hostName\":\"10.155.19.33\",\"ipConfigs\":{\"MGT_NETWORK\":[{\"portGroupName\":\"PG-1902\",\"networkName\":\"PG-1902\",\"trafficType\":\"MGT_NETWO

RK\",\"ipAddress\":\"10.155.21.197\"}]},\"status\":null,\"action\":null,\"role\":\"Controller\",\"memory\":16384,\"volumes\":null,\"cpuNumber\":8,\"attributes\":null,\"actionFailed\":false,\"errMessage\":null,\"version\":\"Earlier\",\"targetSystemDs\":\"ds33-04\"},{\"na

me\":\"VIO-DB-0\",\"moId\":null,\"rack\":null,\"hostName\":\"10.155.19.33\",\"ipConfigs\":{\"REPL_NETWORK\":[{\"portGroupName\":\"PG-1902\",\"networkName\":\"PG-1902\",\"trafficType\":\"REPL_NETWORK\",\"ipAddress\":\"10.155.21.193\"}],\"MGT_NETWORK\":[{\"portGroupName\"

:\"PG-1902\",\"networkName\":\"PG-1902\",\"trafficType\":\"MGT_NETWORK\",\"ipAddress\":\"10.155.21.193\"}]},\"status\":null,\"action\":null,\"role\":\"Database\",\"memory\":16384,\"volumes\":null,\"cpuNumber\":4,\"attributes\":null,\"actionFailed\":false,\"errMessage\":

null,\"version\":\"Earlier\",\"targetSystemDs\":\"ds33-05 (1)\"},{\"name\":\"VIO-DB-1\",\"moId\":null,\"rack\":null,\"hostName\":\"10.155.19.33\",\"ipConfigs\":{\"REPL_NETWORK\":[{\"portGroupName\":\"PG-1902\",\"networkName\":\"PG-1902\",\"trafficType\":\"REPL_NETWORK\"

,\"ipAddress\":\"10.155.21.194\"}],\"MGT_NETWORK\":[{\"portGroupName\":\"PG-1902\",\"networkName\":\"PG-1902\",\"trafficType\":\"MGT_NETWORK\",\"ipAddress\":\"10.155.21.194\"}]},\"status\":null,\"action\":null,\"role\":\"Database\",\"memory\":16384,\"volumes\":null,\"cp

uNumber\":4,\"attributes\":null,\"actionFailed\":false,\"errMessage\":null,\"version\":\"Earlier\",\"targetSystemDs\":\"ds33-03\"},{\"name\":\"VIO-DB-2\",\"moId\":null,\"rack\":null,\"hostName\":\"10.155.19.33\",\"ipConfigs\":{\"REPL_NETWORK\":[{\"portGroupName\":\"PG-1

902\",\"networkName\":\"PG-1902\",\"trafficType\":\"REPL_NETWORK\",\"ipAddress\":\"10.155.21.195\"}],\"MGT_NETWORK\":[{\"portGroupName\":\"PG-1902\",\"networkName\":\"PG-1902\",\"trafficType\":\"MGT_NETWORK\",\"ipAddress\":\"10.155.21.195\"}]},\"status\":null,\"action\"

:null,\"role\":\"Database\",\"memory\":16384,\"volumes\":null,\"cpuNumber\":4,\"attributes\":null,\"actionFailed\":false,\"errMessage\":null,\"version\":\"Earlier\",\"targetSystemDs\":\"ds33-01\"},{\"name\":\"VIO-RabbitMQ-0\",\"moId\":null,\"rack\":null,\"hostName\":\"1

0.155.19.33\",\"ipConfigs\":{\"MGT_NETWORK\":[{\"portGroupName\":\"PG-1902\",\"networkName\":\"PG-1902\",\"trafficType\":\"MGT_NETWORK\",\"ipAddress\":\"10.155.21.202\"}]},\"status\":null,\"action\":null,\"role\":\"MessageQueue\",\"memory\":16384,\"volumes\":null,\"cpuN

umber\":4,\"attributes\":null,\"actionFailed\":false,\"errMessage\":null,\"version\":\"Earlier\",\"targetSystemDs\":\"ds33-03\"},{\"name\":\"VIO-RabbitMQ-1\",\"moId\":null,\"rack\":null,\"hostName\":\"10.155.19.33\",\"ipConfigs\":{\"MGT_NETWORK\":[{\"portGroupName\":\"P

G-1902\",\"networkName\":\"PG-1902\",\"trafficType\":\"MGT_NETWORK\",\"ipAddress\":\"10.155.21.203\"}]},\"status\":null,\"action\":null,\"role\":\"MessageQueue\",\"memory\":16384,\"volumes\":null,\"cpuNumber\":4,\"attributes\":null,\"actionFailed\":false,\"errMessage\":

null,\"version\":\"Earlier\",\"targetSystemDs\":\"ds33-02\"},{\"name\":\"VIO-ObjectStorage-0\",\"moId\":null,\"rack\":null,\"hostName\":\"10.155.19.33\",\"ipConfigs\":{\"MGT_NETWORK\":[{\"portGroupName\":\"PG-1902\",\"networkName\":\"PG-1902\",\"trafficType\":\"MGT_NETW

ORK\",\"ipAddress\":\"10.155.21.204\"}]},\"status\":null,\"action\":null,\"role\":\"ObjectStorage\",\"memory\":4096,\"volumes\":null,\"cpuNumber\":2,\"attributes\":null,\"actionFailed\":false,\"errMessage\":null,\"version\":\"Earlier\",\"targetSystemDs\":\"ds33-03\"},{\

"name\":\"VIO-Memcache-0\",\"moId\":null,\"rack\":null,\"hostName\":\"10.155.19.33\",\"ipConfigs\":{\"MGT_NETWORK\":[{\"portGroupName\":\"PG-1902\",\"networkName\":\"PG-1902\",\"trafficType\":\"MGT_NETWORK\",\"ipAddress\":\"10.155.21.198\"}]},\"status\":null,\"action\":

null,\"role\":\"MemoryCache\",\"memory\":4096,\"volumes\":null,\"cpuNumber\":2,\"attributes\":null,\"actionFailed\":false,\"errMessage\":null,\"version\":\"Earlier\",\"targetSystemDs\":\"ds33-03\"},{\"name\":\"VIO-Memcache-1\",\"moId\":null,\"rack\":null,\"hostName\":\"

10.155.19.33\",\"ipConfigs\":{\"MGT_NETWORK\":[{\"portGroupName\":\"PG-1902\",\"networkName\":\"PG-1902\",\"trafficType\":\"MGT_NETWORK\",\"ipAddress\":\"10.155.21.199\"}]},\"status\":null,\"action\":null,\"role\":\"MemoryCache\",\"memory\":4096,\"volumes\":null,\"cpuNu

mber\":2,\"attributes\":null,\"actionFailed\":false,\"errMessage\":null,\"version\":\"Earlier\",\"targetSystemDs\":\"ds33-03\"},{\"name\":\"VIO-ComputeDriver-0\",\"moId\":null,\"rack\":null,\"hostName\":\"10.155.19.33\",\"ipConfigs\":{\"MGT_NETWORK\":[{\"portGroupName\"

:\"PG-1902\",\"networkName\":\"PG-1902\",\"trafficType\":\"MGT_NETWORK\",\"ipAddress\":\"10.155.21.205\"}]},\"status\":null,\"action\":null,\"role\":\"Compute\",\"memory\":4096,\"volumes\":null,\"cpuNumber\":2,\"attributes\":null,\"actionFailed\":false,\"errMessage\":nu

ll,\"version\":\"Earlier\",\"targetSystemDs\":\"ds33-02\"}]"}}

Reply
0 Kudos
EGV
Contributor
Contributor

I've compared your JSON object to mine, but I couldn't find any real differences.

I've rebuilt my complete VIO Openstack in a fake environment (ESX in a VM) with the exact same IP addresses, Distributed switch settings, etc, as my production environment. The only difference is that there was no existing portgroup dvS Cluster_VIO for the dhcp servers, so the installer created it.

Unfortunately I couldn't find any differences in the oms database tables and json objects, so I can't find the source of the error message "Network dvS Cluster_VIO not found." which is causing the upgrade to fail. Has anybody any other ideas? Otherwise I'm going to destroy my production cluster and rebuild it using a database backup

Reply
0 Kudos
ZhangAdam
VMware Employee
VMware Employee

We have found the root cause and fixed in 2.5 branch.

When we do upgrade from dvs setup, it fails in the plan phase because it is looking for DATA NETWORK which is created by in the first deployment, in your case "dvs Cluster_VIO1” and it is not exist in the spec, So we need to change its value some network that exist.

So the fix is to update the value in the cluster table. Update  column cluster.network_config, set the DATA NETWORK to the same value with MGT NETWORK.

For example:

{"EXTERNAL_NETWORK":[{"portGroupName":"PG-2004","networkName":"PG-2004","trafficType":"EXTERNAL_NETWORK"}],"DATA_NETWORK":[{"portGroupName”:”dvs xxx","networkName”:”dvs xxx","trafficType":"DATA_NETWORK"}],"REPL_NETWORK":[{"portGroupName":"PG-1902","networkName":"PG-190

2","trafficType":"REPL_NETWORK"}],"MGT_NETWORK":[{"portGroupName":"PG-1902","networkName":"PG-1902","trafficType":"MGT_NETWORK"}]}

Replace the value to

{"EXTERNAL_NETWORK":[{"portGroupName":"PG-2004","networkName":"PG-2004","trafficType":"EXTERNAL_NETWORK"}],"DATA_NETWORK":[{"portGroupName":"PG-1902","networkName":"PG-1902","trafficType":"DATA_NETWORK"}],"REPL_NETWORK":[{"portGroupName":"PG-1902","networkName":"PG-190

2","trafficType":"REPL_NETWORK"}],"MGT_NETWORK":[{"portGroupName":"PG-1902","networkName":"PG-1902","trafficType":"MGT_NETWORK"}]}

Reply
0 Kudos
JunW
Enthusiast
Enthusiast

The update of column cluster.network_config may have other issues. We will provide a hot fix soon. Thanks.

Reply
0 Kudos