VMware Cloud Community
JegwangLee
Contributor
Contributor
‎07-02-2015 10:04 PM

Unable to create snapshot

Hello

I am testing VIO. and I have deployed VIO with NSX, and created network , uploaded the image and created instance.

Most are successful. but unable to create snapshot.

I attach log file(compute, controller0 controller1) and VIO configuration screen shot .

please let me know what i do.

이미지 249.png

이미지 250.png

Compute0_nova-compute.log.zip
Contoller1_glance_api.log.zip
Contoller0_glance_api.log.zip
11 Replies
admin
Immortal
Immortal
‎07-05-2015 09:34 PM

You mention you are using NSX. Would  you mind sharing your VMware NSX contact.

What version of the product are you using? 1.0 , 1.0.1 or 1.0.2 ?

I will open a bug to investigate the issue. Will post back if we need more info.

arvind

0 Kudos
rgerganov
VMware Employee
VMware Employee
‎07-06-2015 03:39 AM

How big is the instance that you try to snapshot (disk size)? Do you the same problem with smaller instances (e.g. 1GB disk)?

0 Kudos
raxidex
Contributor
Contributor
‎07-06-2015 09:55 AM

I'm facing the same issue. After a few moments the snapshot status is displayed with "killed".

The only time it worked was when I created a new flavor with the minimum requisites of the ubuntu image (512 RAM and 5 GB disk).

I'd one problem with another openstack installation (not the VIO), where the snapshot took so long that the token expired during the process, but it doesn't seem related here since the process is stopped in a few minutes.

0 Kudos
JegwangLee
Contributor
Contributor
‎07-06-2015 07:28 PM

NSX version is 6.1.3

이미지 272.png

This error is  because of nsx??

0 Kudos
JegwangLee
Contributor
Contributor
‎07-06-2015 08:38 PM

I tested two instance(ubuntu 14.04 and Win7)

but all failed

이미지 275.png

이미지 280.png

Thanks

0 Kudos
rgerganov
VMware Employee
VMware Employee
‎07-07-2015 02:10 AM

You can try to workaround the problem by increasing the timeout on each ESX host and each load balancer.

To increase the HTTP timeout on a ESX host:

1. SSH into ESX

2. Open /etc/vmware/rhttpproxy/config.xml for editing

3. Set 'readTimeoutMs' to 300000ms like this:

<vmacore>

  ...

  <http>

    ...

    <readTimeoutMs>300000</readTimeoutMs>

  </http>

  ...

</vmacore>

4. Restart the rhttpproxy process:

# /etc/init.d/rhttpproxy restart

To increase the timeout on a loadbalancer:

1. SSH into loadbalancer

2. Open /etc/haproxy/haproxy.cfg for editing

3. Add the following to the 'glance-api' section:

listen glance-api

   timeout client 300000ms

   timeout server 300000ms

   ...

4. Restart the haproxy service:

# /etc/init.d/haproxy restart

dalo
Hot Shot
Hot Shot
‎07-08-2015 06:57 AM

Had the same issue here, but with the workaround it works for most images. Thank you rgerganov.

But with the ova from Ubuntu 14.04.2 LTS (Trusty Tahr) Daily Build [20150702] I get the Status "killed". Does this make sense, is this a issue with the image? Can someone confirm this?

0 Kudos
JegwangLee
Contributor
Contributor
‎07-08-2015 07:49 PM

I tested as you lead me but result is killed.


이미지 283.png

0 Kudos
sabarim
VMware Employee
VMware Employee
‎07-22-2015 02:41 PM

In addition to the rhttproxy timeout on the ESX hosts as suggested by rgerganov, you also need to increase the timeout value on the hostd configuration.

Host Configuration Change

1.  SSH to each ESXi Host in the environment.

2.  Backup both /etc/vmware/rhttpproxy/config.xml and /etc/hostd/config.xml. Edit the original config.xml files with same content changes described earlier  i.e

<vmacore>

…

<http>

…

<readTimeoutMs>600000</readTimeoutMs>

…

</http>

…

<ssl>

…

<handshakeTimeoutMs>600000</handshakeTimeoutMs>

…

</ssl>

</vmacore>

2.  /etc/init.d/rhttpproxy restart

3.  /etc/init.d/hostd restart

Thanks

RKuchukbaev
Enthusiast
Enthusiast
‎08-13-2015 02:30 AM

Sabari, it worked for me.

0 Kudos
HiroyasuOHYAMA
Contributor
Contributor
‎01-08-2016 12:12 AM

I'm facing the same problem using NSX-V (Version: 6.2.0 Build 2986609)

I could create snapshot at compute node, but I received 500 error at uploading. Here is the syslog error-log from compute node.

Jan 8 07:40:02 compute01 2016-01-08 07:40:02.074 8039 ERROR oslo_messaging.rpc.dispatcher [req-d6bf535d-0b89-47fb-95d6-c6c6ea6cbe11 c83590b83b774f80b32d4f09edb65504 eac0feaf667a4ec98534a897b630544d - - -] Exception during message handling: <html>#012 <head>#012 <title>500 Internal Server Error</title>#012 </head>#012 <body>#012 <h1>500 Internal Server Error</h1>#012 Failed to upload image fe9ec08c-63ca-4016-82ad-f4ff54fb4f60<br /><br />#012#012 </body>#012</html> (HTTP 500)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher executor_callback))#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher executor_callback)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6891, in snapshot_instance#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher return self.manager.snapshot_instance(ctxt, image_id, instance)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher payload)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 346, in decorated_function#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance_uuid=instance_uuid)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 317, in decorated_function#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 374, in decorated_function#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 362, in decorated_function#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 422, in decorated_function#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher instance=instance)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 412, in decorated_function#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher *args, **kwargs)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3304, in snapshot_instance#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher task_states.IMAGE_SNAPSHOT)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3334, in _snapshot_instance#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher update_task_state)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 527, in snapshot#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher self._vmops.snapshot(context, instance, image_id, update_task_state)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vmops.py", line 1067, in snapshot#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher vmdk_size=vmdk.capacity_in_bytes)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/images.py", line 479, in upload_image_stream_optimized#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher IMAGE_API.update(context, image_id, image_metadata, data=read_handle)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/image/api.py", line 130, in update#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher purge_props=purge_props)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 398, in update#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher _reraise_translated_image_exception(image_id)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 396, in update#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher image_id, **image_meta)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 219, in call#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher return getattr(client.images, method)(*args, **kwargs)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 329, in update#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher resp, body = self.client.put(url, headers=hdrs, data=image_data)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 265, in put#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher return self._request('PUT', url, **kwargs)#0122016-01-08 07:40:02.074 8039 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", l

And following the error log of Glance server (glance-api.log) when this problem is occurred.

2016-01-08 07:39:59.094 7331 ERROR glance.api.v1.upload_utils [req-1a0d7ebc-987a-4a9b-9b74-668f072eff1a c83590b83b774f80b32d4f09edb65504 eac0feaf667a4ec98534a897b630544d - - -] Failed to upload image fe9ec08c-63ca-4016-82ad-f4ff54fb4f60

2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils Traceback (most recent call last):

2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/dist-packages/glance/api/v1/upload_utils.py", line 113, in upload_data_to_store

2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils context=req.context)

2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/dist-packages/glance_store/backend.py", line 339, in store_add_to_backend

2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils context=context)

2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/dist-packages/glance_store/capabilities.py", line 226, in op_checker

2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils return store_op_fun(store, *args, **kwargs)

2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/dist-packages/glance_store/_drivers/vmware_datastore.py", line 516, in add

2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils response = session.put(url, data=data, headers=headers)

2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 510, in put

2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils return self.request('PUT', url, data=data, **kwargs)

2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 457, in request

2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils resp = self.send(prep, **send_kwargs)

2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 569, in send

2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils r = adapter.send(request, **kwargs)

2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 389, in send

2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils r = low_conn.getresponse()

2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/httplib.py", line 1051, in getresponse

2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils response.begin()

2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/httplib.py", line 415, in begin

2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils version, status, reason = self._read_status()

2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/httplib.py", line 379, in _read_status

2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils raise BadStatusLine(line)

2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils BadStatusLine: ''

2016-01-08 07:39:59.094 7331 TRACE glance.api.v1.upload_utils

2016-01-08 07:39:59.099 7331 DEBUG glance.registry.client.v1.api [req-1a0d7ebc-987a-4a9b-9b74-668f072eff1a c83590b83b774f80b32d4f09edb65504 eac0feaf667a4ec98534a897b630544d - - -] Updating image metadata for image fe9ec08c-63ca-4016-82ad-f4ff54fb4f60... update_image_metadata /usr/lib/python2.7/dist-packages/glance/registry/client/v1/api.py:168

2016-01-08 07:39:59.099 7331 DEBUG glance.common.client [req-1a0d7ebc-987a-4a9b-9b74-668f072eff1a c83590b83b774f80b32d4f09edb65504 eac0feaf667a4ec98534a897b630544d - - -] Constructed URL: http://192.168.84.30:9191/images/fe9ec08c-63ca-4016-82ad-f4ff54fb4f60 _construct_url /usr/lib/python2.7/dist-packages/glance/common/client.py:401

2016-01-08 07:39:59.134 7331 DEBUG glance.registry.client.v1.client [req-1a0d7ebc-987a-4a9b-9b74-668f072eff1a c83590b83b774f80b32d4f09edb65504 eac0feaf667a4ec98534a897b630544d - - -] Registry request PUT /images/fe9ec08c-63ca-4016-82ad-f4ff54fb4f60 HTTP 200 request id req-req-9581d029-e8b2-471e-845c-10647f42db45 do_request /usr/lib/python2.7/dist-packages/glance/registry/client/v1/client.py:124

2016-01-08 07:39:59.135 7331 INFO eventlet.wsgi.server [req-1a0d7ebc-987a-4a9b-9b74-668f072eff1a c83590b83b774f80b32d4f09edb65504 eac0feaf667a4ec98534a897b630544d - - -] 192.168.84.45,192.168.84.42 - - [08/Jan/2016 07:39:59] "PUT /v1/images/fe9ec08c-63ca-4016-82ad-f4ff54fb4f60 HTTP/1.1" 500 430 620.496563

0 Kudos