Transfer Storage growth

[HenningM]

Hi,

I was wondering if the transfer storge is just growing "forever" or is it supposed to be flushed at some point? I am trying to upload some large vApps but after the upload and and copy into the catalog the data just remains at /opt/vmware/vcloud-director/data/transfer. Any suggestions? Can I just delete the contents of this directory to regain the available space?

Regards,

Henning

[admin]

vCD will periodically purge any files in the transfer server spool area once the transfer session expires or the transfer is completed. I would not recommend deleting the files yourself.

[cfor]

VCloud will remove items from transfer space when it is done with them.  A couple setting in VCloud allows you a little control over these (we had to move our settings way up as the default time it keeps an OVF for is not even close to what is needed for WAN downloads of content).

System->Administration->General:

Transfer session timeout - If you just let then download run this will still trigger unless you are using the vcloud interface

Maximum session timeout - This is the max value no matter what.

We found we had to increase both to allow for 12 hours downloads, this is one of the things we hope to see fixed, the transfer space should allow an admin set a spade value, and if the space is not reaches allow vcloud to keep the files longer.

Hope that helps some.

[HenningM]

This is strange. I have waited more then 3 days now and the data is still there. The uploads successfully complete but nothing gets purged. I forgot to mention that this environment has only 1 vCloud cell... Any other ideas?

[cfor]

Is your transfer space NFS backed, or local disk backed.

Who is the owner of your transfer space dir (/opt/vmware/vcloud-director/data/transfer) -- it should be vcloud

Who is the owner of your trasnfer folder /opt/vmware/vcloud-director/data/transfer/SOME_GUID  -- it should be vcloud

If you check in the SOME_GUID folder are the files still in the folder, not just the structure?

As root

"sudo -u vcloud ls -lash SOME_GUID/"

Does this command work? Errors?

I have only seen the issue you are seeing two times, maybe it will be one of them:

1: The cell got restarted and lost track of this folder, in that case just delete it.

2: The cell did not have full access on a NFS volumn that was used for transfer.

[HenningM]

I had the problem already when the space was backed locally, hence I moved it to a NFS share. So now it is NFS backed... The permissions are correct, and the ls command works too:

[root@efra1vcd1 data]# ls -lash
total 24K
4.0K drwxr-x---  5 vcloud vcloud 4.0K Feb 16 15:15 .
4.0K drwxr-x--- 18 vcloud vcloud 4.0K Feb 16 15:12 ..
4.0K drwx------  3 vcloud vcloud 4.0K Feb 21 13:25 activemq
4.0K drwxr-x---  2 vcloud vcloud 4.0K Aug 11  2011 generated-bundles
8.0K drwxr-x--- 46 vcloud vcloud 4.0K Feb 21 16:20 transfer

[root@efra1vcd1 transfer]# sudo -u vcloud ls -lash 01e93184-c9e9-4314-969d-ba0e289e7b22/
total 7.7G
8.0K drwx------  2 vcloud vcloud 1.0K Feb 21 14:47 .
8.0K drwxr-x--- 46 vcloud vcloud 4.0K Feb 21 16:20 ..
7.7G -rw-------  1 vcloud vcloud 7.7G Feb 21 14:09 A1-2001#DM#WIN-5-disk1.vmdk

[cfor]

Something interesting here...

The directory listing of the 01e93184-c9e9-4314-969d-ba0e289e7b22 location.  It does not have the .ovf file in it.  Is this a full listing?

If not then that idea is gone

If it is, then vCloud does not think it is a full copy so it will not remove it.  The ovf file is the last file the "enable download" operation writes.  Once that file exists vcloud will watch the location and delete it after the timeouts.  If this is a full listing I have a feeling that an error was generated during the enable download operation (might be in the vcloud-?-debug.log)

Hope this helps some, I have not seen this exact issue you are seeing - might need to open a support case.

[HenningM]

It is a full listing. After checking, all directories in the transfer storage only have the vmdk files left. I have imported another VM and looked at the debug log:

2012-02-22 15:41:47,503 | DEBUG    | Quartz-pool-1-thread-122  | VAppUploadManagerImpl          | Transfer session: f37ffdfa-c8fc-4912-b674-2b6d0327608a, vApp template: _gw. Completed OVF import to datastore: [vcId=57f6717f-8a81-40ca-ba5c-a8be0c19a0b9, moref=datastore-40] | vcd=b1b060406fb649459ddeb069ddd3d96f,task=0bf9b5df5e3f41bda83988fa8d28180d
2012-02-22 15:41:53,319 | DEBUG    | Quartz-pool-1-thread-122  | VAppUploadManagerImpl          | Transfer session: f37ffdfa-c8fc-4912-b674-2b6d0327608a. vApp template: _gw. Transfer processing is complete. | vcd=b1b060406fb649459ddeb069ddd3d96f,task=0bf9b5df5e3f41bda83988fa8d28180d
2012-02-22 15:42:00,034 | DEBUG    | Quartz-pool-1-thread-122  | TransferSessionCleanerJob      | Removing successful transfer session: f37ffdfa-c8fc-4912-b674-2b6d0327608a |
2012-02-22 15:42:00,035 | DEBUG    | Quartz-pool-1-thread-122  | TransferSessionUtils           | Transfer session: f37ffdfa-c8fc-4912-b674-2b6d0327608a, resource file: A1-2001%23_gw-disk1.vmdk(1de2d0c7-ae67-4639-97f5-732501dbe973). Resource file has already been deleted: /opt/vmware/vcloud-director/data/transfer/f37ffdfa-c8fc-4912-b674-2b6d0327608a/A1-2001%23_gw-disk1.vmdk. Removing database entry. |
2012-02-22 15:42:00,038 | DEBUG    | Quartz-pool-1-thread-122  | TransferSessionUtils           | Transfer session: f37ffdfa-c8fc-4912-b674-2b6d0327608a. Failed to delete transfer session directory or transfer session directory does not exist: /opt/vmware/vcloud-director/data/transfer/f37ffdfa-c8fc-4912-b674-2b6d0327608a |
2012-02-22 15:44:00,037 | DEBUG    | Quartz-pool-1-thread-122  | TransferSessionCleanerJob      | Removing successful transfer session: f37ffdfa-c8fc-4912-b674-2b6d0327608a |
2012-02-22 15:44:00,038 | DEBUG    | Quartz-pool-1-thread-122  | TransferSessionUtils           | Transfer session: f37ffdfa-c8fc-4912-b674-2b6d0327608a. Failed to delete transfer session directory or transfer session directory does not exist: /opt/vmware/vcloud-director/data/transfer/f37ffdfa-c8fc-4912-b674-2b6d0327608a |
2012-02-22 15:46:00,107 | DEBUG    | Quartz-pool-1-thread-122  | TransferSessionCleanerJob      | Removing successful transfer session: f37ffdfa-c8fc-4912-b674-2b6d0327608a |
2012-02-22 15:46:00,108 | DEBUG    | Quartz-pool-1-thread-122  | TransferSessionUtils           | Transfer session: f37ffdfa-c8fc-4912-b674-2b6d0327608a. Failed to delete transfer session directory or transfer session directory does not exist: /opt/vmware/vcloud-director/data/transfer/f37ffdfa-c8fc-4912-b674-2b6d0327608a |

So it looks like the transfer finishes, and then for some reason it thinks that the file already got deleted, which in turn triggers deletion of the database entry. When trying to remove the directory it cannot be removed as it is not empty. This is very strange