VMware Cloud Community
waldorfo2
Contributor
Contributor

Very slow transfers over SAMBA after upgrade to 3.0.2 build-52542

Hello,

Has anyone noticed any problems with samba after upgrading to 3.0.2?

We’ve observed that our backups take forever after upgrading to 3.0.2. We are using vcbmounter command in the service console to transfer snapshots to samba share on Windows server. Before upgrade, 20GB VM took 2 hours to complete, now it’s 20 hours or more.

I think the problem is related to samba client in ESX console. I tried shares on different Windows servers and the problem is the same. To rule out vcbmounter I tried to just copy file to a share. It takes 10 minutes to copy 50MB file!

The problem occurs across all 3 ESX servers in our env.

I thought that it might be something to do with network card/drivers etc. but tried scp copy to the service console and speed was fine. I tried with both e1000 and tg3 network cards.

Is anyone experiencing similar problem?

Reply
0 Kudos
23 Replies
amr77
Contributor
Contributor

I have the same problem - 0.4MBs transfer rates to smb mounts.

Please see the following thread:

http://www.vmware.com/community/thread.jspa?threadID=97117

There seems to be a problem with the service console network in 3.0.2.

I have raised a service request but I have not been given any feedback at the moment.

Texiwill
Leadership
Leadership

Hello,

You could try the latest Samba release. This is one I compiled just for ESX.

http://www.astroarch.com/virtual/samba.html

I do not know if it will solve your problem or not. The one ESX uses is very much out of date.

Best regards,

Edward

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
waldorfo2
Contributor
Contributor

Thanks Ed

I will give it a try. At least we will know if the problem is related to samba packages or is it something else.

However, I don't think it's a good solution for production machines. I would prefer official package from VMWare:( support etc.

Reply
0 Kudos
Texiwill
Leadership
Leadership

Hello,

I wish as well, the problem is that ESX is based on RHEL3 U6 and adding in the latest packages will be an issue for them I would think. You can open a feature request but these are just rebuilt versions of Fedora 7 packages. Pretty much as close as we can get to RHEL packages at the moment.

Best regards,

Edward

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
Reply
0 Kudos
waldorfo2
Contributor
Contributor

Edward,

Unfortunately I did not have luck with your packages. When I tried to mount my share I got error message that smb fs is not supported.

There is no smbmount in your packages. But I think it's ok for the latest version of samba-client. However mount.cifs didn't work for me either...

Is there something I am missing?

Reply
0 Kudos
Texiwill
Leadership
Leadership

Hello,

I will investigate... mount.cifs will NOT work or be present in RHEL3 as it requires a 2.6 kernel to use. Hopefully the next version of ESX.

smbmount should have been present.... Hrm. I will get back to you.

Best regards,

Edward

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
Reply
0 Kudos
DMcCoy
Contributor
Contributor

I'm not sure if its related but my transfer rates with the backup exec client from 3.0.1 to 3.0.2 has been reduced dramatically, so it could be a networking issue more than smb

I don't know by how much yet seeing as the backup is still going (90 mins longer so far with everything still the same except for being patched to 3.0.2)

Reply
0 Kudos
DMcCoy
Contributor
Contributor

Something is deffinately wrong with 3.0.2, here at least. My backup time for the host servers has increased by 1000%!

Reply
0 Kudos
Texiwill
Leadership
Leadership

Hello,

I am rebuilding my samba packages with smbmount support. I will have them later today. Any takers on testing with them?

Best regards,

Edward

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
Reply
0 Kudos
waldorfo2
Contributor
Contributor

Hi Edward,

I can try your new packages.

Reply
0 Kudos
bluepenguin
Enthusiast
Enthusiast

I believe that the problem is related to the high network latency in ESX 3.0.2, see this:

http://www.vmware.com/community/thread.jspa?threadID=97117&tstart=0

CIFS (SMB) on ESX COS transfers 3 packages (4096 byte) to the windows server with the share, then the COS waits for an acknowledge.

If acknowledge takes 10 ms each 4096 bytes you transfer, you can have a maximum transfer rate of about 400 KB/s

I filed an SR on this, this did not happen with ESX 3.0.1

Reply
0 Kudos
Texiwill
Leadership
Leadership

Hello,

New Samba packages are available at http://www.astroarch.com/virtual/samba.html

Best regards,

Edward

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
Reply
0 Kudos
PezJunkie
Enthusiast
Enthusiast

All this stuff is just installed via .rpm when you upgrade, right? Is it possible to remove the newer version of samba and then reinstall from the 3.0.1 .rpm file?

(I am not a linux guru by any means... feel free to correct me if I'm wrong here.)

Reply
0 Kudos
ForgeFlakshack
Enthusiast
Enthusiast

Just wanted to add that I am also seeing this problem with very slow SAMBA transfers. Previously we would see ~30,000 KBps on our SAMBA backups (using vcbMounter). After installing 3.0.2, we only see ~400 KBps.

I am also seeing the high network latency reported by others. Here's a sample ping: vmware5 is on 3.0.2, and vmware1 is on 3.0.1:

Pinging vmware5.rbh.local \[10.1.19.135] with 32 bytes of data:

Reply from 10.1.19.135: bytes=32 time=2ms TTL=64

Reply from 10.1.19.135: bytes=32 time=9ms TTL=64

Reply from 10.1.19.135: bytes=32 time=9ms TTL=64

Reply from 10.1.19.135: bytes=32 time=9ms TTL=64

Ping statistics for 10.1.19.135:

Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 2ms, Maximum = 9ms, Average = 7ms

Pinging vmware1.rbh.local \[10.1.19.101] with 32 bytes of data:

Reply from 10.1.19.101: bytes=32 time<1ms TTL=64

Reply from 10.1.19.101: bytes=32 time<1ms TTL=64

Reply from 10.1.19.101: bytes=32 time<1ms TTL=64

Reply from 10.1.19.101: bytes=32 time<1ms TTL=64

Ping statistics for 10.1.19.101:

Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 0ms, Maximum = 0ms, Average = 0ms

\* I should add that these are identical machines plugged into the same switch. I see the same issues on our vmware4 server which is also on 3.0.2.

Reply
0 Kudos
ForgeFlakshack
Enthusiast
Enthusiast

Just got off the phone with VMWare support and they said that this issue is a confirmed bug in 3.0.2 and that I should watch the download page for the hotfix (though no firm date provided).

Reply
0 Kudos
Grandpa_Moses
Contributor
Contributor

Hi,

I can see the same problems. Unfortunately I found this thread after testing nearly everything I could imagine. The last thing I did was installing ttcp on my 3.0.2 box and pcattcp on my w2k3 Server. I wanted to see what transfer rates I could get without writing/reading from local disks or Lun's on our SAN. What is puzzling me is that when I use the w2k3 box as a sender and the 3.0.2 as the receiver, I get 0,3 GBit/s transfer rate and vice versa 0,81 Gbit/s. When I measure the transfer rate while transferring a file (regardless if i use scp or cp) I get rates mentioned in the earlier posts (about 400 KB/s). Has anyone an idea why the rates for the straight network transmit are so much higher? If there is a problem with service console networking this should affect every transmit, shouldn't it?

Thanks in advance!

Reply
0 Kudos
mvennitti
Contributor
Contributor

Has there been any resolution to this problem. I am experiencing the same problem after upgrading to 3.0.2. Backups using vcbMounter used to take approx 1 -2 hours, now they are 8+ hours long.

I haven't noticed any patches for this or am I looking in the wrong area.

Any assistance would be appreciated.

Reply
0 Kudos
Texiwill
Leadership
Leadership

Hello,

THere have not been any patches for this yet. You will need to open a support case with your VMware support agent. The more that get opened the faster it will be fixed.

Best regards,

Edward

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
Reply
0 Kudos
snantel
Enthusiast
Enthusiast

Guys, this problem already been submit to ESX....

We still waiting for a patch. The patch is already up but VMware engeneer still testing it.

We should be able to get this patch for October I think.

Original request:

SR VMWare is: 193838551

Reply
0 Kudos