NFS slow from ESXi 3.5

[mvainer]

Hello,

I have a problem with symptoms that are exact replica of the following thread:

http://communities.vmware.com/thread/58117

In my case, i have a solaris ZFS server x4540 attached to ESXi over NFS.

When doing iozone from windows VM, the moment reclen hits 512 the write performance goes from good to unacceptable.

Same iozone test run on a non VM linux machine, there is no slow down at reclen 512.

Here's an example dump from iozone.

random random bkwd record stride

KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread

512 4 135236 170248 202533 210464 190432 131737 167433 149891 182688 132537 169242 192029 197470

512 8 222878 298684 371599 409456 364286 231932 305607 281093 337518 227526 296174 339331 378428

512 16 315769 480526 693164 736034 669367 402001 553861 496270 577965 354904 516552 565831 651288

512 32 436467 748968 1062450 1196297 1171071 627861 807013 861648 1080617 476156 749581 957037 1054503

512 64 539195 1055110 1672196 2080280 1907104 963072 1125753 1359589 1781075 612337 1006992 1287027 1473253

512 128 622529 1269201 1893313 2854715 2607008 1208125 1444229 1755485 2480009 640589 1216950 1478005 1931219

512 256 715071 1458016 2607008 3672799 3517710 1494883 1774179 1987773 2927678 850453 1451090 609081 885375

512 512 7237 7667 3085399 4480995 4081797 7707 1639290 7788 2989766 6674 7162 512794 908640

1024 4 137525 173037 229794 211265 191747 132312 174761 152650 185667 138921 172135 159652 200890

1024 8 234019 304895 375866 391608 364504 239228 317492 291487 343851 239368 298806 260441 376021

1024 16 358795 479206 646692 677783 626252 407635 542949 522144 612849 356769 501910 400858 645894

1024 32 491084 749888 1162897 1306757 1177466 662471 887089 924219 1072397 532459 794592 854617 1048171

1024 64 610095 1018181 1450516 1791522 1843789 933160 1282524 1470888 1819083 665961 1019030 1310026 1534947

1024 128 694477 1172195 2028474 2879382 2723219 1204157 1706449 2011775 2642721 732651 1212121 1503467 1893313

1024 256 747441 1437432 2136045 2556104 2982468 1438561 1819987 2392594 3179058 892924 1424583 710221 855814

1024 512 7292 6616 2743603 3992869 3954103 7275 2072048 7080 3611284 7078 7160 643739 920968

1024 1024 4526 4725 2661912 4302176 4160560 4854 2062720 4833 2970384 4872 4492 487427 495733

2048 4 138152 154367 175183 189654 188053 118651 175367 152644 182270 139453 158987 158454 171411

2048 8 234424 275204 328122 384058 352092 238908 312552 282610 337316 231375 265247 297243 366765

2048 16 365485 435120 553192 629532 646863 419292 568243 539235 607466 358076 408931 398830 460542

2048 32 512758 627377 759364 1161976 1129918 662591 626734 958789 1041766 515825 619112 706389 996453

2048 64 608222 781713 1113612 1832268 1815929 952684 1353065 1548238 1594368 639583 754752 667903 837913

2048 128 685323 880272 909993 1122478 1482189 1223449 1646285 2166344 2770562 429788 808704 754907 897515

2048 256 744859 956413 1545626 2882779 2986113 1444229 1001079 2825012 3595344 821206 989059 652564 877953

2048 512 6485 6272 263332 2137914 2796988 7043 2252199 7489 3924469 6380 7191 618642 901932

2048 1024 4623 4852 1552171 3240896 3403392 4466 1610834 3878 3225212 5117 4685 536944 858420

2048 2048 4091 4322 1428470 2906783 2766380 3550 1779346 4060 2098742 3621 3199 365266 705030

When connected to a different Solaris box ( a much slower box), this effect happens, but not to the same degree:

random random bkwd record stride

KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread

256 4 134030 164429 165170 212170 191747 128216 160652 144605 180528 140546 164695 167341 195470

256 8 206807 270873 394304 402797 368609 222905 307504 269360 362916 227329 309060 334439 374178

256 16 310106 498023 645326 596202 666931 397899 459330 480022 590060 352447 495331 618746 644872

256 32 400508 714792 1096128 1278052 1161423 636805 718153 814545 1064301 468249 777897 733090 1023869

256 64 491084 1040140 1775898 1962234 1722487 962566 1004784 1207330 1709633 588166 1076807 1208922 1422924

256 128 529078 1294298 2456738 2768469 2588597 1210519 1267446 1449942 2192257 573803 1269201 1445368 1866320

256 256 625931 1509659 2663847 3725054 3238033 1473253 1339712 1447651 3024302 803123 1610480 456356 875227

512 4 147860 169555 204294 211387 191647 129658 155605 142924 183144 146489 218389 193918 202176

512 8 238047 295839 325355 234094 351501 192756 277601 277517 301336 198111 278529 347502 378037

512 16 346516 460715 722399 740495 670346 399373 559440 501842 646464 376639 529843 582002 646236

512 32 431229 769083 1125062 1204948 1170323 655716 848877 866128 1086382 519627 783886 981117 1055718

512 64 554363 941308 1716036 2080280 1901169 973301 1176333 1404388 1733895 646236 916363 1286124 1536233

512 128 618120 1251862 2414660 2899884 2637017 1278944 1624758 1779346 2610722 731919 1273611 1545301 1931219

512 256 697119 1416326 2151088 3198476 3069894 1246753 1336781 1453391 2268226 719563 1118198 591583 884520

512 512 28914 34174 3266893 4470065 4081797 34023 1913076 35007 3484271 33370 30191 489902 919120

1024 4 148760 175792 203896 212934 187952 132835 177512 153154 183888 152024 163753 198993 196275

1024 8 240310 310974 383817 406595 364612 240815 327565 224475 265805 249724 299563 353842 377259

1024 16 362342 515390 663190 648752 670469 418192 543352 532924 618224 377920 506838 591106 651057

1024 32 497482 772324 1197469 1291562 1191630 667052 947635 906169 1148325 547082 821482 905945 1035146

1024 64 631104 1082212 1701696 1925133 1932237 983750 1407624 1500390 1827245 688865 1087671 1352067 1517158

1024 128 725832 1334833 2093348 2913715 2778964 1266570 1771606 2011775 2679425 814726 1336781 842633 1963285

1024 256 722399 1476814 2830466 3621990 3487587 1544649 2236396 2433900 3572567 898836 1515276 735444 891187

1024 512 33673 33956 3132866 4213165 4146441 33973 2264023 34505 3911904 33657 34136 631104 909767

1024 1024 23066 22956 3021808 4332688 4272091 23151 2088577 22916 3154435 22687 22488 469628 897515

2048 4 146712 163709 183135 201992 183584 130902 173467 136424 181368 146270 161848 156619 168499

2048 8 233773 282446 333587 399156 357221 239548 330087 295065 345471 257450 282109 312272 370397

2048 16 366637 428958 509055 724756 659728 410764 574747 493830 561540 392152 449363 502530 640589

2048 32 510224 628561 848582 1238329 1158487 678724 956413 914534 1060452 538048 639806 723040 1008378

2048 64 519480 816905 1192599 1918081 1474141 981248 1479795 1551186 1538167 637248 823790 914648 1434339

2048 128 690487 936019 1435181 2490118 2577675 1247389 1902649 2152983 2738478 755140 904603 779220 954421

2048 256 773873 881436 1615805 3000781 2999553 1470003 2250816 2787417 3639974 828632 1032376 657716 884947

2048 512 33035 32967 1612961 3344392 3526170 34017 2448533 33677 3632759 33195 32908 472809 865003

2048 1024 22178 22720 1621523 3263984 3361260 22961 2319907 22978 3539791 13970 13584 533933 839064

2048 2048 18905 10878 1384757 2146678 2687283 15414 1907600 19277 2130458 12649 19383 397382 712776

The other person in the above mentioned thread said he somehow solved it, but unfortunately no solution was posted.

Thanks in advance...

[mvainer]

Another interesting data point - linux VMs don't seem to suffer from the same slowdown, IOzone on linux doesn't report slow down when it hits 512.