marauli's Posts

5 hours into this rabbit hole (figuring out how to download, deploy and configure this thing), it looks like it requires a peer vCenter instance? ... which something we don't have: All sites and ESXis are managed with a single central vCenter instance. I.e. vSphere Replication is not for us? P.S. The "vSphere Replication" article you linked to lists this as one of the use cases: Data protection locally, within a single site Does this use case still require a "peer" vCenter instance? If not: Does it do what I need it to do? (Replicate VMs between two hosts in a remote site.) is there a decent "explain this to me like I am a 10-year-old" article telling me how to configure it, what the prerequisites are? Is it an overkill for this use case? (Deploying and configuring this appliance to each of the 15+ remote sites where there are limited support resources, i.e. can't afford spending hours on configuring and maintaining it.)

Looks like with respect to scripting, there are fairly similar questions already covered here: Clone a VM with a script ...Where a PowerCLI script could check for an existing clone, delete it if needed, start a new clone. "Scheduled Tasks" was a promising option until it became clear a new clone would fail if there's an existing clone with the same name, and that this existing clone couldn't be renamed or deleted via "Scheduled Tasks". The two remaining questions are: What are the best practices in hosting such a script? (Where to run from - some random Windows machine?, how to securely store needed credentials, how to maintain it - e.g. track changes.) Does vSphere DR/HA offer a good alternative? (I.e. can this task be automated there?)

Resolved, all four ESXis can now see the iSCSI target in question. Misconfigured iSCSI Port Binding configuration was likely one of the culprits, as @a_p_ suggested. The other one was likely some stale configuration where rebooting, rescanning, detaching stale devices, rebooting, rescanning several times - seems to have helped. Thank you all very very much for the help! (Been at it for quite some time, feels good to get some progress.)

did you ever perform a Rescan operation from your ESXi Hosts (or Cluster wide) to verify if the target (and it's LUNs) are still visible after the Rescan? At least a few dozen times - I rescan any time I try any change with a network or storage configuration. I have a feeling something on the iSCSI target (r730b) might be misconfigured given it disappeared even from ESXis it was present before, while my test iSCSI target (on the same Ubuntu version) seems to be working OK, visible on multiple ESXis. Will have to dive into Ubuntu networking and iSCSI (tgt) configuration.

yet it doesn't appear to affect anything. I was wrong on that one. Put one of the affected hosts in maintenance, removed the binding, re-scanned the adapter - and while the ESXi still doesn't see the iSCSI target in question, it does now see a test one I just created yesterday on an Ubuntu VM. (In green are the paths to the new test iSCSI target, in red - to iSCSI target the hosts can no longer see.) Another new development is that the 4th ESXI that used to see the iSCSI target in question - can no longer see it. (The only change was the server running that iSCSI storage - Ubuntu Linux on Dell r730 - was updated via 'apt-get upgrade' and rebooted today.) With all this, the culprit is possibly with the Ubuntu Linux iSCSI service configuration (tgt)?

Agreed, that was my bad - enabled it before reading up on it - but after this issue occurred - yet it doesn't appear to affect anything. First three hosts have the same configuration, the fourth one doesn't (as it's not connected to the direct-attached ME4024 iSCSI target, unlike the first three). As soon as I can put one of them in maintenance, I'll remove the binding, see if that does anything - but am not expecting it to. (Putting it in maintenance is tricky as our cluster is running low on memory - so I am trying to tread gently.)

Are you able to ping the iSCSI target(s) ip addresses using vmkping -I vmk0 -d -s 1450 <ip address iSCSI target)? [root@***-ESXi-01:~] vmkping -I vmk0 -d -s 1450 <IP> PING <IP> (<IP>): 1450 data bytes 1458 bytes from <IP>: icmp_seq=0 ttl=64 time=0.224 ms 1458 bytes from <IP>: icmp_seq=1 ttl=64 time=0.277 ms 1458 bytes from <IP>: icmp_seq=2 ttl=64 time=0.278 ms --- <IP> ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.224/0.260/0.278 ms How do the NIC stats look like using esxcli network nic stats get -n vmnicX? NIC statistics for vmnic0 Packets received: 1696266074 Packets sent: 2119143975 Bytes received: 645191082460 Bytes sent: 2243595138013 Receive packets dropped: 0 Transmit packets dropped: 0 Multicast packets received: 108342319 Broadcast packets received: 30483370 Multicast packets sent: 1932358 Broadcast packets sent: 1573782 Total receive errors: 0 Receive length errors: 0 Receive over errors: 0 Receive CRC errors: 0 Receive frame errors: 0 Receive FIFO errors: 0 Receive missed errors: 0 Total transmit errors: 0 Transmit aborted errors: 0 Transmit carrier errors: 0 Transmit FIFO errors: 0 Transmit heartbeat errors: 0 Transmit window errors: 0 NIC statistics for vmnic1 Packets received: 471521582 Packets sent: 212317143 Bytes received: 314177704382 Bytes sent: 80328464786 Receive packets dropped: 0 Transmit packets dropped: 0 Multicast packets received: 110083599 Broadcast packets received: 32038626 Multicast packets sent: 257459 Broadcast packets sent: 19461 (all "error" values - 0) NIC statistics for vmnic2 Packets received: 140071964 Packets sent: 0 Bytes received: 13097333291 Bytes sent: 0 Receive packets dropped: 0 Transmit packets dropped: 0 Multicast packets received: 107577113 Broadcast packets received: 32255292 Multicast packets sent: 0 Broadcast packets sent: 0 (all "error" values - 0) NIC statistics for vmnic3 Packets received: 140073820 Packets sent: 0 Bytes received: 13097515909 Bytes sent: 0 Receive packets dropped: 0 Transmit packets dropped: 0 Multicast packets received: 107578538 Broadcast packets received: 32255717 Multicast packets sent: 0 Broadcast packets sent: 0 (all "error" values - 0) NIC statistics for vmnic4 Packets received: 12515272569 Packets sent: 2463924551 Bytes received: 17668531459730 Bytes sent: 2695041739606 Receive packets dropped: 0 Transmit packets dropped: 0 Multicast packets received: 0 Broadcast packets received: 2 Multicast packets sent: 0 Broadcast packets sent: 1699 (all "error" values - 0) NIC statistics for vmnic5 Packets received: 140778 Packets sent: 266174 Bytes received: 16919128 Bytes sent: 24241914 Receive packets dropped: 0 Transmit packets dropped: 0 Multicast packets received: 0 Broadcast packets received: 2 Multicast packets sent: 0 Broadcast packets sent: 1695 (all "error" values - 0)  

Sorry about that - it's all very confusing to me too. vmk1 and 2 should not handle connections to the iSCSI target in question (located on a switched network). Only vmk0 can handle those as that's the only adapter with a path to it. (vmk1 and 2 are adapters for direct-attached connections only, i.e. for targets not on a switched network) If that still doesn't clear it up - let me know.

Could this be relevant? All iSCSI connection failures occur on vmk1 and vmk2 (which have no network connection to the target), and there isn't anything for vmk1 - which is the NIC that the has the network path to the iSCSI target in question.   2023-08-07T22:49:25.324Z: [iscsiCorrelator] 569241542us: [esx.problem.storage.iscsi.target.connect.error] Login to iSCSI target iqn.1988-11.com.dell:01.array.bc305bf24e32 on vmhba64 @ vmk2 failed. The iSCSI initiator could not establish a network connection to the target. 2023-08-07T22:49:25.325Z: [iscsiCorrelator] 569233690us: [vob.iscsi.target.connect.error] vmhba64 @ vmk2 failed to login to iqn.1988-11.com.dell:01.array.bc305bf24e32 because of a network connection failure. 2023-08-07T22:49:25.326Z: [iscsiCorrelator] 569242709us: [esx.problem.storage.iscsi.target.connect.error] Login to iSCSI target iqn.1988-11.com.dell:01.array.bc305bf24e32 on vmhba64 @ vmk2 failed. The iSCSI initiator could not establish a network connection to the target. 2023-08-07T22:49:25.326Z: [iscsiCorrelator] 569234520us: [vob.iscsi.target.connect.error] vmhba64 @ vmk1 failed to login to iqn.1988-11.com.dell:01.array.bc305bf24e32 because of a network connection failure. 2023-08-07T22:49:25.326Z: [iscsiCorrelator] 569243520us: [esx.problem.storage.iscsi.target.connect.error] Login to iSCSI target iqn.1988-11.com.dell:01.array.bc305bf24e32 on vmhba64 @ vmk1 failed. The iSCSI initiator could not establish a network connection to the target. 2023-08-07T22:49:25.327Z: [iscsiCorrelator] 569235334us: [vob.iscsi.target.connect.error] vmhba64 @ vmk1 failed to login to iqn.1988-11.com.dell:01.array.bc305bf24e32 because of a network connection failure. 2023-08-07T22:49:25.327Z: [iscsiCorrelator] 569244253us: [esx.problem.storage.iscsi.target.connect.error] Login to iSCSI target iqn.1988-11.com.dell:01.array.bc305bf24e32 on vmhba64 @ vmk1 failed. The iSCSI initiator could not establish a network connection to the target.   vmk1 and vmk2 are kernel NICs dedicated to direct-attached iSCSI connections. They do not have a path to the iSCSI target in question that is on a switched network. I guess I am puzzled why doesn't the ESXi attempt to connect to the target using vmk0, and what I can do to force it to.  

Sorry it's taking me so long to respond! (Was fighting other fires.) i. Could you confirm that it was only the switches firmware that was updated, and that none of, or any other parts of the infrastructure or configuration was changed (i.e. that switches swapped out, switches config, routing, cabling, and any config on the ESXi or iSCSI target were not changed in any way, what so ever) ? To my knowledge, just the firmware - although can't be 100% sure. The network infra is handled by someone else, and I have limited access to it. On VMware side - I do have full access, and do not see any changes made to ESXis or to the iSCSI system. (One possible relevant bit of info: the affected ESXis (3 out of 4) pre-date me joining the team, i.e. configured by someone else originally. The last one, that is unaffected by the change - added to the cluster and configured by yours truly. I pored over network config pages on all ESXis trying to zero in on what could be different between the affected and unaffected ESXis - can't find anything. Did the same in Meraki - ditto.) ii. How many switches are involved, if more than one, which ESXi and iSCSI target is connected to which switch ? Around 4-5: there are four 10Gb ports on each switch, each connected system uses 2 of them, and between 4 ESXis and 1 iSCSI target, ten 10Gb ports are used across a number of switches. iii. Is there only one iSCSI target for all of the four ESXi servers ? Two for the 1st three ESXis, one for the last one. The 2nd one is a Dell/EMC ME4024 flash array direct-attached (via direct 10Gb links, no switches involved) to the 1st three ESXis. iv. Are all of the switches at the same firmware version ? v. Are all the switches the same make/model/version ? vi. Have you reviewed the switches firmware version update notes to determine what was changed ? Yes, yes; see no notes. vii. Are you using VLANs ? We do use VLANs, and the ports on the switches are configured the same away across all ESXis and the iSCSI target - at least while we're troubleshooting the issue:   Type Trunk Native VLAN <masked> Allowed VLANs all Access policy Open     viii. You mentioned that you spun up an 'older standalone ESXi 6.7' and a 'Windows Desktop' both of which could "also see the device".  Would I be correct to assume that 'see the device' to mean that they could see the iSCSI target in question, and mount the storage ? Correct. ix. How many network connections do the each of the ESXi servers have ? x. How many network connections does the iSCSI target have ? The first three (affected) ESXis: six total: two 10Gb NICs for general traffic, vMotion, switched iSCSI two 1Gb legacy NICs: still connected but no longer active (no attached port groups, vSwitches or VMkernel adapters are connected to them) two 10Gb NICs for direct-attached iSCSI (ME4024 mentioned above) The 4th (unaffected): two 10Gb NICs for general traffic, vMotion, switched iSCSI (it's not connected to ME4024) iSCSI target: two 10Gb NICs; only one active now; the 2nd one is connected to a switch port that was disabled by our network admin as for some reason there were IP conflict alarms from Meraki on the two ports for the target despite no apparent conflicts (the IPs are different). - I would suggest moving one of the ESXi's that cannot connect to iSCSI target to one of the switches network ports that you know is working (either the, ESXi that works after the switches firmware upgrade, ESXi 6.7 or Win Desktop (assuming 'viii' to be correct)). - As an experiment, you could exclude the switches from the equation altogether and make an appropriate direct connection . . or alternatively via another type/make of switch. Thank you! I'll check with the network admin on both options.

My bad: the article suggesting to run the script only applies to ESXi 7, and this one is 6.7. The question then is, how do I bypass those "vmware.esximage.Errors.MetadataFormatError: File is not a zip file" errors and patch the server?

Thank you - some of the steps in those articles may be exactly what I was looking for - I'll try them. Check network connectivity: vmkping I < SW iSCSI vmkernel > Target_IP Check SW iSCSI port: nc z < Target_IP > 3260

Thanks @trobertson! The switches in question are MS225-48FP (the ESXis and iSCSI devices are connecting to the 10Gb ports on them) and I don't believe it's an option to try a different switch. Perhaps what I am looking for is something like `telnet <iSCSI target's IP> 3260` from the affected ESXi host. If this gets dropped or times out - whereas the behavior on an unaffected ESXi is different - that would confirm the issue is likely with the switches (or less likely, in the ESXi network configuration, or the iSCSI target's firewall.) I.e. basic troubleshooting steps from someone more knowledgeable about ESXis.