tamisoft
Contributor
Contributor

IPSEC behavior changed between v6.0 and v6.7

Jump to solution

Hi,

did anyone come across the problem that two identically configured ESXi talking to a linux server over ipv6, one with version 6.0 will work but v6.7 won't. Further debugging the problem lies in the ESP Auth section of the frames. The v6.7 stack generates a 4 byte longer ESP Auth with the same settings (transport mode, 3des-cbc, hmac-sha2-256 on both).

On the linux end I configured tcpdump with the keys, and v6.0's ping can be decoded, also the v6.7 payload seems correct, but the latter never gets a response, I am thinking it gets thrown out. Also when the linux end initiates the ping it generates frames with the same length as v6.0.

esxcli network ip ipsec sa list's output is identical.

so is the linux side's setkey -D

And since tcpdump can decode at least the data, suggest that the ESP should be correct, but even tcpdump is weirding out on unpacking it.

On the working one:

ESP(spi=0x00000200,seq=0x26804), length 100: [bad icmp6 cksum 0x2508 -> 0x4394!] ICMP6, echo request, seq 0

gets a reply:

ESP(spi=0x00000300,seq=0x1f2a0), length 100: [bad icmp6 cksum 0x2408 -> 0x9506!] ICMP6, echo reply, seq 0

But on the broken one:

sometimes: ESP(spi=0x00000201,seq=0x364), length 104

sometimes: ESP(spi=0x00000201,seq=0x365), length 104: ip-proto-77 54

no replies.

Any ideas? I am down to try to decode the captured frames manually and see what hmac the esxi used in reality, but that's kinda tedious.

Thank you

Levi

Tags (3)
0 Kudos
1 Solution

Accepted Solutions
tamisoft
Contributor
Contributor

More precisely ipsec-tools's setkey should not be used any more.

iproute2 has ip xfrm state add ... command that takes an additional auth-trunc parameter where one can define 128 for esxi6.7 and leave the default 96 for esxi v6.0.

View solution in original post

0 Kudos
2 Replies
tamisoft
Contributor
Contributor

Well wireshark was helpful, but the problem lead to this patch proposed to ipsec-tools and the kernel: https://sourceforge.net/p/ipsec-tools/mailman/message/34146970/

https://sourceforge.net/p/ipsec-tools/mailman/message/34146970/

Basically vmware moved from draft-ietf-ipsec-ciph-sha-256-00 to rfc4868. And there is no 128 bit truncation support in the kernel for hmac-sha2-256.

I will try strongswan next or just move to hmac-sha1...

0 Kudos
tamisoft
Contributor
Contributor

More precisely ipsec-tools's setkey should not be used any more.

iproute2 has ip xfrm state add ... command that takes an additional auth-trunc parameter where one can define 128 for esxi6.7 and leave the default 96 for esxi v6.0.

0 Kudos