Does ESXi/VSA really need OpenSSL 0.9.8 and/or the string_mask nombstr option?

[azimuthjm]

The guides I've read for creating certificates for ESXi 5.5 (I have ESXi 5.5u2), some say you must use OpenSSL 0.9.8, but they don't state why. I'm not sure if they mean it must be 0.9.8 and not a higher revision like 1.0.1, but one guide did mention 0.9.8 or higher (but with no idea what the reason is, it could be someone just making an assumption).

Some guides list the OpenSSL config to have "string_mask = nombstr", this means the X.509 certificate strings will be of type PrintableString rather than UTF8String. This caused my CA to reject the CSR due to the organization not matching, I bypassed the policy, but the produced X.509 certificate's Subject is of string type PrintableString. When I browse to https://myesxiserver.corp Chrome whinges the connection is not secure, there is https with a line through it in red and the padlock has an red x on it. It says the hostname doesn't match, lexicographically they do match. If I go to https://myesxiserver (which is a Subject Alternative Name (SAN)) in the certificate I get a green https:// and no warnings/errors (this is because there is only one encoding in the standard for the SAN extension).

All the other certificates I have produced for other infrastructure do not have the string_mask = nombstr, and hence are the Subjects use UTF8Strings encoding and Chrome does not complain about certificate's not matching domain name issues.

So my question is - is it really necessary to have the sring_mask nombstr option in ESXi/VSA - will things break if I don't use that option (e.g. I have a UTF8String subject rather than a PrintableString Subject).

Why is this restriction necessary?

Thanks!

[azimuthjm]

After further research it appears that because the certificate is using Subject Alternative Names (SANs), all names must be listed, not just the alternative names, so the PrintableString encoding was a redherring.

I am going to generate new certificates to test this.

I'd still like clarification around the OpenSSL 0.9.8 requirement and the string_mask = nombstr requirement.

Can OpenSSL > 0.9.8 be used? Will it cause incompatibilities due to differences in the generated X.509 certificate?

Can nombstr be left off, so that UTF8Strings are used, (I do not have any non-ASCII characters in my hostnames or subjects etc, so it should be effectively the same, but it will mean my CA's policy check will work and not need to be bypassed which will help catch typos in CSRs etc.

Thanks.

[azimuthjm]

After replacing the certificates with ones that had SANs for all names (including the one used in the subject) it worked correctly.

I'd still like to know the following: -

  1. Is OpenSSL version exactly equal to 0.9.8 a requirement or can greater than 0.9.8, such as 1.0.x be used with no problems?

     b. If there are problems with > 0.9.8, what are these problems, what is the technical reason for the 0.9.8 requirement?

  2. Can the string_mask = nombstr be left out (so that UTF8 strings are used) without causing problems?

    b. If not, what are these problems, what is the technical reason that nombstr setting must be used?

Thanks