After upgrading vCenter from 6.7U3 to v7.0U1, a utility that we use to upload boot images to a datastore, stopped working.
The utility first tries to send the request without an “Authorization”-header as it is supposed to. A client should only authenticate after being asked to do so by the server. The client adds an “Expect: 100-continue” header and vCenter 7 does not seem to handle this correctly.
We have created a case with VMware, but they "refuse to take it", in the way that they say we need SDK support for this one. I think we don't but who am I to tell them.. We can show the problem with a simple curl command, so it's not something that really needs an SDK to trigger. So I'm posting here (wasn't sure which forum category to use, so mods, feel free to move it).
Here is the command used for testing (note that the payload must be 1MB or more for curl to use the “Expect: 100-continue” header):
$ curl -v -i -k -X PUT --http1.1 --tlsv1.2 \ -H 'Content-type: application/octet-stream' \ --data-binary @/path/to/payload \ 'https://vc.example.com/folder/RHEL/install-images/image.iso?dcPath=Datacenter_Name&dsName=Datastore_Name'
The test command is intentionally without credentials to imitate the behavior of our utility.
Running the command against an old 6.7 vCenter server gives the expected behavior (and upload works):
> Host: vc67.example.com > User-Agent: curl/7.61.1 > Accept: */* > Content-type: application/octet-stream > Content-Length: 247695360 > Expect: 100-continue > < HTTP/1.1 401 Unauthorized HTTP/1.1 401 Unauthorized
Running the same command against a version 7 vCenter we get this:
> Host: vc70.example.com > User-Agent: curl/7.61.1 > Accept: */* > Content-type: application/octet-stream > Content-Length: 247695360 > Expect: 100-continue > < HTTP/1.1 100 Continue HTTP/1.1 100 Continue < HTTP/1.1 401 Unauthorized HTTP/1.1 401 Unauthorized
This is an unexpected behavior as the client starts to send the payload when it sees the “100 Continue” response. It then gets the “401 Unauthorized” response after having sent the payload and is unable to retry the request because it reads the payload as a stream.
Note: curl is only used to demonstrate the problem. The utility is a Java-application, using Apache Httpcomponents (https://hc.apache.org/)
Due to this problem, our deployment system for VMs is down - the install iso isn't transferred to the datastore and mounted to VM.