NSX was initially release with its dedicated virtual switch, the NVDS. As of NSX 3.0, it was possible to simply turn a regular VDS into an NSX-aware VDS. As of NSX 4.0, the NVDS is deprecated and this NSX on VDS model is becoming the only supported virtual switch for ESXi transport nodes (the NVDS remains the only virtual switch for edge transport nodes.)
You will need to convert your NVDS to VDS in order to upgrade to NSX 4.0. This document details the different options available.
