Background

Some VMware products, such as ESX, Workstation, Server, and Fusion, come with a built-in VNC server to access guests. This is convenient if you need to access a guest before it has fully booted, or cannot or do not want to install remote desktop software inside the guest. For security purposes, it may be desirable to limit who can connect to various guests, so you can specify a password. This is hashed and stored in the .vmx file as RemoteDisplay.vnc.key. Some versions of our software (e.g. older versions of Workstation) allow a plaintext password to be stored in RemoteDisplay.vnc.password, but RemoteDisplay.vnc.key is preferred.

This document explains how to compute the hash for RemoteDisplay.vnc.key.

I'm posting this on behalf of rrdharan; please direct questions to him, not me.

High-level overview

DES-encode the password, then base64-encode it.

Note that this uses only the first 8 characters of the plaintext password, anything extra is discarded.

Note that this hash is (non-trivially) reversible; given a .vmx file, a user can figure out the password (though it may take some effort to brute force DES). Therefore, you should guard the .vmx file appropriately and should not use extremely sensitive passwords.

Code

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdbool.h>
#include <errno.h>

#include "d3des.h"
#include "base64.h"

static const char *progname = "vmware-vncpassword";
static const char *keystr = "RemoteDisplay.vnc.key";

void
usage(int ret)
{
   printf("Usage: %s [password]\n", progname);
   exit(ret);
}

int
main(int argc, char *argv[])
{
   char input[DES_HEXKEY_LEN + 1] = { 0 };
   char *output = NULL;
   size_t outlen = 0;
   unsigned int key[32];

   if (argc != 2 || !strlen(argv[1])) {
      printf("invalid arguments specified!\n", progname);
      usage(EINVAL);
   }

   strncat(input, argv[1], DES_HEXKEY_LEN);

   deskey((unsigned char *) input, EN0);
   cpkey(key);

   outlen = Base64_EncodedLength((char *) key, sizeof key);
   output = malloc(outlen);
   if (!output) {
      printf("%s: failed to allocate output buffer!", progname);
      return ENOMEM;
   }

   Base64_Encode((char *) key, sizeof key, output, outlen, NULL); 

   printf("%s = \"%s\"\n", keystr, output);

   return 0;
}

Attached is the source (note d3des is lightly modified to fit our internal build system) as well as precompiled binaries for Linux, OS X, and Windows.