Compute hashed password for use with RemoteDisplay.vnc.key

Version 5

    Background

    Some VMware products, such as ESX, Workstation, Server, and Fusion, come  with a built-in VNC server to access guests. This is convenient if you  need to access a guest before it has fully booted, or cannot or do not  want to install remote desktop software inside the guest. For security  purposes, it may be desirable to limit who can connect to various  guests, so you can specify a password. This is hashed and stored in the  .vmx file as RemoteDisplay.vnc.key. Some versions of our software (e.g.  older versions of Workstation) allow a plaintext password to be stored  in RemoteDisplay.vnc.password, but RemoteDisplay.vnc.key is preferred.

    This document explains how to compute the hash for RemoteDisplay.vnc.key.

    I'm posting this on behalf of rrdharan; please direct questions to him, not me.

    High-level overview

    DES-encode the password, then base64-encode it.

    Note that this uses only the first 8 characters of the plaintext password, anything extra is discarded.

    Note that this hash is (non-trivially) reversible; given a .vmx file, a  user can figure out the password (though it may take some effort to  brute force DES). Therefore, you should guard the .vmx file  appropriately and should not use extremely sensitive passwords.

    Code

    #include <stdio.h>
    #include <stdlib.h>
    #include <string.h>
    #include <stdbool.h>
    #include <errno.h>

    #include "d3des.h"
    #include "base64.h"

    static const char *progname = "vmware-vncpassword";
    static const char *keystr = "RemoteDisplay.vnc.key";

    void
    usage(int ret)
    {
       printf("Usage: %s [password]\n", progname);
       exit(ret);
    }

    int
    main(int argc, char *argv[])
    {
       char input[DES_HEXKEY_LEN + 1] = { 0 };
       char *output = NULL;
       size_t outlen = 0;
       unsigned int key[32];

       if (argc != 2 || !strlen(argv[1])) {
          printf("invalid arguments specified!\n", progname);
          usage(EINVAL);
       }

       strncat(input, argv[1], DES_HEXKEY_LEN);

       deskey((unsigned char *) input, EN0);
       cpkey(key);

       outlen = Base64_EncodedLength((char *) key, sizeof key);
       output = malloc(outlen);
       if (!output) {
          printf("%s: failed to allocate output buffer!", progname);
          return ENOMEM;
       }

       Base64_Encode((char *) key, sizeof key, output, outlen, NULL);

       printf("%s = \"%s\"\n", keystr, output);

       return 0;
    }



    Attached is the source (note d3des is lightly modified to fit our  internal build system) as well as precompiled binaries for Linux, OS X,  and Windows.