First check the networking for your VM. You should use Bridged networking to make this easier. Next check the IP address of your Vista box, you can run Start > cmd > ipconfig to get the address. Then in OS X you should be to change "localhost" to the numeric IP, e.g. 192.168.0.3 or whatever.

SQL Server (not Express) can be configured to listen on localhost so you should look at the Surface Area Configuration Manager (SQL 2005) to make sure it allows connections from remote hosts.

Oh and of course you'll have to configure the Vista firewall to allow your SQL connections too.