I recently setup a file server with a 1.5TB volume. So I will share my thoughts on your questions..
OS: Windows Ent. 2003R2x64 or 2008R2 --->Server 2008 offers a more managable interface for file shares/permissions, etc... so if you see yourself using a lot of groups and security permissions server 2008 may be a better choice. Security is enhanced on 2008 plus you might as well 'keep up with the times' and run 2008.
We chose 2008 R2
RAM: 4Gb or more? -
> 4GB for a 2008 R2 box should be more than enough. 2008 is a little bit more of a hog than 2003 for RAM.
We went with 4GB and after reviewing it for awhile the system doesn't even need that much.
Number of CPUs: 1 or more -
> 1 vCPU should be fine for now unless this thing is pounded HARD
We are running 1 vCPU and have had no issues
Volume type RDM or VMDK
We went with a 1.5 TB RDM, but that's because the 1.5TB volume was migrated from a physical server to a VM. The volume lives on a fibre SAN. So we just brought it in as a RDM.
Also if down the line we decide to do a clustered file share the RDM will be needed vs a VMDK.
Hope that helps!