We are exploring the possibility of using the Spare VxRack nodes (R630) and connect to the third party managed Network switches
There is no such thing as VxRack nodes in the context of Cloud Foundation. You can buy VxRack SDDC as an integrated system which includes servers and ToR/management switches and VMware software. You cannot buy "VxRack nodes" separately. If you happen to have the servers already, then you will need to ask Dell EMC if they would trade them in in exchange for a VxRack SDDC system. You cannot get it in pieces, only all at once. (There are ways to trade-in VMware licenses, though).
Can we manage exsi Nodes in the new network given the network switches are only allow pass through traffic instead of managed by SDDC mgr?
Cloud Foundation and VxRack SDDC are natively integrated systems, you might call them rack-scale appliances. The included SDDC Manager is how the ESXi hosts in those systems are managed. It looks like your specific use case is better suited for a manual deployment of servers.
I do not see what value you would get out of VxRack SDDC or Cloud Foundation in your specific situation. Maybe this is just a misunderstanding on my side? Can you clarify?