I have not seen that case before. To Test the load balancer, I did shut down the NIC of my primary Broker to verify users were redirected to Broker #2 and it worked.
Can you launch the users desktop by connecting to the replica Brokers IP address directly? Or from an Admin side, have you ever logged into the View Administrator on the replica Broker? Sounds like a configuration issue of that the replica Broker is not fully functional.