I have done this recently and I agree with the above.

Upgrade Connection Server,Sec Server, Replicas ( if you have them ) and all should be good to go.

Agent 4.5 should be ok for a little bit while backend is being upgraded.