Keep in mind that SRM requires 2 vCenters, and 2 SRM servers. Linked mode makes managing licenses and inventory between two vCenters easier, and if you meet the requirements for linked mode (see documentation) it usually makes sense to do it. Other than meeting the requirements for linked mode there aren't significant pitfalls though I would recommend reading the documentation about it.
Linked mode in vSphere 6 has lower requirements, is easier to setup and better if upgrading is an option.