Social interaction increases significantly the performance of a wide range of cooperative systems, from flocking bids and schooling fish to social networks. However, it has been pointed out that some natural swarms seem to limit the number of social connections, which suggests there are detrimental consequences to excessive social activity. Using a canonical model of collective motion, we find that the responsiveness of a swarm to local perturbations is reduced when the social interaction exceeds a certain threshold. This effect is mirrored in collective decision-making models of distributed consensus operating over a range of networks. While increasing the amount of interaction always increases the capacity of these systems to adapt to slow changes, an excess of social behavior can hinder the swiftness of their response to fast disruptions.