Just had to troubleshoot a not-working Agent backup job. Up to now, everything worked fine. Since last night, job fails. Error shown:
[Task failed. Error: Failed to connect to servername:11731]
Even in logs this error is shown:
The problem seems to be communication with port 11731. This port is used for agent-deployment. No changes were made to the firewalls in this particular environment. So this could not be the cause of the problem.
After a short phone call it turned out that the account used to communicate with the agent had been added to the Protected Users Security Group. But this group is not intended for adding service accounts. After removing the user, backup worked fine again.
I hope this post saves some time in troubleshooting 
so you are well prepared now! 