ipa-replica-conncheck on replica side is not correctly working with listening sockets and it is opening and closing them in meanwhile.

Naturally master side cannot now when the port is open and when not so it is causing random failures.

This can be seen with netstat utility in shell.

Steps to reproduce

1. Run ipa-replica-conncheck on replica:

   $ ipa-replica-conncheck -m ipa.demo1.freeipa.org.

2. Let it work until it reaches following line:

   Please run the following command on remote master:
   /usr/sbin/ipa-replica-conncheck --replica <FQDN of replica>

3. Run following command on the replica. It should always list all ports used by FreeIPA. Interestingly, it sometimes prints only subset of the ports. This is what is causing random failues in ipa-replica-conncheck from master's side.

   $ for i in $(seq 1 100)
   do
   date
   netstat -ltup | grep python
   sleep 0.33333333333
   done

I'm attaching the log from my experiments. It shows that number of open ports varies over time.

Probable cause

The listener side is using socket timeout options combined with messy error handling and no logging.

It seems that it is broken since its implementation in ticket:1107 but it manifested now because we finally have automated tests which are executed often enough to detect this.