ipa-replica-conncheck on replica side is not correctly working with listening sockets and it is opening and closing them in meanwhile.
Naturally master side cannot now when the port is open and when not so it is causing random failures.
This can be seen with netstat utility in shell.
Run ipa-replica-conncheck on replica:
$ ipa-replica-conncheck -m ipa.demo1.freeipa.org.
Let it work until it reaches following line:
Please run the following command on remote master:
/usr/sbin/ipa-replica-conncheck --replica <fqdn of="" replica="">
Run following command on the replica. It should always list all ports used by FreeIPA. Interestingly, it sometimes prints only subset of the ports. This is what is causing random failues in ipa-replica-conncheck from master's side.
$ for i in $(seq 1 100)
netstat -ltup | grep python
I'm attaching the log from my experiments. It shows that number of open ports varies over time.
The listener side is using socket timeout options combined with messy error handling and no logging.
It seems that it is broken since its implementation in ticket:1107 but it manifested now because we finally have automated tests which are executed often enough to detect this.
Petr, do you have a suggestion what would be a better approach to check open ports from master side?
Well, I would find a random article about programming server applications and implement that.
Let me Google this:
seems like reasonable start.
Linked to Bugzilla bug: https://bugzilla.redhat.com/show_bug.cgi?id=1379029 (Red Hat Enterprise Linux 7)
#6456 was closed as duplicate of this bug.
I didn't implement a proper solution and af0ba66 introduced another race condition. Sorry about that. Fix is ready in PR https://github.com/freeipa/freeipa/pull/309
Metadata Update from @pspacek:
- Issue assigned to tkrizek
- Issue set to the milestone: FreeIPA 4.5
to comment on this ticket.