Description of problem:

During ipa-replica-install, sometimes httpd restart fails:

Configuring the web interface: Estimated time 1 minute
  [1/12]: disabling mod_ssl in httpd
  [2/12]: setting mod_nss port to 443
  [3/12]: setting mod_nss password file
  [4/12]: enabling mod_nss renegotiate
  [5/12]: adding URL rewriting rules
  [6/12]: configuring httpd
  [7/12]: setting up ssl
  [8/12]: publish CA cert
  [9/12]: creating a keytab for httpd
  [10/12]: configuring SELinux for httpd
  [11/12]: restarting httpd
  [12/12]: configuring httpd to start on boot
done configuring httpd.
creation of replica failed: Command '/sbin/service httpd restart ' returned
non-zero exit status 1

I've seen this more than once now but, not yet able to reproduce.

Version-Release number of selected component (if applicable):
ipa-server-2.1.3-9.el6.i686
httpd-2.2.15-15.el6.i686

How reproducible:
Unknown

Steps to Reproduce:
1. <Set up RHEL6.2 IPA Master server>
2. On Master: ipa-replica-prepare -p $ADMINPW --ip-address=$SLAVE_IP
$SLAVE_S.$DOMAIN
3. On Replica: sftp $MASTER:/var/lib/ipa/replica-info-$SLAVEFQDN.gpg /dev/shm
4. On Replica: ipa-replica-install -U --setup-dns --forwarder=$DNSFORWARD -w
$ADMINPW -p $ADMINPW /dev/shm/replica-info-$SLAVEFQDN.gpg

Actual results:
Error from description above.

Expected results:
No error.

Additional info:

From ipareplica-install.log:

2012-08-01 15:19:40,811 DEBUG stdout=pk12util: PKCS12 IMPORT SUCCESSFUL

2012-08-01 15:19:40,812 DEBUG stderr=
2012-08-01 15:19:52,763 DEBUG args=/sbin/service httpd restart
2012-08-01 15:19:52,770 DEBUG stdout=Stopping httpd: [  OK  ]
Starting httpd: [FAILED]

2012-08-01 15:19:52,770 DEBUG stderr=(98)Address already in use: make_sock:
could not bind to address [::]:80
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs

2012-08-01 15:19:52,799 DEBUG Command '/sbin/service httpd restart ' returned
non-zero exit status 1
  File "/usr/sbin/ipa-replica-install", line 482, in <module>
    main()

  File "/usr/sbin/ipa-replica-install", line 447, in main
    ipaservices.knownservices.httpd.restart()

  File "/usr/lib/python2.6/site-packages/ipapython/platform/redhat.py", line
47, in restart
    ipautil.run(["/sbin/service", self.service_name, "restart", instance_name],
capture_output=capture_output)

  File "/usr/lib/python2.6/site-packages/ipapython/ipautil.py", line 273, in
run
    raise CalledProcessError(p.returncode, args)


From http/error_log:

[root@ipaqavmb httpd]# cat error_log
[Wed Aug 01 15:19:38 2012] [notice] SELinux policy enabled; httpd running as
context unconfined_u:system_r:httpd_t:s0
[Wed Aug 01 15:19:38 2012] [notice] suEXEC mechanism enabled (wrapper:
/usr/sbin/suexec)
[Wed Aug 01 15:19:40 2012] [notice] Digest: generating secret for digest
authentication ...
[Wed Aug 01 15:19:40 2012] [notice] Digest: done
[Wed Aug 01 15:19:40 2012] [warn] mod_wsgi: Compiled for Python/2.6.2.
[Wed Aug 01 15:19:40 2012] [warn] mod_wsgi: Runtime using Python/2.6.6.
[Wed Aug 01 15:19:41 2012] [notice] Apache/2.2.15 (Unix) DAV/2
mod_auth_kerb/5.4 mod_nss/2.2.15 NSS/3.12.9.0 mod_wsgi/3.2 Python/2.6.6
configured -- resuming normal operations
[Wed Aug 01 15:19:42 2012] [error] Exception KeyError: KeyError(-1216170160,)
in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
[Wed Aug 01 15:19:43 2012] [error] Exception KeyError: KeyError(-1216170160,)
in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
[Wed Aug 01 15:19:45 2012] [error] ipa: INFO: *** PROCESS START ***
[Wed Aug 01 15:19:45 2012] [error] ipa: INFO: *** PROCESS START ***



PS showed httpd processes still running like they were orphaned when service
httpd restart ran the stop:
[root@ipaqavmb ~]# ps -ef|grep http
root     25610     1  0 16:35 ?        00:00:00 /usr/sbin/nss_pcache 983043 off
/etc/httpd/alias
apache   25637     1  8 16:35 ?        00:00:02 /usr/sbin/httpd
apache   25639     1  7 16:35 ?        00:00:02 /usr/sbin/httpd
apache   25640     1  1 16:35 ?        00:00:00 /usr/sbin/httpd
apache   25642     1  1 16:35 ?        00:00:00 /usr/sbin/httpd
apache   25644     1  1 16:35 ?        00:00:00 /usr/sbin/httpd
apache   25645     1  1 16:35 ?        00:00:00 /usr/sbin/httpd
apache   25646     1  1 16:35 ?        00:00:00 /usr/sbin/httpd
apache   25647     1  1 16:35 ?        00:00:00 /usr/sbin/httpd
root     25936 22439  1 16:35 pts/6    00:00:00 grep http


So, it looks like a problem with the httpd stop from the init script?