https://bugzilla.redhat.com/show_bug.cgi?id=739089
Description of problem: Version-Release number of selected component (if applicable): ipa-server-2.1.1-1.el6.x86_64 How reproducible: Always Steps to Reproduce: 1. [root@ratchet ~]# ipa-server-install --setup-dns 2. [root@ratchet ~]# ipactl status Directory Service: RUNNING KDC Service: RUNNING KPASSWD Service: RUNNING DNS Service: RUNNING HTTP Service: RUNNING CA Service: RUNNING 3. [root@ratchet ~]# nslookup -type=AAAA ratchet.lab.eng.pnq.redhat.com Server: 2620:52:0:41c9:5054:ff:fea6:ec8 Address: 2620:52:0:41c9:5054:ff:fea6:ec8#53 ratchet.lab.eng.pnq.redhat.com has AAAA address 2620:52:0:41c9:5054:ff:fea6:ec8 4. [root@ratchet ~]# nslookup 2620:52:0:41c9:5054:ff:fea6:ec8 Server: 2620:52:0:41c9:5054:ff:fea6:ec8 Address: 2620:52:0:41c9:5054:ff:fea6:ec8#53 8.c.e.0.6.a.e.f.f.f.0.0.4.5.0.5.9.c.1.4.0.0.0.0.2.5.0.0.0.2.6.2.ip6.arpa name = ratchet.lab.eng.pnq.redhat.com. 5. [root@ratchet ~]# ipa user-add shanks Actual results: [root@ratchet ~]# ipa user-add shanks ipa: ERROR: cannot connect to 'any of the configured servers': https://ratchet.lab.eng.pnq.redhat.com/ipa/xml, https://ratchet.lab.eng.pnq.redhat.com/ipa/xml Expected results: user should be added successfuly. Additional info: [root@ratchet ~]# ipa -d user-add shanks ipa: DEBUG: importing all plugin modules in '/usr/lib/python2.6/site-packages/ipalib/plugins'... ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/aci.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/automember.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/automount.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/baseldap.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/batch.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/cert.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/config.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/delegation.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/dns.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/entitle.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/group.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/hbacrule.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/hbacsvc.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/hbacsvcgroup.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/hbactest.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/host.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/hostgroup.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/internal.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/kerberos.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/krbtpolicy.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/migration.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/misc.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/netgroup.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/passwd.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/permission.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/ping.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/privilege.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/pwpolicy.py' ipa: DEBUG: args=klist -V ipa: DEBUG: stdout=Kerberos 5 version 1.9 ipa: DEBUG: stderr= ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/role.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/selfservice.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/service.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/sudocmd.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/sudocmdgroup.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/sudorule.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/user.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/virtual.py' ipa: DEBUG: importing plugin module '/usr/lib/python2.6/site-packages/ipalib/plugins/xmlclient.py' ipa: INFO: trying https://ratchet.lab.eng.pnq.redhat.com/ipa/xml ipa: DEBUG: NSSConnection init ratchet.lab.eng.pnq.redhat.com ipa: DEBUG: connect_socket_family: host=ratchet.lab.eng.pnq.redhat.com port=443 family=PR_AF_INET ipa: DEBUG: connect_socket_family: host=ratchet.lab.eng.pnq.redhat.com port=443 family=PR_AF_INET6 ipa: DEBUG: connecting: [2620:52:0:41c9:5054:ff:fea6:ec8]:443 ipa: DEBUG: auth_certificate_callback: check_sig=True is_server=False Data: Version: 3 (0x2) Serial Number: 10 (0xa) Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Issuer: CN=Certificate Authority,O=LAB.ENG.PNQ.REDHAT.COM Validity: Not Before: Fri Sep 16 20:47:35 2011 UTC Not After : Mon Sep 16 20:47:35 2013 UTC Subject: CN=ratchet.lab.eng.pnq.redhat.com,O=LAB.ENG.PNQ.REDHAT.COM Subject Public Key Info: Public Key Algorithm: PKCS #1 RSA Encryption RSA Public Key: Modulus: 95:e4:ed:80:59:42:b6:24:a3:15:ce:d6:31:c4:d4:02: 69:ab:d5:0b:ba:5d:38:f9:cd:d0:b1:5e:62:f0:76:5b: 57:69:4e:20:f9:5c:4b:cb:96:6e:23:a7:75:17:35:de: 2b:5d:63:b1:e9:aa:41:89:f8:15:9c:0b:98:71:f7:8d: 7f:b9:64:69:fd:f8:da:92:d7:05:80:89:0a:4b:62:bc: b9:87:dd:75:23:3d:82:82:9b:74:55:7c:77:3b:96:3a: ea:e4:c0:3e:85:ab:08:48:ef:c0:4b:7a:9d:69:4b:7e: 8e:0b:b8:f0:48:b3:57:b7:3e:df:67:bc:6f:2a:ff:fc: ab:47:83:4d:de:db:10:81:3f:97:f6:d8:29:a4:02:e5: b1:46:b4:ee:55:dd:c5:7c:3f:a6:d3:d8:2f:53:cf:dc: 42:df:37:80:bd:a8:0a:be:39:5f:1f:62:99:fd:09:c5: 1e:c1:ea:2d:ca:84:1a:86:c2:3e:2f:18:e9:c2:04:8f: 25:a3:ca:79:5d:a0:27:55:f5:f1:0f:15:59:d4:66:1c: b7:0f:25:90:b5:4e:63:69:99:1e:6e:39:98:52:69:6d: 39:a8:5d:79:ae:ad:bc:92:e5:68:a2:73:ca:cb:6c:1c: b4:4b:c0:90:f7:ee:82:07:62:da:bd:bb:78:28:de:0f Exponent: 65537 (0x10001) Signed Extensions: (4) Name: Certificate Authority Key Identifier Critical: False Key ID: 8e:87:7f:68:a3:86:16:5f:16:3a:b2:8d:36:6e:d4:4e: 9b:b2:33:0a Serial Number: None General Names: [0 total] Name: Authority Information Access Critical: False Name: Certificate Key Usage Critical: True Usages: Digital Signature Non-Repudiation Key Encipherment Data Encipherment Name: Extended Key Usage Critical: False Usages: TLS Web Server Authentication Certificate TLS Web Client Authentication Certificate Fingerprint (MD5): 5f:ba:ca:71:a5:fb:53:d2:ef:0d:04:87:0d:94:42:0d Fingerprint (SHA1): 6d:fb:e3:e9:4c:d8:22:42:bd:8f:7f:98:22:fc:65:37: af:60:3c:26 Signature: Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Signature Data: 9f:f4:61:6a:23:05:3b:8b:10:27:b0:42:0f:c5:ba:88: a6:e6:12:b2:3e:84:11:51:8f:75:7c:86:90:1a:d3:ea: cb:c1:fc:65:38:b5:1b:87:14:cf:51:58:2f:98:63:c8: e6:ba:f0:70:e4:ee:9b:0e:cc:7c:0f:a6:e7:0a:4c:be: ce:2d:d8:a9:e7:b0:e1:2b:6b:49:6c:e8:56:2d:04:a9: cf:a4:af:2f:6c:26:f2:f6:9d:8d:5c:4d:4e:85:3d:f3: f8:6a:bf:65:ea:1e:17:b8:04:37:cd:7c:8c:8b:1d:13: 34:da:4d:a4:1b:57:14:cb:73:ba:cb:66:e2:bd:54:0f: 09:d8:7c:85:aa:fd:0a:09:5c:75:5f:85:75:95:69:12: e2:6d:ae:11:4e:42:f2:d4:d4:be:43:50:fc:81:f0:6a: 94:64:b4:af:11:ef:ff:39:03:4c:d3:99:bf:ec:3e:79: 6d:ac:87:a1:c5:0e:cf:b1:7f:35:f6:2e:a6:ec:19:89: 69:ef:d0:0d:0e:09:18:e6:80:70:55:82:92:91:dc:b3: e5:80:a3:3c:58:36:a4:b0:ed:df:fc:64:56:cd:31:3a: cb:19:4b:00:78:5c:d6:c0:49:fb:ae:85:fb:ff:f4:9e: 68:e4:e7:bb:43:f9:40:b6:57:10:b0:1d:13:fb:7a:9e ipa: DEBUG: approved_usage = SSLServer intended_usage = SSLServer ipa: DEBUG: cert valid True for "CN=ratchet.lab.eng.pnq.redhat.com,O=LAB.ENG.PNQ.REDHAT.COM" ipa: DEBUG: handshake complete, peer = [2620:52:0:41c9:5054:ff:fea6:ec8]:443 ipa: INFO: trying https://ratchet.lab.eng.pnq.redhat.com/ipa/xml ipa: DEBUG: NSSConnection init ratchet.lab.eng.pnq.redhat.com ipa: ERROR: cannot connect to 'any of the configured servers': https://ratchet.lab.eng.pnq.redhat.com/ipa/xml, https://ratchet.lab.eng.pnq.redhat.com/ipa/xml [root@ratchet ~]#
I have created a separate ticket to fix duplicate master IPA server in the server list:
https://fedorahosted.org/freeipa/ticket/1817
This would let /usr/bin/ipa try connecting to all IPA masters just once and not print confusing server list like:
/usr/bin/ipa
# ipa user-show admin ... ipa: ERROR: cannot connect to 'any of the configured servers': https://ratchet.lab.eng.pnq.redhat.com/ipa/xml, https://ratchet.lab.eng.pnq.redhat.com/ipa/xml
Simpler demonstration program, works on 32 and 64-bit. On an IPA server this will return a 301 redirect.
import nss.nss as nss import nss.ssl as ssl from ipapython import nsslib import logging logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(levelname)s %(message)s') h = nsslib.NSSHTTPS("ipa.example.com", 443, dbdir="/etc/pki/nssdb") h.connect() h.putrequest('GET', '/') h.endheaders() http_status, http_reason, headers = h.getreply() print "status = %s %s" % (http_status, http_reason) print "headers:\n%s" % headers f = h.getfile() #data = f.read() # Get the raw HTML #f.close() #print data ssl.clear_session_cache() nss.nss_shutdown()
The NSS shutdown problem was (hopefully) fixed in python-nss 0.12, I think in IPA we're still running with 0.11. Which version of python-nss did you see this with? Since this is with RHEL6 I presume it's 0.11.
When we discussed if python-nss had to be rebased in RHEL6 the question of the NSS shutdown was brought up and it was stated it didn't seem to be an issue so we pushed the python-nss rebase out to the next cycle. I wonder why this suddenly started biting us now. Does it only show up with IPv6?
BTW, I noticed in your reproducer you're not closing the socket, I think that's necessary to free up some of the NSPR/NSS resources.
Not sure but when I did the failover code it must have been working, or maybe code changed in the interim, I'm not really sure.
This happens on IPv4 as well.
The socket is closed by the http module:
/tmp/test(13)<module>() -> http_status, http_reason, headers = h.getreply() /usr/lib/python2.6/httplib.py(1064)getreply() -> response = self._conn.getresponse() /usr/lib/python2.6/httplib.py(996)getresponse() -> self.close()
/usr/lib/python2.6/httplib.py(728)close()
Would it be easy for you to test this with python-nss-0.12? If not I could.
It is still failing with python-nss-0.12
The problem is in the HTTPResponse class, it makes another pointer to the socket in /usr/lib/python2.6/httplib.py with:
def __init__(self, sock, debuglevel=0, strict=0, method=None): self.fp = sock.makefile('rb', 0)
This is set in:
self.headers = response.msg self.file = response.fp return response.status, response.reason, response.msg
In pdb I waited until after self.file is set, ran self.file.close() and things shut down as expected.
We need to override getreply() and close self.file if the response != 200. I've worked up a basic patch.
attachment freeipa-rcrit-879-shutdown.patch
As of python-nss-0.9 the need for makefile was obsoleted. makefile was being called to add some Python file methods to a socket (e.g. readlines(), iteration, etc.). But those missing methods were added in python-nss-0.9 and makefile() essentially became a no-op which simply incremented a reference count, seemingly the exact problem you hit. Previously we had used makefile() because we want to read a line at a time from the HTTP headers (but now that is supported directly on the socket object).
Does this make sense?
Oh, almost forgot ...
So that means you can
1) remove the call to makefile() and read right off the socket object (delete .fp),
2) remember to close .fp created by makefile()
3) assign the socket to .fp (that's all that makefile() is doing, it returns the same socket object, just with an extra internal ref count).
I think option 1 is the cleanest, there is no reason to have .fp and options 2 and 3 simple obscure this simple fact and make the code more confusing.
If you want to assign this to me go ahead.
The call to makefile is in httplib.py which we don't own. I could override that function but httplib makes references to it all over the place.
My patch is option #2.
As you point out we don't own httplib.py, the reason socket.makefile() exists at all is to please httplib.py. So yes, option 2 is the right approach.
I forget how .fp gets closed using the vanilla python code.
master: a90e50c[[BR]] ipa-2-1: b8461e8
Metadata Update from @mkosek: - Issue assigned to rcritten - Issue set to the milestone: FreeIPA 2.1.2 (bug fixing)
Login to comment on this ticket.