$ kdestroy -A $ KRB5_TRACE=/dev/stdout centos-cert -u jpopelka [+] centos-cert -> Validating user [jpopelka] with realm [FEDORAPROJECT.ORG] against https://fasjson.fedoraproject.org [+] centos-cert -> Not able to negotiate kerberos with https://fasjson.fedoraproject.org ... [+] centos-cert -> Forcing kinit to obtain valid kerberos ticket Password for jpopelka@FEDORAPROJECT.ORG: Generating CSR... Uploading CSR for signature... [44321] received creds for desired service HTTP/fasjson.fedoraproject.org@FEDORAPROJECT.ORG [44321] Storing jpopelka@FEDORAPROJECT.ORG -> HTTP/fasjson.fedoraproject.org@FEDORAPROJECT.ORG in KEYRING:persistent:1000:krb_ccache_C0jrUvx [44321] Creating authenticator for jpopelka@FEDORAPROJECT.ORG -> HTTP/fasjson.fedoraproject.org@FEDORAPROJECT.ORG, seqnum 347127657, subkey aes256-cts/03CE, session key aes256-cts/779D Error: could not sign the CSR (400: <!DOCTYPE html> <html> ... </html> centos-cert -> [ISSUE] : Unable to retrieve TLS cert $ klist -A Ticket cache: KEYRING:persistent:1000:krb_ccache_C0jrUvx Default principal: jpopelka@FEDORAPROJECT.ORG Valid starting Expires Service principal 04/22/2021 13:13:15 04/22/2021 23:13:05 HTTP/fasjson.fedoraproject.org@FEDORAPROJECT.ORG renew until 04/29/2021 13:13:05 04/22/2021 13:13:13 04/22/2021 23:13:05 krbtgt/FEDORAPROJECT.ORG@FEDORAPROJECT.ORG renew until 04/29/2021 13:13:05
See https://paste.centos.org/view/ecdfbb8d for complete output.
Not sure what exactly the problem is, but the command worked for me and our bot account, packit.
While looking at /etc/krb5.conf, I can see there is also krb5.conf.rpmnew and notable difference is
/etc/krb5.conf
krb5.conf.rpmnew
+dns_canonicalize_hostname = fallback +qualify_shortname = ""
in the .rpmnew config.
.rpmnew
I already have those in my krb5.conf, thanks Tomas.
krb5.conf
Metadata Update from @mohanboddu: - Issue tagged with: medium-gain, medium-trouble, ops
Metadata Update from @mohanboddu: - Issue priority set to: Waiting on Assignee (was: Needs Review)
@arrfab have you heard about this issue from more people on the centos-infra side?
@pingou nope, that's the first issue about it and as it's fasjson related, can't have a look at log , reason why I asked @jpopelka to create ticket here, so that someone from fedora infra team could investigate at server side
Can you login ok to https://accounts.fedoraproject.org ?
What OS version / centos-cert version are you using?
Yes, I can login to https://accounts.fedoraproject.org & https://accounts.centos.org
centos-packager-0.7.0-4.fc33 fasjson-client-0.1.1-6.fc33
The paste is gone so putting the log directly here:
$ kdestroy -A $ centos-cert -u jpopelka [+] 20210427-09:50 centos-cert -> Validating user [jpopelka] with realm [FEDORAPROJECT.ORG] against https://fasjson.fedoraproject.org [+] 20210427-09:50 centos-cert -> Not able to negotiate kerberos with https://fasjson.fedoraproject.org ... [+] 20210427-09:50 centos-cert -> Forcing kinit to obtain valid kerberos ticket : Password for jpopelka@FEDORAPROJECT.ORG: Generating CSR... Uploading CSR for signature... Error: could not sign the CSR (400: <!DOCTYPE html> <html> <head> <meta charset="utf-8"> <title>Identity Management</title> <script type="text/javascript" src="../ui/js/libs/loader.js"></script> <script type="text/javascript"> var dojoConfig = { baseUrl: "../ui/js", has: { 'dojo-firebug': false, 'dojo-debug-messages': true }, parseOnLoad: false, async: true, packages: [ { name:'dojo', location:'dojo' }, { name: 'freeipa', location: 'freeipa' } ] }; (function() { var icons = [ '../ui/favicon.ico' ]; var styles = [ '../ui/css/patternfly.css', '../ui/css/ipa.css' ]; var scripts = [ '../ui/js/libs/jquery.js', '../ui/js/libs/jquery.ordered-map.js', '../ui/js/dojo/dojo.js' ]; ipa_loader.scripts(scripts, function() { require([ 'dojo/dom', 'freeipa/core', 'dojo/domReady!' ], function(dom) { var text = require('freeipa/text'); var msg = text.get('@i18n:unauthorized-page'); if (msg) { dom.byId('unauthorized-msg').innerHTML=msg; } }); }); ipa_loader.styles(styles); ipa_loader.icons(icons); })(); </script> </head> <body class="info-page"> <nav class="navbar navbar-default navbar-pf" role="navigation"> <div class="navbar-header"> <a class="brand" href="../ui/index.html"><img src="../ui/images/header-logo.png" alt="Identity Management"></a> </div> </nav> <div class="container-fluid"> <div class="row"> <div class="col-sm-12"> <div id="unauthorized-msg"> <noscript> <h1>Unable to verify your Kerberos credentials</h1> <p> Please make sure that you have valid Kerberos tickets (obtainable via <strong>kinit</strong>), and that you have configured your browser correctly. </p> <h2>Browser configuration</h2> <div id="first-time"> <p> If this is your first time, please <a href="ssbrowser.html">configure your browser</a>. </p> </div> </noscript> </div> </div> </div> </div> </body> </html> , {'message': '<!DOCTYPE html>\n<html>\n<head>\n <meta charset="utf-8">\n <title>Identity Management</title>\n <script type="text/javascript" src="../ui/js/libs/loader.js"></script>\n <script type="text/javascript">\n var dojoConfig = {\n baseUrl: "../ui/js",\n has: {\n \'dojo-firebug\': false,\n \'dojo-debug-messages\': true\n },\n parseOnLoad: false,\n async: true,\n packages: [\n {\n name:\'dojo\',\n location:\'dojo\'\n },\n {\n name: \'freeipa\',\n location: \'freeipa\'\n }\n ]\n };\n (function() {\n var icons = [\n \'../ui/favicon.ico\'\n ];\n var styles = [\n \'../ui/css/patternfly.css\',\n \'../ui/css/ipa.css\'\n ];\n var scripts = [\n \'../ui/js/libs/jquery.js\',\n \'../ui/js/libs/jquery.ordered-map.js\',\n \'../ui/js/dojo/dojo.js\'\n ];\n\n ipa_loader.scripts(scripts, function() {\n require([\n \'dojo/dom\',\n \'freeipa/core\',\n \'dojo/domReady!\'\n ],\n function(dom) {\n var text = require(\'freeipa/text\');\n var msg = text.get(\'@i18n:unauthorized-page\');\n if (msg) {\n dom.byId(\'unauthorized-msg\').innerHTML=msg;\n }\n });\n });\n ipa_loader.styles(styles);\n ipa_loader.icons(icons);\n })();\n </script>\n</head>\n\n<body class="info-page">\n\n <nav class="navbar navbar-default navbar-pf" role="navigation">\n <div class="navbar-header">\n <a class="brand" href="../ui/index.html"><img src="../ui/images/header-logo.png" alt="Identity Management"></a>\n </div>\n </nav>\n\n <div class="container-fluid">\n <div class="row">\n <div class="col-sm-12">\n <div id="unauthorized-msg">\n <noscript>\n\n <h1>Unable to verify your Kerberos credentials</h1>\n <p>\n Please make sure that you have valid Kerberos tickets (obtainable via <strong>kinit</strong>), and that you have configured your browser correctly.\n </p>\n\n <h2>Browser configuration</h2>\n\n <div id="first-time">\n <p>\n If this is your first time, please <a href="ssbrowser.html">configure your browser</a>.\n </p>\n </div>\n </noscript>\n </div>\n </div>\n </div>\n </div>\n\n</body>\n\n</html>\n', 'code': None, 'source': 'RPC'}). [+] 20210427-09:51 centos-cert -> [ISSUE] : Unable to retrieve TLS cert $ klist -A Ticket cache: KEYRING:persistent:1000:1000 Default principal: jpopelka@FEDORAPROJECT.ORG Valid starting Expires Service principal 04/27/2021 09:51:06 04/27/2021 19:50:56 HTTP/fasjson.fedoraproject.org@FEDORAPROJECT.ORG renew until 05/04/2021 09:50:56 04/27/2021 09:51:00 04/27/2021 19:50:56 krbtgt/FEDORAPROJECT.ORG@FEDORAPROJECT.ORG renew until 05/04/2021 09:50:56
As discussed last week on irc, that's fasjson-client sending request to fasjson API and fasjson answers Unable to verify your Kerberos credentials so someone should verify in fasjson log why it refused the kerberos ticket for the transaction
fasjson-client
fasjson API
Unable to verify your Kerberos credentials
Here's what the server has:
[2021-04-27 07:50:55,382] ERROR in app: Exception on /v1/me/ [GET] Traceback (most recent call last): File "/usr/lib/python3.8/site-packages/flask/app.py", line 1949, in full_dispatch_request rv = self.dispatch_request() File "/usr/lib/python3.8/site-packages/flask/app.py", line 1935, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/usr/local/lib/python3.8/site-packages/flask_restx/api.py", line 375, in wrapper resp = resource(*args, **kwargs) File "/usr/lib/python3.8/site-packages/flask/views.py", line 89, in view return self.dispatch_request(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/flask_restx/resource.py", line 44, in dispatch_request resp = meth(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/flask_restx/marshalling.py", line 248, in wrapper resp = f(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/fasjson/web/resources/me.py", line 27, in get client = ldap_client() File "/usr/local/lib/python3.8/site-packages/fasjson/web/utils/ipa.py", line 10, in ldap_client return get_client( File "/usr/local/lib/python3.8/site-packages/fasjson/lib/ldap/__init__.py", line 5, in get_client return LDAP(uri, basedn, **kwargs) File "/usr/local/lib/python3.8/site-packages/fasjson/lib/ldap/client.py", line 43, in __init__ self.conn.sasl_gssapi_bind_s(authz_id=login) File "/usr/lib64/python3.8/site-packages/ldap/ldapobject.py", line 498, in sasl_gssapi_bind_s self.sasl_non_interactive_bind_s('GSSAPI',serverctrls,clientctrls,sasl_flags,authz_id) File "/usr/lib64/python3.8/site-packages/ldap/ldapobject.py", line 486, in sasl_non_interactive_bind_s self.sasl_interactive_bind_s('',auth,serverctrls,clientctrls,sasl_flags) File "/usr/lib64/python3.8/site-packages/ldap/ldapobject.py", line 1255, in sasl_interactive_bind_s res = self._apply_method_s(SimpleLDAPObject.sasl_interactive_bind_s,*args,**kwargs) File "/usr/lib64/python3.8/site-packages/ldap/ldapobject.py", line 1224, in _apply_method_s return func(self,*args,**kwargs) File "/usr/lib64/python3.8/site-packages/ldap/ldapobject.py", line 476, in sasl_interactive_bind_s return self._ldap_call(self._l.sasl_interactive_bind_s,who,auth,RequestControlTuples(serverctrls),RequestControlTuples(clientctrls),sasl_flags) File "/usr/lib64/python3.8/site-packages/ldap/ldapobject.py", line 340, in _ldap_call reraise(exc_type, exc_value, exc_traceback) File "/usr/lib64/python3.8/site-packages/ldap/compat.py", line 46, in reraise raise exc_value File "/usr/lib64/python3.8/site-packages/ldap/ldapobject.py", line 324, in _ldap_call result = func(*args,**kwargs) ldap.LOCAL_ERROR: {'result': -2, 'desc': 'Local error', 'ctrls': [], 'info': "SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (KDC can't fulfill requested option)"}
@abompard any ideas?
The krb5kdc.log file on the IPA server hopefully has more info, I'll go check.
I found these logs:
Apr 27 07:50:56 ipa01.iad2.fedoraproject.org krb5kdc[557792](info): AS_REQ (7 etypes {aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), DEPRECATED:arcfour-hmac(23), camellia128-cts-cmac(25), camellia256-cts-cmac(26)}) 10.3.163.54: NEEDED_PREAUTH: WELLKNOWN/ANONYMOUS@FEDORAPROJECT.ORG for krbtgt/FEDORAPROJECT.ORG@FEDORAPROJECT.ORG, Additional pre-authentication required Apr 27 07:50:56 ipa01.iad2.fedoraproject.org krb5kdc[557792](info): AS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25)}) 10.3.163.54: NEEDED_PREAUTH: jpopelka@FEDORAPROJECT.ORG for krbtgt/FEDORAPROJECT.ORG@FEDORAPROJECT.ORG, Additional pre-authentication required Apr 27 07:51:06 ipa01.iad2.fedoraproject.org krb5kdc[557792](info): TGS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25)}) 10.3.163.69: EVIDENCE_TKT_NOT_FORWARDABLE: authtime 0, etypes {rep=UNSUPPORTED:(0)} HTTP/fasjson.fedoraproject.org@FEDORAPROJECT.ORG for HTTP/ipa01.iad2.fedoraproject.org@FEDORAPROJECT.ORG, KDC can't fulfill requested option
I don't know what the EVIDENCE_TKT_NOT_FORWARDABLE message means. But since it's kerberos it might be caused by something else entirely. I'll try to reproduce it.
EVIDENCE_TKT_NOT_FORWARDABLE
Here's log with KRB5_TRACE set:
KRB5_TRACE=/dev/stdout centos-cert -u jpopelka [+] 20210428-19:45 centos-cert -> Validating user [jpopelka] with realm [FEDORAPROJECT.ORG] against https://fasjson.fedoraproject.org [+] 20210428-19:45 centos-cert -> Not able to negotiate kerberos with https://fasjson.fedoraproject.org ... [+] 20210428-19:45 centos-cert -> Forcing kinit to obtain valid kerberos ticket : : Getting initial credentials for jpopelka@FEDORAPROJECT.ORG : Sending unauthenticated request : Sending request (211 bytes) to FEDORAPROJECT.ORG : Resolving hostname id.fedoraproject.org : TLS certificate name matched "id.fedoraproject.org" : Sending HTTPS request to https 2a05:...:443 : Received answer (314 bytes) from https 2a05:...:443 : Terminating TCP connection to https 2a05:...:443 : Response was not from master KDC : Received error from KDC: -1765328359/Additional pre-authentication required : Preauthenticating using KDC method data : Processing preauth types: PA-PK-AS-REQ (16), PA-FX-FAST (136), PA-ETYPE-INFO2 (19), PA-PKINIT-KX (147), PA-ENC-TIMESTAMP (2), PA_AS_FRESHNESS (150), PA-FX-COOKIE (133) : Selected etype info: etype aes256-cts, salt "...", params "" : Received cookie: MIT : PKINIT client has no configured identity; giving up : Preauth module pkinit (147) (info) returned: 0/Success : PKINIT client received freshness token from KDC : Preauth module pkinit (150) (info) returned: 0/Success : PKINIT client has no configured identity; giving up : Preauth module pkinit (16) (real) returned: 22/Invalid argument Password for jpopelka@FEDORAPROJECT.ORG: : AS key obtained for encrypted timestamp: aes256-cts/F674 : Encrypted timestamp (for 123): plain ABCD, encrypted ABCD : Preauth module encrypted_timestamp (2) (real) returned: 0/Success : Produced preauth for next request: PA-FX-COOKIE (133), PA-ENC-TIMESTAMP (2) : Sending request (306 bytes) to FEDORAPROJECT.ORG : Resolving hostname id.fedoraproject.org : TLS certificate name matched "id.fedoraproject.org" : Sending HTTPS request to https 2a05:...:443 : Received answer (791 bytes) from https 2a05:...:443 : Terminating TCP connection to https 2a05:...:443 : Response was not from master KDC : Processing preauth types: PA-ETYPE-INFO2 (19) : Selected etype info: etype aes256-cts, salt "...", params "" : Produced preauth for next request: (empty) : AS key determined by preauth: aes256-cts/F674 : Decrypted AS reply; session key is: aes256-cts/C383 : FAST negotiation: available : Initializing KEYRING:persistent:1000:1000 with default princ jpopelka@FEDORAPROJECT.ORG : Storing jpopelka@FEDORAPROJECT.ORG -> krbtgt/FEDORAPROJECT.ORG@FEDORAPROJECT.ORG in KEYRING:persistent:1000:1000 : Storing config in KEYRING:persistent:1000:1000 for krbtgt/FEDORAPROJECT.ORG@FEDORAPROJECT.ORG: fast_avail: yes : Storing jpopelka@FEDORAPROJECT.ORG -> krb5_ccache_conf_data/fast_avail/krbtgt\/FEDORAPROJECT.ORG\@FEDORAPROJECT.ORG@X-CACHECONF: in KEYRING:persistent:1000:1000 : Storing config in KEYRING:persistent:1000:1000 for krbtgt/FEDORAPROJECT.ORG@FEDORAPROJECT.ORG: pa_type: 2 : Storing jpopelka@FEDORAPROJECT.ORG -> krb5_ccache_conf_data/pa_type/krbtgt\/FEDORAPROJECT.ORG\@FEDORAPROJECT.ORG@X-CACHECONF: in KEYRING:persistent:1000:1000 Generating CSR... Uploading CSR for signature... : Getting credentials jpopelka@FEDORAPROJECT.ORG -> HTTP/fasjson.fedoraproject.org@FEDORAPROJECT.ORG using ccache KEYRING:persistent:1000:1000 : Retrieving jpopelka@FEDORAPROJECT.ORG -> HTTP/fasjson.fedoraproject.org@FEDORAPROJECT.ORG from KEYRING:persistent:1000:1000 with result: -1765328243/Matching credential not found : Retrieving jpopelka@FEDORAPROJECT.ORG -> krbtgt/FEDORAPROJECT.ORG@FEDORAPROJECT.ORG from KEYRING:persistent:1000:1000 with result: 0/Success : Starting with TGT for client realm: jpopelka@FEDORAPROJECT.ORG -> krbtgt/FEDORAPROJECT.ORG@FEDORAPROJECT.ORG : Requesting tickets for HTTP/fasjson.fedoraproject.org@FEDORAPROJECT.ORG, referrals on : Generated subkey for TGS request: aes256-cts/E9F8 : etypes requested in TGS request: aes256-cts, aes256-sha2, camellia256-cts, aes128-cts, aes128-sha2, camellia128-cts : Encoding request body and padata into FAST request : Sending request (998 bytes) to FEDORAPROJECT.ORG : Resolving hostname id.fedoraproject.org : TLS certificate name matched "id.fedoraproject.org" : Sending HTTPS request to https 2a05:...:443 : Received answer (975 bytes) from https 2a05:...:443 : Terminating TCP connection to https 2a05:...:443 : Response was not from master KDC : Decoding FAST response : FAST reply key: aes256-cts/9B64 : TGS reply is for jpopelka@FEDORAPROJECT.ORG -> HTTP/fasjson.fedoraproject.org@FEDORAPROJECT.ORG with session key aes256-cts/B86F : TGS request result: 0/Success : Received creds for desired service HTTP/fasjson.fedoraproject.org@FEDORAPROJECT.ORG : Storing jpopelka@FEDORAPROJECT.ORG -> HTTP/fasjson.fedoraproject.org@FEDORAPROJECT.ORG in KEYRING:persistent:1000:1000 : Creating authenticator for jpopelka@FEDORAPROJECT.ORG -> HTTP/fasjson.fedoraproject.org@FEDORAPROJECT.ORG, seqnum 93029989, subkey aes256-cts/E590, session key aes256-cts/B86F Error: could not sign the CSR (400: <!DOCTYPE html>
And here's my /etc/krb5.conf
includedir /etc/krb5.conf.d/ [logging] default = FILE:/var/log/krb5libs.log kdc = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.log [libdefaults] dns_lookup_realm = false ticket_lifetime = 36000 renew_lifetime = 7d # forwardable = true rdns = false # pkinit_anchors = FILE:/etc/pki/tls/certs/ca-bundle.crt # spake_preauth_groups = edwards25519 dns_canonicalize_hostname = fallback qualify_shortname = "" default_realm = REDHAT.COM default_ccache_name = KEYRING:persistent:%{uid} dns_lookup_kdc = false [realms] REDHAT.COM = { kdc = kerberos01.core.prod.int.phx2.redhat.com.:88 kdc = kerberos.rdu.redhat.com.:88 kdc = kerberos.bos.redhat.com.:88 kdc = kerberos.brq.redhat.com.:88 admin_server = kerberos.corp.redhat.com.:749 default_domain = redhat.com } [domain_realm] .redhat.com = REDHAT.COM redhat.com = REDHAT.COM [kdc] profile = /var/kerberos/krb5kdc/kdc.conf [pam] debug = false ticket_lifetime = 36000 renew_lifetime = 36000 forwardable = true krb4_convert = false
And /etc/krb5.conf.d/fedoraproject_org
[realms] FEDORAPROJECT.ORG = { kdc = https://id.fedoraproject.org/KdcProxy pkinit_anchors = FILE:/etc/pki/ipa/fedoraproject_ipa_ca.crt } [domain_realm] .fedoraproject.org = FEDORAPROJECT.ORG fedoraproject.org = FEDORAPROJECT.ORG
I'm a bit out of my league here, could somebody with kerberos knowledge look at this? Maybe @cheimes ? I couldn't reproduce it locally.
FreeIPA uses credential delegation with s4u2proxy to archive privilege separation.
The problem could be related to FAS proxy setup. Is the principal HTTP/fasjson.fedoraproject.org allowed perform delegation? It should be a memberPrincipal of cn=fasjson-http-delegation,cn=s4u2proxy,cn=etc,$SUFFIX.
cn=fasjson-http-delegation,cn=s4u2proxy,cn=etc,$SUFFIX
Scratch that
I think it's missing forwardable = true in @jpopelka 's krb5.conf. Jiri has commented out the flag and its default setting is "false". IPA requires forwardable tickets for s4u2proxy.
forwardable = true
Yes, that was it! No idea, why I had it commented out. Thank you all! The ticket can be closed.
Certificate generated, signed and written to /home/jpopelka/.centos-jpopelka.crt [+] 20210430-11:33 centos-cert -> Concatenating cert to ~/.centos.cert [+] 20210430-11:33 centos-cert -> Downloading correct CA cert .. [+] 20210430-11:33 centos-cert -> Verifying if TLS cert is still valid ... [+] 20210430-11:33 centos-cert -> Validating TLS cert against ~/.centos-server-ca.cert ... /home/jpopelka/.centos.cert: OK [+] 20210430-11:33 centos-cert -> [SUCCESS] ~/.centos.cert TLS cert verified by ~/.centos-server-ca.cert CA crt [+] 20210430-11:33 centos-cert -> [SUCCESS] Your TLS cert is still valid for [730] days
Thanks everyone!
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.