As admin , I want krb5kdc not to crash so that Kerberos works.
Based on a howto somewhere, I have set up various Windows machines so that users can log into those machines using FreeIPA principal names. This works and has worked for quite a while.
However, at some point, krb5kdc started dumping core all the time. I finally got around to debugging it, and it appears to caused by aliases that are part of the Windows setup.
Based on some howto somewhere, I've created a principal alias of a particular form for each of these hosts. The aliases are called "<bare-hostname>$", so for example the host vrtoy.kdjf.net, with primary principal name "host/vrtoy.kdjf.net@KDJF.NET", has the principal alias "vrtoy$@KDJF.NET" (see LDIF below).
It appears that whenever anything tries to get a ticket for one of these aliases, the KDC crashes. I can cause a crash from the command line with a command like "kinit 'vrtoy$'". The crash is immediate. I am not asked for a password. Crash dump details are below.
kinit 'vrtoy$'
Much later edit for the benefit of future Web searchers: "a howto somewhere" appears to have been https://www.freeipa.org/page/Windows_authentication_against_FreeIPA . In addition to what's there, at least on Windows 10, if you've defined "hostname$" principals as described below, you may want to do "ksetup /setrealmflags your-realm NcSupported" on each Windows host. The benefit of the "hostname$" aliases is to let Windows hosts do dynamic DNS updates against FreeIPA, which will not happen without that setting, at least not with Windows 10 as of 2020-05-04.
ksetup /setrealmflags your-realm NcSupported
Here's the LDIF for the host that causes the most crashes:
[root@ipa-0 systemd]# ldapsearch -b 'dc=kdjf,dc=net' -o 'ldif-wrap=99999' '(cn=vrtoy.kdjf.net)' SASL/EXTERNAL authentication started SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth SASL SSF: 0 # extended LDIF # # LDAPv3 # base <dc=kdjf,dc=net> with scope subtree # filter: (cn=vrtoy.kdjf.net) # requesting: ALL # # vrtoy.kdjf.net, computers, accounts, kdjf.net dn: fqdn=vrtoy.kdjf.net,cn=computers,cn=accounts,dc=kdjf,dc=net cn: vrtoy.kdjf.net description: VR machine in living room. Formerly in Warren's room. enrolledBy: uid=bossjbash,cn=users,cn=accounts,dc=kdjf,dc=net fqdn: vrtoy.kdjf.net ipaUniqueID: 9f7bdb68-b57d-11e9-b0c4-001cc4edc5c4 krbCanonicalName: host/vrtoy.kdjf.net@KDJF.NET krbExtraData:: AAL9okhdaG9zdC92cnRveS5rZGpmLm5ldEBLREpGLk5FVAA= krbLastPwdChange: 20190805214325Z krbPrincipalKey:: <removed-yeah-i-know-its-encrypted> krbPrincipalName: host/vrtoy.kdjf.net@KDJF.NET krbPrincipalName: vrtoy$@KDJF.NET macAddress: E0:3F:49:E7:B7:FC macAddress: E0:3F:49:E7:B7:FD managedBy: fqdn=vrtoy.kdjf.net,cn=computers,cn=accounts,dc=kdjf,dc=net memberOf: cn=household,cn=hostgroups,cn=accounts,dc=kdjf,dc=net memberOf: cn=household,cn=ng,cn=alt,dc=kdjf,dc=net memberOf: ipaUniqueID=ee7d027a-b57b-11e9-83bb-001cc4edc5c4,cn=hbac,dc=kdjf,dc=net memberOf: cn=hh-fsclients,cn=hostgroups,cn=accounts,dc=kdjf,dc=net memberOf: cn=hh-fsclients,cn=ng,cn=alt,dc=kdjf,dc=net memberOf: ipaUniqueID=ec04d4fa-b57b-11e9-9db3-001cc4edc5c4,cn=ng,cn=alt,dc=kdjf,dc=net objectClass: ipaobject objectClass: nshost objectClass: ipahost objectClass: pkiuser objectClass: ipaservice objectClass: krbprincipalaux objectClass: krbprincipal objectClass: ieee802device objectClass: ipasshhost objectClass: top objectClass: ipaSshGroupOfPubKeys serverHostName: vrtoy krbPwdPolicyReference: cn=Default Host Password Policy,cn=computers,cn=accounts,dc=kdjf,dc=net # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1
Here's what coredumpctl and gdb tell me (this crash is from a kinit with a different host's principal alias; the crashes from the above host look identical):
[root@ipa-0 systemd]# coredumpctl debug 14477 PID: 14477 (krb5kdc) UID: 0 (root) GID: 0 (root) Signal: 11 (SEGV) Timestamp: Sat 2020-04-25 14:01:37 EDT (27min ago) Command Line: /usr/sbin/krb5kdc -P /var/run/krb5kdc.pid -w 4 Executable: /usr/sbin/krb5kdc Control Group: /system.slice/krb5kdc.service Unit: krb5kdc.service Slice: system.slice Boot ID: 77437c79ff4c43bc8513729a6a601198 Machine ID: 5c5035d7d87d4d1584d233ee52a1324d Hostname: ipa-0.kdjf.net Storage: /var/lib/systemd/coredump/core.krb5kdc.0.77437c79ff4c43bc8513729a6a601198.14477.1587837697000000000000.lz4 Message: Process 14477 (krb5kdc) of user 0 dumped core. Stack trace of thread 14477: #0 0x00007f95879167b0 __GI___libc_free (libc.so.6) #1 0x00007f9586b38944 ber_bvfree_x (liblber-2.4.so.2) #2 0x00007f9586b389cf ber_bvecfree_x (liblber-2.4.so.2) #3 0x00007f958714f867 ipadb_find_principal (ipadb.so) #4 0x00007f9587150d96 ipadb_get_principal (ipadb.so) #5 0x00007f9587badc13 krb5_db_get_principal (libkdb5.so.9) #6 0x000055961cebbd65 process_as_req (krb5kdc) #7 0x000055961cebaa69 dispatch (krb5kdc) #8 0x000055961cecff85 process_packet (krb5kdc) #9 0x00007f9587a5739c verto_fire (libverto.so.1) #10 0x00007f9585d6e9b3 ev_invoke_pending (libev.so.4) #11 0x00007f9585d72353 ev_run (libev.so.4) #12 0x000055961ceb9705 main (krb5kdc) #13 0x00007f95878b11a3 __libc_start_main (libc.so.6) #14 0x000055961ceb9b7e _start (krb5kdc) GNU gdb (GDB) Fedora 8.3.50.20190824-30.fc31 Copyright (C) 2019 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/krb5kdc... Reading symbols from /usr/lib/debug/usr/sbin/krb5kdc-1.17-46.fc31.x86_64.debug... [New LWP 14477] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/krb5kdc -P /var/run/krb5kdc.pid -w 4'. Program terminated with signal SIGSEGV, Segmentation fault. #0 __GI___libc_free (mem=0x7000700070007) at malloc.c:3102 3102 p = mem2chunk (mem); Missing separate debuginfos, use: dnf debuginfo-install cyrus-sasl-gssapi-2.1.27-3.fc31.x86_64 cyrus-sasl-lib-2.1.27-3.fc31.x86_64 cyrus-sasl-md5-2.1.27-3.fc31.x86_64 cyrus-sasl-plain-2.1.27-3.fc31.x86_64 gmp-6.1.2-10.fc31.x86_64 gnutls-3.6.13-1.fc31.x86_64 libdb-5.3.28-38.fc31.x86_64 libev-4.27-1.fc31.x86_64 libffi-3.1-23.fc31.x86_64 libgcc-9.3.1-1.fc31.x86_64 libgcrypt-1.8.5-1.fc31.x86_64 libgpg-error-1.36-2.fc31.x86_64 libidn2-2.3.0-1.fc31.x86_64 libsss_certmap-2.2.3-13.fc31.x86_64 libtalloc-2.3.0-1.fc31.x86_64 libtasn1-4.14-2.fc31.x86_64 libtevent-0.10.1-1.fc31.x86_64 libunistring-0.9.10-6.fc31.x86_64 libxcrypt-4.4.16-1.fc31.x86_64 lz4-libs-1.9.1-1.fc31.x86_64 nettle-3.5.1-3.fc31.x86_64 nspr-4.25.0-1.fc31.x86_64 nss-3.51.0-1.fc31.x86_64 nss-util-3.51.0-1.fc31.x86_64 p11-kit-0.23.20-1.fc31.x86_64 samba-client-libs-4.11.7-0.fc31.x86_64 samba-common-libs-4.11.7-0.fc31.x86_64 systemd-libs-243.8-1.fc31.x86_64 xz-libs-5.2.4-6.fc31.x86_64 (gdb) info args mem = 0x7000700070007 (gdb) up #1 0x00007f9586b38944 in ber_bvfree_x (bv=0x55961ec95010, ctx=0x0) at memory.c:386 386 ber_memfree_x( bv->bv_val, ctx ); (gdb) info args bv = 0x55961ec95010 ctx = 0x0 (gdb) up #2 0x00007f9586b389cf in ber_bvecfree_x (ctx=<optimized out>, bv=<optimized out>) at memory.c:414 414 ber_bvfree_x( bv[i], ctx ); (gdb) info args ctx = <optimized out> bv = <optimized out> (gdb) up #3 ber_bvecfree_x (bv=0x55961ecc5ef0, ctx=0x0) at memory.c:399 399 ber_bvecfree_x( struct berval **bv, void *ctx ) (gdb) info args bv = 0x55961ecc5ef0 ctx = 0x0 (gdb) print *bv[0] $1 = {bv_len = 94103251542704, bv_val = 0x55961ec95010 "\003"} (gdb) print *bv[1] $2 = {bv_len = 1970354902138883, bv_val = 0x7000700070007 <error: Cannot access memory at address 0x7000700070007>} (gdb) up #4 0x00007f958714f867 in ipadb_find_principal (kcontext=kcontext@entry=0x55961ee12290, flags=flags@entry=96, res=<optimized out>, principal=principal@entry=0x7ffce6f8cd88, entry=entry@entry=0x7ffce6f8cda0) at ipa_kdb_principals.c:1130 1130 ldap_value_free_len(vals); (gdb) print *principal $3 = 0x55961ecd94e0 "craft-lenovo$@KDJF.NET" (gdb) print *entry $4 = (LDAPMessage *) 0x55961ee169f0 (gdb) print **entry $5 = {lm_msgid = -1760647422, lm_msgtype = 94103250284512, lm_ber = 0x0, lm_chain = 0x21, lm_chain_tail = 0x54454e2e464a444b, lm_next = 0x0, lm_time = 107374182423} (gdb) print kcontext $6 = (krb5_context) 0x55961ee12290 (gdb) print *kcontext $7 = {magic = -1760647388, in_tkt_etypes = 0x0, tgs_etypes = 0x0, os_context = {magic = -1760647387, time_offset = 0, usec_offset = 0, os_flags = 0, default_ccname = 0x0}, default_realm = 0x55961ece32f0 "KDJF.NET", profile = 0x55961ece48f0, dal_handle = 0x55961ece4e60, ser_ctx_count = 0, ser_ctx = 0x0, clockskew = 300, kdc_default_options = 16, library_options = 1, profile_secure = 0, fcc_default_format = 1284, prompt_types = 0x0, udp_pref_limit = -1, use_conf_ktypes = 0, libkrb5_plugins = { files = 0x0}, preauth_context = 0x0, ccselect_handles = 0x0, localauth_handles = 0x0, hostrealm_handles = 0x0, tls = 0x0, err = { code = 2, msg = 0x0}, err_fmt = 0x0, kdblog_context = 0x0, allow_weak_crypto = 0, ignore_acceptor_hostname = 0, dns_canonicalize_hostname = CANONHOST_TRUE, trace_callback = 0x0, trace_callback_data = 0x0, kdc_send_hook = 0x0, kdc_send_hook_data = 0x0, kdc_recv_hook = 0x0, kdc_recv_hook_data = 0x0, plugins = {{modules = 0x0, configured = 0} <repeats 13 times>}, plugin_base_dir = 0x55961ee0c400 "/usr/lib64/krb5/plugins"} (gdb) quit
I haven't reproduced this outside of my working environment. I suspect that creating a host principal alias of the form "<bare-hostname>$" would be enough, but I do admit that my installation has been around a while and had things done to it.
KDC crashes with SIGSEGV.
KDC doesn't crash, host gets its credentials.
[root@ipa-0 systemd]# rpm -q freeipa-server freeipa-client ipa-server ipa-client 389-ds-base pki-ca krb5-server freeipa-server-4.8.6-1.fc31.x86_64 freeipa-client-4.8.6-1.fc31.x86_64 package ipa-server is not installed package ipa-client is not installed 389-ds-base-1.4.2.11-1.fc31.x86_64 pki-ca-10.8.3-1.fc31.noarch krb5-server-1.17-46.fc31.x86_64
@jbash thanks for the report. Can you please try the following patch? <img alt="freeipa-fix-kdc-crash-8291.patch" src="/freeipa/issue/raw/files/da5ade10aa2a55a40c398198ff0dafdb0f36d6d01ca2ba10bbbb865cabf259ce-freeipa-fix-kdc-crash-8291.patch" />
I am not entirely sure whether we could get into that state but there are some issues with the code of ipadb_find_principal() that my patch fixes. They related also to leaving out older references to vals when processing entries searched.
ipadb_find_principal()
Your stacktrace reports code lines that are quite off -- 1130 is the definition of ipadb_find_principal() function, not the line where vals are freed (and we have few of those). Also, looks like you did too much clean up in the ldap entry output compared to the backtrace as the NetBIOS name for the machine is different in the gdb session output from LDIF.
Can you please try the following patch?
I'll try to install it tonight. Thank you.
Your stacktrace reports code lines that are quite off -- 1130 is the definition of ipadb_find_principal() function, not the line where vals are freed (and we have few of those).
Apparently not in whatever source the version I have was built from. See the listing in the gdb output below.
Also, looks like you did too much clean up in the ldap entry output compared to the backtrace as the NetBIOS name for the machine is different in the gdb session output from LDIF.
As I said, the gdb output was from the principal for a different host (and also from a kinit as opposed to whatever the host itself is doing).
Here's similar gdb output from a core dump that should represent an organic attempt to get credentials by the same host that's in the LDIF.
[root@ipa-0 ~]# coredumpctl debug 14961 PID: 14961 (krb5kdc) UID: 0 (root) GID: 0 (root) Signal: 11 (SEGV) Timestamp: Sat 2020-04-25 15:18:02 EDT (34min ago) Command Line: /usr/sbin/krb5kdc -P /var/run/krb5kdc.pid -w 4 Executable: /usr/sbin/krb5kdc Control Group: /system.slice/krb5kdc.service Unit: krb5kdc.service Slice: system.slice Boot ID: 77437c79ff4c43bc8513729a6a601198 Machine ID: 5c5035d7d87d4d1584d233ee52a1324d Hostname: ipa-0.kdjf.net Storage: /var/lib/systemd/coredump/core.krb5kdc.0.77437c79ff4c43bc8513729a6a601198.14961.1587842282000000000000.lz4 Message: Process 14961 (krb5kdc) of user 0 dumped core. Stack trace of thread 14961: #0 0x00007f6b31ade7b0 __GI___libc_free (libc.so.6) #1 0x00007f6b30d00944 ber_bvfree_x (liblber-2.4.so.2) #2 0x00007f6b30d009cf ber_bvecfree_x (liblber-2.4.so.2) #3 0x00007f6b31317867 ipadb_find_principal (ipadb.so) #4 0x00007f6b31318d96 ipadb_get_principal (ipadb.so) #5 0x00007f6b31d75c13 krb5_db_get_principal (libkdb5.so.9) #6 0x000055b2f0a80d65 process_as_req (krb5kdc) #7 0x000055b2f0a7fa69 dispatch (krb5kdc) #8 0x000055b2f0a94f85 process_packet (krb5kdc) #9 0x00007f6b31c1f39c verto_fire (libverto.so.1) #10 0x00007f6b2ff369b3 ev_invoke_pending (libev.so.4) #11 0x00007f6b2ff3a353 ev_run (libev.so.4) #12 0x000055b2f0a7e705 main (krb5kdc) #13 0x00007f6b31a791a3 __libc_start_main (libc.so.6) #14 0x000055b2f0a7eb7e _start (krb5kdc) GNU gdb (GDB) Fedora 8.3.50.20190824-30.fc31 Copyright (C) 2019 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/krb5kdc... Reading symbols from /usr/lib/debug/usr/sbin/krb5kdc-1.17-46.fc31.x86_64.debug... [New LWP 14961] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/krb5kdc -P /var/run/krb5kdc.pid -w 4'. Program terminated with signal SIGSEGV, Segmentation fault. #0 __GI___libc_free (mem=0x7000700070007) at malloc.c:3102 3102 p = mem2chunk (mem); Missing separate debuginfos, use: dnf debuginfo-install cyrus-sasl-gssapi-2.1.27-3.fc31.x86_64 cyrus-sasl-lib-2.1.27-3.fc31.x86_64 cyrus-sasl-md5-2.1.27-3.fc31.x86_64 cyrus-sasl-plain-2.1.27-3.fc31.x86_64 gmp-6.1.2-10.fc31.x86_64 gnutls-3.6.13-1.fc31.x86_64 libdb-5.3.28-38.fc31.x86_64 libev-4.27-1.fc31.x86_64 libffi-3.1-23.fc31.x86_64 libgcc-9.3.1-1.fc31.x86_64 libgcrypt-1.8.5-1.fc31.x86_64 libgpg-error-1.36-2.fc31.x86_64 libidn2-2.3.0-1.fc31.x86_64 libsss_certmap-2.2.3-13.fc31.x86_64 libtalloc-2.3.0-1.fc31.x86_64 libtasn1-4.14-2.fc31.x86_64 libtevent-0.10.1-1.fc31.x86_64 libunistring-0.9.10-6.fc31.x86_64 libxcrypt-4.4.16-1.fc31.x86_64 lz4-libs-1.9.1-1.fc31.x86_64 nettle-3.5.1-3.fc31.x86_64 nspr-4.25.0-1.fc31.x86_64 nss-3.51.0-1.fc31.x86_64 nss-util-3.51.0-1.fc31.x86_64 p11-kit-0.23.20-1.fc31.x86_64 samba-client-libs-4.11.7-0.fc31.x86_64 samba-common-libs-4.11.7-0.fc31.x86_64 systemd-libs-243.8-1.fc31.x86_64 xz-libs-5.2.4-6.fc31.x86_64 (gdb) set listsize 50 (gdb) info args mem = 0x7000700070007 (gdb) up #1 0x00007f6b30d00944 in ber_bvfree_x (bv=0x55b2f2105010, ctx=0x0) at memory.c:386 386 ber_memfree_x( bv->bv_val, ctx ); (gdb) info args bv = 0x55b2f2105010 ctx = 0x0 (gdb) up #2 0x00007f6b30d009cf in ber_bvecfree_x (ctx=<optimized out>, bv=<optimized out>) at memory.c:414 414 ber_bvfree_x( bv[i], ctx ); (gdb) info args ctx = <optimized out> bv = <optimized out> (gdb) up #3 ber_bvecfree_x (bv=0x55b2f21295a0, ctx=0x0) at memory.c:399 399 ber_bvecfree_x( struct berval **bv, void *ctx ) (gdb) info args bv = 0x55b2f21295a0 ctx = 0x0 (gdb) up #4 0x00007f6b31317867 in ipadb_find_principal ( kcontext=kcontext@entry=0x55b2f2282290, flags=flags@entry=96, res=<optimized out>, principal=principal@entry=0x7ffc648605e8, entry=entry@entry=0x7ffc64860600) at ipa_kdb_principals.c:1130 1130 ldap_value_free_len(vals); (gdb) list 1105 if (!found) { 1106 ldap_value_free_len(vals); 1107 continue; 1108 } 1109 } 1110 #endif 1111 1112 free(*principal); 1113 *principal = strdup(vals[0]->bv_val); 1114 if (!*principal) { 1115 ret = KRB5_KDB_INTERNAL_ERROR; 1116 goto done; 1117 } 1118 break; 1119 } 1120 1121 if (!found || !le) { 1122 ret = KRB5_KDB_NOENTRY; 1123 goto done; 1124 } 1125 1126 ret = 0; 1127 *entry = le; 1128 done: 1129 if (vals) 1130 ldap_value_free_len(vals); 1131 1132 return ret; 1133 } 1134 1135 static krb5_flags maybe_require_preauth(struct ipadb_context *ipactx, 1136 krb5_db_entry *entry) 1137 { 1138 const struct ipadb_global_config *config; 1139 struct ipadb_e_data *ied; 1140 1141 config = ipadb_get_global_config(ipactx); 1142 if (config && config->disable_preauth_for_spns) { 1143 ied = (struct ipadb_e_data *)entry->e_data; 1144 if (ied && ied->ipa_user != true) { 1145 /* not a user, assume SPN */ 1146 return 0; 1147 } 1148 } 1149 1150 /* By default require preauth for all principals */ 1151 return KRB5_KDB_REQUIRES_PRE_AUTH; 1152 } 1153 1154 static krb5_error_code ipadb_fetch_tktpolicy(krb5_context kcontext, (gdb) print *principal $1 = 0x55b2f2135b30 "vrtoy$@KDJF.NET" (gdb) print **entry $2 = {lm_msgid = -1760647422, lm_msgtype = 94227054083184, lm_ber = 0x20, lm_chain = 0xa1, lm_chain_tail = 0x0, lm_next = 0x0, lm_time = 0} (gdb) print *kcontext $3 = {magic = -1760647388, in_tkt_etypes = 0x0, tgs_etypes = 0x0, os_context = {magic = -1760647387, time_offset = 0, usec_offset = 0, os_flags = 0, default_ccname = 0x0}, default_realm = 0x55b2f21532f0 "KDJF.NET", profile = 0x55b2f21548f0, dal_handle = 0x55b2f2154e60, ser_ctx_count = 0, ser_ctx = 0x0, clockskew = 300, kdc_default_options = 16, library_options = 1, profile_secure = 0, fcc_default_format = 1284, prompt_types = 0x0, udp_pref_limit = -1, use_conf_ktypes = 0, libkrb5_plugins = {files = 0x0}, preauth_context = 0x0, ccselect_handles = 0x0, localauth_handles = 0x0, hostrealm_handles = 0x0, tls = 0x0, err = {code = 2, msg = 0x0}, err_fmt = 0x0, kdblog_context = 0x0, allow_weak_crypto = 0, ignore_acceptor_hostname = 0, dns_canonicalize_hostname = CANONHOST_TRUE, trace_callback = 0x0, trace_callback_data = 0x0, kdc_send_hook = 0x0, kdc_send_hook_data = 0x0, kdc_recv_hook = 0x0, kdc_recv_hook_data = 0x0, plugins = {{modules = 0x0, configured = 0} <repeats 13 times>}, plugin_base_dir = 0x55b2f227c400 "/usr/lib64/krb5/plugins"} (gdb) quit
<img alt="freeipa-fix-kdc-crash-8291.patch" src="/freeipa/issue/raw/files/693056b2114b0ab4c8c999718907339ee31e1ecec299eace2c25759288dd8c7d-freeipa-fix-kdc-crash-8291.patch" />
Ok, this version should fix the problem. Since ldap_value_free_len(vals) does not NULL the vals, we need to do that explicitly. Then we wouldn't do a double-free in the done: path.
ldap_value_free_len(vals)
vals
done:
Thank you very much for this investigation, it is very helpful!
... and thank you for your quick response.
Unfortunately, the patch doesn't seem to help. I installed it, and the KDC is still crashing in the same way.
I checked the code in the RPM build directory to verify that the patch was being built. It seems to have applied OK. It's just not having the intended effect.
What else can I get you?
I was looking at old notes and things to see if there was anything weird about my installation that might cause this, and I found something.
This installation is a reinstall of a previously existing FreeIPA setup. The old one lunched itself somehow, and I couldn't restore a working system from the backups I had. Because of previous problems with creeping uncorrectable corruption in FreeIPA, I maintain a shell script that's meant to be able recreate everything, and I used that script to do the reinstall.
However, I didn't want to have to change every user password and rekey every host. Therefore I:
krbPrincipalKey
krbExtraData
At least I THINK I did that. It's possible that I tried to do it, failed, and actually had to rekey everything. I know that I was not able to save my old CA key, and I don't think I was able to save my old DNSSEC keys. However, as I recall, my attempts to save the Kerberos keys did work, and everybody's password stayed the same. All principals other than the Windows hosts have been working fine ever since.
If that Kerberos rekeying hack did happen, I'm also not sure whether the "vrtoy" host (or many of the other Windows hosts) was in the system before the reinstall (and would therefore have been affected), or was added afterwards. I think they were in there beforehand. Assuming they were, I don't think any of their credentials have been changed since.
I found the script that would have generated the LDIF, and tried using it to extract the keys from the the present installation. It generates ldif like this:
### dn: fqdn=vrtoy.kdjf.net,cn=computers,cn=accounts,dc=kdjf,dc=net changetype: modify replace: krbExtraData krbExtraData:: AAL9okhdaG9zdC92cnRveS5rZGpmLm5ldEBLREpGLk5FVAA= - replace: krbPrincipalKey krbPrincipalKey:: <redacted> -
Note that, although the form of that is presumably what would have been used in a rekey, the content is taken from the current database, so it will reflect any changes after the rekey happened. The original LDIF would not be around any more.
I don't think this update could have caused the issue. Windows systems are different because they use principal name to salt the keys so if there was a change in realm, for example, that would make it impossible to migrate to. IPA principals are all using random salt which is independent of a principal name.
If you want, I can make a COPR repo for you to make sure the build really includes the package. If you are using right package, there shouldn't (famous last words, I know) be a crash but if that happens, gdb should report a different location for it. Ideally, I'd like to see the state of all local variables in ipadb_find_principal() as I simply couldn't reproduce the issue locally.
Simo pointed to one place where we might crash. I'll upload updated patch soon.
<img alt="freeipa-fix-kdc-crash-8291.patch" src="/freeipa/issue/raw/files/40df4094f420e64ff9b83b8c34727649cd7f4aeb867c4e1d461b7e8f866ab566-freeipa-fix-kdc-crash-8291.patch" />
Updated patch.
OK, thanks, that patch keeps the KDC from crashing.
The "vrtoy$" principal still doesn't actually work, though (nor do the other similar names). I now get
[jbash@mctl ~]$ kinit 'vrtoy$' kinit: Client 'vrtoy$@KDJF.NET' not found in Kerberos database while getting initial credentials
Other principals, both users and hosts, do get found, including the "host/vrtoy.kdjf.net" principal.
Should I submit another issue for that?
I found some notes saying that the reason those principals have to exist is to let the Windows hosts do dynamic DNS updates.
I ran our standard set of tests against this patch, including Kerberos principal alias tests in ipatests/test_xmlrpc/test_kerberos_principal_aliases.py, it worked just fine.
ipatests/test_xmlrpc/test_kerberos_principal_aliases.py
I added a debug line right before we do a check in if (!found || !le) {...} and I can see in the logs:
if (!found || !le) {...}
$ grep alias krb5kdc.log Apr 28 06:23:33 master1.ipa.test krb5kdc[3531](Error): Searched for krbalias_user@IPA.TEST, found:1, LDAP entry:0x55f5edb9faa0 Apr 28 06:23:33 master1.ipa.test krb5kdc[3531](info): AS_REQ (7 etypes {aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), DEPRECATED:arcfour-hmac(23), camellia128-cts-cmac(25), camellia256-cts-cmac(26)}) 172.18.0.2: NEEDED_PREAUTH: krbalias_user-alias@IPA.TEST for krbtgt/IPA.TEST@IPA.TEST, Additional pre-authentication required Apr 28 06:23:33 master1.ipa.test krb5kdc[3530](Error): Searched for krbalias_user@IPA.TEST, found:1, LDAP entry:0x55f5edc16ee0 Apr 28 06:23:33 master1.ipa.test krb5kdc[3530](info): AS_REQ (7 etypes {aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), DEPRECATED:arcfour-hmac(23), camellia128-cts-cmac(25), camellia256-cts-cmac(26)}) 172.18.0.2: ISSUE: authtime 1588055013, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18), ses=aes256-cts-hmac-sha1-96(18)}, krbalias_user-alias@IPA.TEST for krbtgt/IPA.TEST@IPA.TEST Apr 28 06:23:33 master1.ipa.test krb5kdc[3531](Error): Searched for krbalias_user@IPA.TEST, found:1, LDAP entry:0x55f5edc22410 Apr 28 06:23:33 master1.ipa.test krb5kdc[3531](info): TGS_REQ (7 etypes {aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), DEPRECATED:arcfour-hmac(23), camellia128-cts-cmac(25), camellia256-cts-cmac(26)}) 172.18.0.2: ISSUE: authtime 1588055013, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18), ses=aes256-cts-hmac-sha1-96(18)}, krbalias_user@IPA.TEST for HTTP/master1.ipa.test@IPA.TEST Apr 28 06:23:33 master1.ipa.test krb5kdc[3531](Error): Searched for krbalias_user@IPA.TEST, found:1, LDAP entry:0x55f5edbd4950 Apr 28 06:23:33 master1.ipa.test krb5kdc[3531](info): ... CONSTRAINED-DELEGATION s4u-client=krbalias_user@IPA.TEST
The logs can be pulled from Azure Pipeline I have here: https://dev.azure.com/abbra1freeipa/freeipa/_build/results?buildId=364&view=artifacts&type=publishedArtifacts (in logs-BASE_XMLRPC... archive, in xmlrpc folder).
... but nonetheless, I have an alias in my database and it's not being found. It's not just for manual kinits, either; the host itself can't use the principal. I have many instances of this in my krb5kdc.log:
Apr 28 14:36:03 ipa-0.kdjf.net krb5kdc[2584](info): AS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), DEPRECATED:arcfour-hmac(23), DEPRECATED:arcfour-hmac-exp(24), DEPRECATED:(-135), DEPRECATED:des-cbc-md5(3)}) 2607:f2c0:f00f:1940:54f9:f7f1:32d4:13aa: CLIENT_NOT_FOUND: vrtoy$@KDJF.NET for krbtgt/KDJF.NET@KDJF.NET, Client not found in Kerberos database
I just checked, and the alias is still on the host in LDAP.
I'm not saying I didn't break something, because I have broken many things in my time... but I don't have any idea what it would be or what I could change to find out. If you have any suggestions, I'm very willing to try them.
From your log, I would guess that your tests use a user alias called "krbalias_user". Might I suggest testing with a host alias ending with a dollar sign? I've definitely had a lot of my programs break because of things like that "shouldn't matter".
Do you know if these aliases are commonly used? It wouldnt't be the first time I've been the only one doing something weird...
We have test_smb.py test that creates SMB server on IPA client. This setup requires cifs/smbserver... principal to have an alias SMBSERVER$. Note that this is not a host but a Kerberos service but from the perspective of the behavior of ipadb_find_principal() all this doesn't matter: host, service, and user principals are looked up exactly the same way and all of them can have aliases, so the code flow is exactly the same.
test_smb.py
cifs/smbserver...
SMBSERVER$
I pushed a pull request to https://github.com/freeipa/freeipa/pull/4609 and set it to run test_smb test suite. Soon we'll know the result.
test_smb
Ok, so finally I was able to create a test that demonstrates that authentication with NetBIOS-based alias works just fine.
The test run is here: http://freeipa-org-pr-ci.s3-website.eu-central-1.amazonaws.com/jobs/4602f436-8b01-11ea-89fb-fa163e43b58c/report.html, in 'test_install_samba' part:
INFO ipatests.pytest_ipa.integration.host.Host.client0.IPAOpenSSHTransport:transport.py:391 RUN ['mktemp'] DEBUG ipatests.pytest_ipa.integration.host.Host.client0.cmd44:transport.py:513 RUN ['mktemp'] DEBUG ipatests.pytest_ipa.integration.host.Host.client0.cmd44:transport.py:558 /tmp/tmp.CCaDKL4nK3 DEBUG ipatests.pytest_ipa.integration.host.Host.client0.cmd44:transport.py:217 Exit code: 0 INFO ipatests.pytest_ipa.integration.host.Host.client0.IPAOpenSSHTransport:transport.py:415 STAT /etc/samba/samba.keytab DEBUG ipatests.pytest_ipa.integration.host.Host.client0.cmd45:transport.py:513 RUN ['ls', '/etc/samba/samba.keytab'] DEBUG ipatests.pytest_ipa.integration.host.Host.client0.cmd45:transport.py:217 Exit code: 0 INFO ipatests.pytest_ipa.integration.host.Host.client0.IPAOpenSSHTransport:transport.py:391 RUN ['/usr/bin/klist', '-eK', '-k', '/etc/samba/samba.keytab'] DEBUG ipatests.pytest_ipa.integration.host.Host.client0.cmd46:transport.py:513 RUN ['/usr/bin/klist', '-eK', '-k', '/etc/samba/samba.keytab'] DEBUG ipatests.pytest_ipa.integration.host.Host.client0.cmd46:transport.py:217 Exit code: 0 INFO ipatests.pytest_ipa.integration.host.Host.client0.IPAOpenSSHTransport:transport.py:415 STAT /tmp/tmp.CCaDKL4nK3.keytab DEBUG ipatests.pytest_ipa.integration.host.Host.client0.cmd47:transport.py:513 RUN ['ls', '/tmp/tmp.CCaDKL4nK3.keytab'] DEBUG ipatests.pytest_ipa.integration.host.Host.client0.cmd47:transport.py:558 ls: cannot access '/tmp/tmp.CCaDKL4nK3.keytab': No such file or directory DEBUG ipatests.pytest_ipa.integration.host.Host.client0.cmd47:transport.py:217 Exit code: 2 INFO ipatests.pytest_ipa.integration.host.Host.client0.IPAOpenSSHTransport:transport.py:391 RUN ['/usr/bin/ktutil'] DEBUG ipatests.pytest_ipa.integration.host.Host.client0.cmd48:transport.py:513 RUN ['/usr/bin/ktutil'] DEBUG ipatests.pytest_ipa.integration.host.Host.client0.cmd48:transport.py:558 rkt: No such file or directory while reading keytab "/tmp/tmp.CCaDKL4nK3.keytab" DEBUG ipatests.pytest_ipa.integration.host.Host.client0.cmd48:transport.py:217 Exit code: 0 INFO ipatests.pytest_ipa.integration.host.Host.client0.IPAOpenSSHTransport:transport.py:391 RUN ['/usr/bin/ktutil'] DEBUG ipatests.pytest_ipa.integration.host.Host.client0.cmd49:transport.py:513 RUN ['/usr/bin/ktutil'] DEBUG ipatests.pytest_ipa.integration.host.Host.client0.cmd49:transport.py:217 Exit code: 0 INFO ipatests.pytest_ipa.integration.host.Host.client0.IPAOpenSSHTransport:transport.py:391 RUN ['kinit', '-kt', '/tmp/tmp.CCaDKL4nK3.keytab', 'CLIENT0$'] DEBUG ipatests.pytest_ipa.integration.host.Host.client0.cmd50:transport.py:513 RUN ['kinit', '-kt', '/tmp/tmp.CCaDKL4nK3.keytab', 'CLIENT0$'] DEBUG ipatests.pytest_ipa.integration.host.Host.client0.cmd50:transport.py:217 Exit code: 0 INFO ipatests.pytest_ipa.integration.host.Host.client0.IPAOpenSSHTransport:transport.py:391 RUN ['rm', '-f', '/tmp/tmp.CCaDKL4nK3.keytab'] DEBUG ipatests.pytest_ipa.integration.host.Host.client0.cmd51:transport.py:513 RUN ['rm', '-f', '/tmp/tmp.CCaDKL4nK3.keytab'] DEBUG ipatests.pytest_ipa.integration.host.Host.client0.cmd51:transport.py:217 Exit code: 0 INFO ipatests.pytest_ipa.integration.host.Host.client0.IPAOpenSSHTransport:transport.py:391 RUN ['rm', '-f', '/tmp/tmp.CCaDKL4nK3'] DEBUG ipatests.pytest_ipa.integration.host.Host.client0.cmd52:transport.py:513 RUN ['rm', '-f', '/tmp/tmp.CCaDKL4nK3'] DEBUG ipatests.pytest_ipa.integration.host.Host.client0.cmd52:transport.py:217 Exit code: 0
cifs/client0.... service is created with CLIENT0$ principal alias (you cannot create that using normal calls, for Samba installation I added a special command ipa service-add-smb in FreeIPA 4.8.2+). To authenticate with it, you need to have a keytab with this principal and the key, so the test code performs a copy of cifs/client0... principal key copy out of /etc/samba/samba.keytab and then stores it in a temporary keytab as a key for CLIENT0$ principal. Finally, it runs kinit -k temporary_keytab CLIENT0$ and authentication passes, from KDC logs (visible in http://freeipa-org-pr-ci.s3-website.eu-central-1.amazonaws.com/jobs/4602f436-8b01-11ea-89fb-fa163e43b58c/test_integration-test_smb.py-TestSMB-test_install_samba/master.ipa.test/var/log/krb5kdc.log.gz):
cifs/client0....
CLIENT0$
ipa service-add-smb
cifs/client0...
/etc/samba/samba.keytab
kinit -k temporary_keytab CLIENT0$
Apr 30 17:36:14 master.ipa.test krb5kdc[25682](info): AS_REQ (7 etypes {aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), DEPRECATED:arcfour-hmac(23), camellia128-cts-cmac(25), camellia256-cts-cmac(26)}) 192.168.122.136: NEEDED_PREAUTH: CLIENT0$@IPA.TEST for krbtgt/IPA.TEST@IPA.TEST, Additional pre-authentication required Apr 30 17:36:14 master.ipa.test krb5kdc[25682](info): closing down fd 11 Apr 30 17:36:14 master.ipa.test krb5kdc[25682](info): AS_REQ (7 etypes {aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), DEPRECATED:arcfour-hmac(23), camellia128-cts-cmac(25), camellia256-cts-cmac(26)}) 192.168.122.136: ISSUE: authtime 1588268174, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18), ses=aes256-cts-hmac-sha1-96(18)}, CLIENT0$@IPA.TEST for krbtgt/IPA.TEST@IPA.TEST Apr 30 17:36:14 master.ipa.test krb5kdc[25682](info): closing down fd 11
When you say "you cannot create that using normal calls", does that mean that you actually can't create it, or that it will be created wrong in some way?
According to my shell script, my aliases were created with "ipa host-add-principal". I don't know what version they were created on, but it was probably a lot older than 4.8.2.
Sorry to confuse you, that was about the test itself.
There is a bit of difference how service alias NetBIOS name can be created compared to the host entry. The resulting LDAP entry would be pretty much equivalent to a host entry like you have, from Kerberos principal point of view as it will have several krbPrincipalName and a single krbCanonicalName but service aliases cannot be single-component principals:
krbPrincipalName
krbCanonicalName
[root@hostname ~]# ipa service-remove-principal cifs/`hostname` HOSTNAME\$ ipa: ERROR: invalid 'principal': Service principal is required
So service alias ending $ works in the test, once the service entry is set up. To debug your particular issue, can you provide logs from
$
$ date $ KRB5_TRACE=/dev/stderr kinit vrtoy\$
and corresponding /var/log/krb5kdc.log and connection logs from /var/log/dirsrv/slapd-*/access for the same period. Make sure to wait a bit as access logs are buffered.
/var/log/krb5kdc.log
/var/log/dirsrv/slapd-*/access
access
The output from that is about what you'd expect:
[799638] 1588450509.92194: Getting initial credentials for vrtoy$@KDJF.NET [799638] 1588450509.92196: Sending unauthenticated request [799638] 1588450509.92197: Sending request (173 bytes) to KDJF.NET [799638] 1588450509.92198: Resolving hostname ipa-0.kdjf.net [799638] 1588450509.92199: Sending initial UDP request to dgram 2607:f2c0:f00f:19c2:3400::2:88 [799638] 1588450509.92200: Received answer (143 bytes) from dgram 2607:f2c0:f00f:19c2:3400::2:88 [799638] 1588450509.92201: Response was from master KDC [799638] 1588450509.92202: Received error from KDC: -1765328378/Client not found in Kerberos database kinit: Client 'vrtoy$@KDJF.NET' not found in Kerberos database while getting initial credentials
I added some printfs, and what's going on here is that the KDC is calling ipa_get_principal with the KRB5_KDB_FLAG_ALIAS_OK flag cleared. And in fact kinit does find the principal if I use "kinit -C 'vrtoy$'" instead of just plain "kinit 'vrtoy$'".
ipa_get_principal
KRB5_KDB_FLAG_ALIAS_OK
kinit -C 'vrtoy$'
So apparently nothing to do with the code we've been talking about. Presumably I have some configuration issue somewhere. I'm not sure where, though.
Thanks for clarifying this point. You need to check your krb5.conf configuration then. On IPA enrolled systems SSSD sets canonicalization in the defaults snippet in `/var/lib/sss/pubconf/krb5.include.d/krb5_libdefaults':
[libdefaults] canonicalize = true
This is controlled with krb5_canonicalize option in the domain section of sssd.conf but that's the default in IPA domains.
krb5_canonicalize
sssd.conf
The snipped is then included via includedir option in /etc/krb5.conf:
includedir
/etc/krb5.conf
includedir /etc/krb5.conf.d/ includedir /var/lib/sss/pubconf/krb5.include.d/ ....
... but the systems with the actual problem are Windows, which I don't think has krb5.conf. The KDC keeps logging that it can't find the principal when the Windows host tries to use it. The kinits are just a test.
I was thinking maybe there was some kind of server side "always check for aliases" setting, but it's not in the documentation that I can see.
Anyway, I'll keep poking at it and come back if I find another actual FreeIPA bug. I don't know if your procedure is to close this issue now, or when the crashing patch actually reaches some stage of integration, so I'll leave it alone and let you do whatever's correct.
Thanks for all your time on this.
Windows systems do always ask for canonicalization. This is part of MS-KILE specification if I am not mistaken.
You probably can gather network traces for the case where Windows attempts to authenticate to IPA KDC and see what flags they do send.
Note that "joining" Windows systems to FreeIPA is not supported, I'm not going to work on bugs uncovered by such behavior unless they also affect non-Windows clients as we know we cannot fulfill those requirements. The crash part is real and affects non-Windows systems, of course.
I'm going to submit a pull request with the fixes we discussed in this ticket and once all backports reach the tree, we'll close it automatically.
Unrelated to the reported bug, but Web searches may bring people to this page: to get a Win10 host to request canonicalization, in addition to what's described at https://www.freeipa.org/page/Windows_authentication_against_FreeIPA , do ksetup /setrealmflags your-realm NcSupported.
This will prevent lookup failures and allow the Windows host to dynamically update its DNS records if FreeIPA is configured to allow such DNS updates. It may even prevent the reported crashes for sites that don't have the patch, but I did not test that.
master:
ipa-4-8:
Metadata Update from @cheimes: - Issue close_status updated to: fixed - Issue status updated to: Closed (was: Open)
Metadata Update from @abbra: - Custom field changelog adjusted to Memory handling in various FreeIPA KDC functions was improved, preventing potential crashes when looking up machine account aliases for Windows machines.
Login to comment on this ticket.