Issue

After rebooting a fresh install of FreeIPA single node install the web ui reports just "Internal Server Error" instead of showing the login page.

Steps to Reproduce

1. Install a Fedora 34 server
2. Install FreeIPA 4.9.3 onto the server with or without dns
3. reboot the server

Actual behavior

web ui reports "Internal Server Error".

Expected behavior

I'd expect to get the login page to login to the web ui.

Version/Release/Distribution

$ rpm -q freeipa-server freeipa-client ipa-server ipa-client 389-ds-base pki-ca krb5-server
freeipa-server-4.9.3-2.fc34.x86_64
freeipa-client-4.9.3-2.fc34.x86_64
package ipa-server is not installed
package ipa-client is not installed
389-ds-base-2.0.3-3.fc34.x86_64
pki-ca-10.10.5-6.fc34.noarch
krb5-server-1.19.1-8.fc34.x86_64

Additional info:

Additionally I was discussing this in Libera Chat #freeipa channel and looked like the /var/lib/ipa/gssproxy/http.keytab permissions were incorrectly apache:apache 0600 until I executed ipa-server-upgrade once on the server.

apache error log notifies:
[Thu Jun 03 10:23:41.370491 2021] [wsgi:error] [pid 1162:tid 1645] [remote 192.168.10.45:49252] ValueError: "Major (458752): No credentials were supplied, or the credentials were unavailable or inaccessible, Minor (2598845123): No credentials cache found
[Thu Jun 03 10:23:42.975701 2021] [:warn] [pid 1164:tid 1352] [client 192.168.10.45:49254] KRB5CCNAME file (/run/ipa/ccaches/admin@DOMAIN.ORG-7ZzV8j) lookup failed!, referer: https://ipa1.domain.org/ipa/ui/
[Thu Jun 03 10:23:42.979048 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254] mod_wsgi (pid=1161): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'.
[Thu Jun 03 10:23:42.979852 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254] Traceback (most recent call last):
[Thu Jun 03 10:23:42.979889 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254] File "/usr/lib/python3.9/site-packages/ipalib/krb_utils.py", line 153, in get_credentials
[Thu Jun 03 10:23:42.979893 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254] return gssapi.Credentials(usage='initiate', name=name, store=store)
[Thu Jun 03 10:23:42.979900 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254] File "/usr/lib64/python3.9/site-packages/gssapi/creds.py", line 63, in new
[Thu Jun 03 10:23:42.979903 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254] res = cls.acquire(name, lifetime, mechs, usage,
[Thu Jun 03 10:23:42.979909 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254] File "/usr/lib64/python3.9/site-packages/gssapi/creds.py", line 146, in acquire
[Thu Jun 03 10:23:42.979911 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254] res = rcred_cred_store.acquire_cred_from(store, name,
[Thu Jun 03 10:23:42.979918 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254] File "gssapi/raw/ext_cred_store.pyx", line 186, in gssapi.raw.ext_cred_store.acquire_cred_from
[Thu Jun 03 10:23:42.979935 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254] gssapi.raw.exceptions.MissingCredentialsError: Major (458752): No credentials were supplied, or the credentials were unavailable or inaccessible, Minor (2598845123): No credentials cache found
[Thu Jun 03 10:23:42.979944 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254]
[Thu Jun 03 10:23:42.979947 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254] During handling of the above exception, another exception occurred:
[Thu Jun 03 10:23:42.979949 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254]
[Thu Jun 03 10:23:42.979955 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254] Traceback (most recent call last):
[Thu Jun 03 10:23:42.979980 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254] File "/usr/share/ipa/wsgi.py", line 59, in application
[Thu Jun 03 10:23:42.979983 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254] return api.Backend.wsgi_dispatch(environ, start_response)
[Thu Jun 03 10:23:42.979989 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254] File "/usr/lib/python3.9/site-packages/ipaserver/rpcserver.py", line 296, in call
[Thu Jun 03 10:23:42.979999 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254] return self.route(environ, start_response)
[Thu Jun 03 10:23:42.980005 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254] File "/usr/lib/python3.9/site-packages/ipaserver/rpcserver.py", line 308, in route
[Thu Jun 03 10:23:42.980008 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254] return app(environ, start_response)
[Thu Jun 03 10:23:42.980013 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254] File "/usr/lib/python3.9/site-packages/ipaserver/rpcserver.py", line 894, in call
[Thu Jun 03 10:23:42.980016 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254] ccache_name = self.get_environ_creds(environ)
[Thu Jun 03 10:23:42.980022 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254] File "/usr/lib/python3.9/site-packages/ipaserver/rpcserver.py", line 672, in get_environ_creds
[Thu Jun 03 10:23:42.980024 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254] creds = get_credentials_if_valid(name=gss_name,
[Thu Jun 03 10:23:42.980030 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254] File "/usr/lib/python3.9/site-packages/ipalib/krb_utils.py", line 199, in get_credentials_if_valid
[Thu Jun 03 10:23:42.980033 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254] creds = get_credentials(name=name, ccache_name=ccache_name)
[Thu Jun 03 10:23:42.980039 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254] File "/usr/lib/python3.9/site-package s/ipalib/krb_utils.py", line 158, in get_credentials
[Thu Jun 03 10:23:42.980041 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254] raise ValueError('"%s", ccache="%s"' % (e, ccache_name))
[Thu Jun 03 10:23:42.980052 2021] [wsgi:error] [pid 1161:tid 1648] [remote 192.168.10.45:49254] ValueError: "Major (458752): No credentials were supplied, or the credentials were unavailable or inaccessible, Minor (2598845123): No credentials cache found

[root@ipa1 ~]# tree /tmp/systemd-private-ab6d7028dea54a96b22f521733fdc030-*
/tmp/systemd-private-ab6d7028dea54a96b22f521733fdc030-chronyd.service-t2ViZm
└── tmp
/tmp/systemd-private-ab6d7028dea54a96b22f521733fdc030-dbus-broker.service-SQZLQE
└── tmp
/tmp/systemd-private-ab6d7028dea54a96b22f521733fdc030-dirsrv@DOMAIN-ORG.service-9ZFHxX
└── tmp
└── slapd-DOMAIN-ORG
├── DOMAIN.ORG20IPA20CA.pem
├── Server-Cert-Key.pem
└── Server-Cert.pem
/tmp/systemd-private-ab6d7028dea54a96b22f521733fdc030-httpd.service-gpK4jG
└── tmp
/tmp/systemd-private-ab6d7028dea54a96b22f521733fdc030-ipa-custodia.service-zFeiqF
└── tmp
/tmp/systemd-private-ab6d7028dea54a96b22f521733fdc030-ipa-dnskeysyncd.service-EcqbSB
└── tmp
├── ipa-dnskeysyncd.ccache
└── ipa-dnskeysync-replica.ccache
/tmp/systemd-private-ab6d7028dea54a96b22f521733fdc030-ModemManager.service-VDkGGw
└── tmp
/tmp/systemd-private-ab6d7028dea54a96b22f521733fdc030-named.service-DmFN1R
└── tmp
/tmp/systemd-private-ab6d7028dea54a96b22f521733fdc030-systemd-logind.service-CZIY9X
└── tmp
/tmp/systemd-private-ab6d7028dea54a96b22f521733fdc030-systemd-oomd.service-b69DCQ
└── tmp
/tmp/systemd-private-ab6d7028dea54a96b22f521733fdc030-systemd-resolved.service-DzO5xv
└── tmp

12 directories, 5 files