| |
@@ -0,0 +1,560 @@
|
| |
+ .. highlight:: none
|
| |
+
|
| |
+ Debugging and troubleshooting SSSD
|
| |
+ ==================================
|
| |
+
|
| |
+ This document should help users who are trying to troubleshoot why their SSSD
|
| |
+ setup is not working as expected. After following the steps described here,
|
| |
+ the user should be able to either fix the configuration themselves or provide
|
| |
+ the developers/support a complete set of debug information to follow on
|
| |
+ in a :doc:`bug report <reporting_bugs>` or on the `user support list <https://lists.fedorahosted.org/admin/lists/sssd-users.lists.fedorahosted.org/>`_.
|
| |
+
|
| |
+ The SSSD provides two major features - obtaining information about users
|
| |
+ and authenticating users. Each of these hooks into different system APIs
|
| |
+ and should be viewed separately. However, a successfull authentication can
|
| |
+ only be performed when the information about a user can be retrieved, so if
|
| |
+ authentication doesn't work in your case, please make sure you can at least
|
| |
+ obtain info from about the user with ``getent passwd $user`` and ``id``.
|
| |
+
|
| |
+ Let tools help you!
|
| |
+ -------------------
|
| |
+
|
| |
+ Rather than hand-crafting the SSSD and system configuration yourself, it's
|
| |
+ much wiser to let an automated tool do its job. If you want to connect an
|
| |
+ IPA client, use ``ipa-client-install``. For connecting a machine to an Active
|
| |
+ Directory domain, `realmd <https://freedesktop.org/software/realmd/>`_
|
| |
+ is the best tool for the job. If you're on
|
| |
+ RHEL-6, where realmd is not available, you can still use
|
| |
+ `adcli <https://www.freedesktop.org/software/realmd/adcli/adcli.html>`_.
|
| |
+
|
| |
+ On Fedora or RHEL, the ``authconfig`` utility can also help you set up
|
| |
+ the Name Service Switch and/or the PAM stack while allowing you to use
|
| |
+ a custom sssd.conf with the ``--enablesssd`` and ``--enablesssdauth``
|
| |
+ options. Please note these options only enable SSSD in the NSS and PAM
|
| |
+ stacks but do not configure the SSSD service itself!
|
| |
+
|
| |
+ If you are using a different distribution or operating system, please let
|
| |
+ us know if there are any special instructions to set the system up and
|
| |
+ we'll be glad to either link or include the information.
|
| |
+
|
| |
+ SSSD debug logs
|
| |
+ ---------------
|
| |
+
|
| |
+ Each process that SSSD consists of is represented by a section in the
|
| |
+ ``sssd.conf`` config file. To enable debugging persistently across SSSD service
|
| |
+ restarts, put the directive ``debug_level=N``, where N typically stands for
|
| |
+ a number between 1 and 10 into the particular section. Levels up to 3
|
| |
+ should log mostly failures (although we haven't really been consistent
|
| |
+ especially earlier in the SSSD development) and anything above level 8
|
| |
+ provides a large number of log messages. Level 6 might be a good starting
|
| |
+ point for debugging problems. You can also use the
|
| |
+ `sss_debuglevel(8) <https://jhrozek.fedorapeople.org/sssd/1.14.0/man/sss_debuglevel.8.html>`_
|
| |
+ tool to enable debugging on the fly without having to restart the daemon.
|
| |
+
|
| |
+ On Fedora/RHEL, the debug logs are stored under ``/var/log/sssd``. There
|
| |
+ is one log file per SSSD process. The services (also called responders)
|
| |
+ log into a log file called ``sssd_$service``, for example NSS responder logs
|
| |
+ into ``/var/log/sssd/sssd_nss.log``. The domain sections log into files called
|
| |
+ ``sssd_$domainname.log``. The short-lived helper processes also log into their
|
| |
+ own log files, such as ``ldap_child.log`` or ``krb5_child.log``.
|
| |
+
|
| |
+ Keep in mind that enabling ``debug_level`` in the ``[sssd]`` section only
|
| |
+ enables debugging of the sssd process itself, not all the worker processes!
|
| |
+
|
| |
+ Troubleshooting User Information
|
| |
+ --------------------------------
|
| |
+
|
| |
+ Before diving into the SSSD logs and config files it is very beneficial to know how does the
|
| |
+ `SSSD request flow <https://jhrozek.wordpress.com/2015/03/11/anatomy-of-sssd-user-lookup/>`_
|
| |
+ looks like. Having that in mind, you can go through the following check-list
|
| |
+ to identify where the problem might be.
|
| |
+
|
| |
+ For even more in-depth information on SSSD's architecture, refer to `Pavel Brezina's thesis
|
| |
+ <https://is.muni.cz/th/359290/fi_m/thesis.pdf>`_.
|
| |
+
|
| |
+ Please note the examples of the ``DEBUG`` messages are subject to change
|
| |
+ in future SSSD versions.
|
| |
+
|
| |
+ General tips
|
| |
+ ^^^^^^^^^^^^
|
| |
+
|
| |
+ * To avoid SSSD caching, it is often useful to reproduce the bugs with an
|
| |
+ empty cache or at least invalid cache. However, keep in mind that also
|
| |
+ the cached credentials are stored in the cache! `Do not` remove the cache
|
| |
+ files if your system is offline and it relies on SSSD authentication!
|
| |
+ * Before sending the logs and/or config files to a publicly-accessible
|
| |
+ space, such as mailing lists or bug trackers, check the files for any
|
| |
+ sensitive information.
|
| |
+ * Please only send log files relevant to the occurrence of the issue. Issues
|
| |
+ in log files that are mega- or gigabytes large are more likely to be skipped
|
| |
+ * Unless the problem you're trying to diagnose is related to enumeration
|
| |
+ per se, always reproduce the issue with ``enumerate=false``. Triaging logs
|
| |
+ with enumeration enabled obfuscates the normal data flow..
|
| |
+
|
| |
+ ``getent passwd`` or ``id`` doesn't print the user or ``getent group`` doesn't print the group at all
|
| |
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| |
+ Please follow the usual name-service request flow:
|
| |
+
|
| |
+ 1. Is sssd running at all? On most recent systems, calling::
|
| |
+
|
| |
+ # systemctl status sssd
|
| |
+
|
| |
+ would display the service status. Alternatively, check for the sssd processes with ``ps -ef | grep sssd``.
|
| |
+ or similar.
|
| |
+
|
| |
+ 2. Is the ``sss`` module present in ``/etc/nsswitch.conf`` for all databases?
|
| |
+
|
| |
+ * If there is a separate initgroups database configured, make sure it
|
| |
+ either contains the ``sss`` module as well or comment the ``initgroups``
|
| |
+ line completely
|
| |
+
|
| |
+ 3. Does the request reach the SSSD responder processes? Enable debugging by
|
| |
+ putting ``debug_level=6`` (or higher) into the ``[nss]`` section. Restart
|
| |
+ SSSD and check the nss log for incoming requests with the matching timestamp
|
| |
+ to your getent or id command. Here is how an incoming request looks like
|
| |
+ with SSSD-1.15::
|
| |
+
|
| |
+ [sssd[nss]] [get_client_cred] (0x4000): Client creds: euid[10327] egid[10327] pid[18144].
|
| |
+ [sssd[nss]] [setup_client_idle_timer] (0x4000): Idle timer re-set for client [0x13c9960][22]
|
| |
+ [sssd[nss]] [accept_fd_handler] (0x0400): Client connected!
|
| |
+ [sssd[nss]] [sss_cmd_get_version] (0x0200): Received client version [1].
|
| |
+ [sssd[nss]] [sss_cmd_get_version] (0x0200): Offered version [1].
|
| |
+ [sssd[nss]] [nss_getby_name] (0x0400): Input name: admin
|
| |
+
|
| |
+ 4. If the command is reaching the NSS responder, does it get forwarded to
|
| |
+ the Data Provider? Does the Data Provider request end successfully? With
|
| |
+ SSSD 1.15, an unsuccessful request would look like this::
|
| |
+
|
| |
+ [sssd[nss]] [cache_req_search_dp] (0x0400): CR #3: Looking up [admin@ipa.test] in data provider
|
| |
+ [sssd[nss]] [sss_dp_issue_request] (0x0400): Issuing request for [0x41e51c:1:admin@ipa.test@ipa.test]
|
| |
+ [sssd[nss]] [sss_dp_get_account_msg] (0x0400): Creating request for [ipa.test][0x1][BE_REQ_USER][name=admin@ipa.test:-]
|
| |
+ [sssd[nss]] [sss_dp_internal_get_send] (0x0400): Entering request [0x41e51c:1:admin@ipa.test@ipa.test]
|
| |
+ [sssd[nss]] [sss_dp_req_destructor] (0x0400): Deleting request: [0x41e51c:1:admin@win.trust.test@win.trust.test]
|
| |
+ [sssd[nss]] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.sssd.Error.DataProvider.Offline]
|
| |
+ [sssd[nss]] [cache_req_common_dp_recv] (0x0040): CR #3: Data Provider Error: 3, 5, Failed to get reply from Data Provider
|
| |
+ [sssd[nss]] [cache_req_common_dp_recv] (0x0400): CR #3: Due to an error we will return cached data
|
| |
+
|
| |
+ In contrast, a request that ran into completion would look like this::
|
| |
+
|
| |
+ [sssd[nss]] [cache_req_search_dp] (0x0400): CR #3: Looking up [admin@ipa.test] in data provider
|
| |
+ [sssd[nss]] [sss_dp_issue_request] (0x0400): Issuing request for [0x41e51c:1:admin@ipa.test@ipa.test]
|
| |
+ [sssd[nss]] [sss_dp_get_account_msg] (0x0400): Creating request for [ipa.test][0x1][BE_REQ_USER][name=admin@ipa.test:-]
|
| |
+ [sssd[nss]] [sss_dp_get_reply] (0x1000): Got reply from Data Provider - DP error code: 0 errno: 0 error message: Success
|
| |
+ [sssd[nss]] [cache_req_search_cache] (0x0400): CR #3: Looking up [admin@ipa.test] in cache
|
| |
+
|
| |
+
|
| |
+ If the Data Provider request had finished completely, but you're
|
| |
+ still not seeing any data, then chances are the search didn't match
|
| |
+ any object. Either, way, the next step is to look into the logs from
|
| |
+ the ``[domain]`` section. Put ``debug_level=6`` or higher into the appropriate
|
| |
+ [domain] section, restart SSSD, re-run the lookup and continue debugging
|
| |
+ in the next section.
|
| |
+
|
| |
+ Troubleshooting general ``sssd_be`` problems
|
| |
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| |
+
|
| |
+ * The back end performs several different operations, so it might be
|
| |
+ difficult to see where the problem is at first. At the highest level,
|
| |
+ the back end performs these steps, in this order
|
| |
+
|
| |
+ * The request is received from the responder
|
| |
+ * The back end resolves the server to connect to. This step might
|
| |
+ involve locating the client site or resolving a SRV query
|
| |
+ * The back end establishes connection to the server. In case the
|
| |
+ connection is authenticated, then a proper keytab or a certificate
|
| |
+ might be required
|
| |
+ * Once connection is established, the back end runs the search. You
|
| |
+ should see the LDAP filter, search base and requested attributes.
|
| |
+ * After the search finishes, the entries that matched are stored to
|
| |
+ the cache
|
| |
+ * When the request ends (correctly or not), the status code is returned
|
| |
+ to the responder
|
| |
+
|
| |
+ * Make sure the back end is in "neutral" or "online" state when you run
|
| |
+ the search.
|
| |
+
|
| |
+ * With some responder/provider combinations, SSSD might run a search
|
| |
+ immediately after startup, which, in case of misconfiguration, might mark
|
| |
+ the back end offline even before the first request by the user arrives.
|
| |
+
|
| |
+ * You can forcibly set SSSD into offline or online state
|
| |
+ using the ``SIGUSR1`` and ``SIGUSR2`` signals, see the `sssd(8)
|
| |
+ <https://jhrozek.fedorapeople.org/sssd/1.14.0/man/sssd.8.html>`_ man
|
| |
+ page for details.
|
| |
+
|
| |
+ * Can the remote server be resolved? Check if the DNS servers in ``/etc/resolv.conf`` are correct. With AD or IPA back ends, you generally want them to point to the AD or IPA server directly.
|
| |
+
|
| |
+ * Use the ``dig`` utility to test SRV queries, for instance::
|
| |
+
|
| |
+ dig -t SRV _ldap._tcp.ad.example.com @127.0.0.1
|
| |
+
|
| |
+ * Can the connection be established with the same security properties SSSD uses?
|
| |
+
|
| |
+ * Many back ends require the connection to be authenticated. In case
|
| |
+ of AD and IPA, the connection is authenticated using the system keytab,
|
| |
+ the LDAP back end often uses certificates.
|
| |
+ * ``ldapsearch -ZZ`` is useful to test problems with certificates, since
|
| |
+ SSSD uses openldap libraries under the hood.
|
| |
+ * For debugging GSSAPI authentication, ``kinit`` is useful, often together
|
| |
+ with ``KRB5_TRACE``. Take care to select the correct principal, especially
|
| |
+ with the AD back end. If you select the highest ``debug_level = 10``, then
|
| |
+ ``ldap_child.log`` would contain the Kerberos tracing information as well.
|
| |
+
|
| |
+ * Are the LDAP search properties correct?
|
| |
+
|
| |
+ * Check if all the attributes required by the search are present on
|
| |
+ the server. This is especially important with the AD provider where
|
| |
+ the entries might not contain the POSIX attributes at all or might not
|
| |
+ have the POSIX attributes replicated to Global Catalog, in case SSSD
|
| |
+ is connecting to the GC.
|
| |
+
|
| |
+ * As of SSSD-1.15, try looking for ``DEBUG`` messages from
|
| |
+ ``sdap_get_generic_ext_step``
|
| |
+
|
| |
+ * Is the search base correct, especially with trusted
|
| |
+ subdomains? Incorrect search base with an AD subdomain would yield
|
| |
+ a referral.
|
| |
+
|
| |
+ * Try running the same search with the ldapsearch utility. Don't forget
|
| |
+ to use the same authentication method as SSSD uses! For ``id_provider=ad``
|
| |
+ or ``ipa`` this means adding ``-Y GSSAPI`` to the ``ldapsearch
|
| |
+ invocation``.
|
| |
+
|
| |
+ Assorted common SSSD problems
|
| |
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| |
+
|
| |
+ * Logins take too long or the time to execute ``id $username`` takes too long.
|
| |
+
|
| |
+ * First, make sure to understand `what does id username do
|
| |
+ <https://jhrozek.wordpress.com/2014/01/27/why-is-id-so-slow-with-sssd/>`_.
|
| |
+ Do you really care about its performance? Chances are you're more
|
| |
+ interested in ``id -G`` performance.
|
| |
+ * Check out the ``ignore_group_members`` options in the ``sssd.conf(5)``
|
| |
+ manual page.
|
| |
+ * Some users improved their SSSD performance a lot by mounting the
|
| |
+ cache into ``tmpfs``
|
| |
+
|
| |
+ * ``getent passwd`` and ``getent group`` doesn't display any users or groups.
|
| |
+
|
| |
+ * Enumeration is disabled by design. See the FAQ page for explanation
|
| |
+
|
| |
+ * Changes on the server are not reflected on the client for quite some time
|
| |
+
|
| |
+ * The SSSD caches identity information for some time. You can force
|
| |
+ cache refresh on next lookup using the `sss_cache(8)
|
| |
+ <https://jhrozek.fedorapeople.org/sssd/1.14.0/man/sss_cache.8.html>`_
|
| |
+ tool.
|
| |
+ * Please note that during login, updated information is `always`
|
| |
+ re-read from the server
|
| |
+
|
| |
+ * After enrolling the same machine to a domain with different users
|
| |
+ (perhaps a test VM was enrolled to a newly provisioned server), no users
|
| |
+ can be resolved or log in
|
| |
+
|
| |
+ * Probably the new server has different ID values even if the users are
|
| |
+ named the same (like admin in an IPA domain). Currently UID changes are
|
| |
+ not supported even though `we plan on addressing that eventually
|
| |
+ <https://pagure.io/SSSD/sssd/issue/884>`_. At the moment, caches must
|
| |
+ be removed.
|
| |
+
|
| |
+ Common LDAP provider issues
|
| |
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| |
+
|
| |
+ * When running ``id user``, no groups are displayed.
|
| |
+ * When running ``getent group $groupname``, no group members are displayed
|
| |
+
|
| |
+ * In both cases, make sure the selected schema is correct. By default,
|
| |
+ SSSD will use the more common RFC 2307 schema. The difference between
|
| |
+ RFC 2307 and RFC 2307bis is the way which group membership is stored
|
| |
+ in the LDAP server. In an RFC 2307 server, group members are stored
|
| |
+ as the multi-valued attribute ``memberuid`` which contains the name of
|
| |
+ the users that are members. In an RFC2307bis server, group members are
|
| |
+ stored as the multi-valued attribute ``member`` (or sometimes
|
| |
+ ``uniqueMember``) which contains the DN of the user or group that
|
| |
+ is a member of this group. RFC2307bis allows nested groups to be
|
| |
+ maintained as well.
|
| |
+ * If using the LDAP provider with Active Directory, the back end randomly
|
| |
+ goes offline and performs poorly.
|
| |
+
|
| |
+ * Make sure the referrals are disabled. See the FAQ page for
|
| |
+ explanation. Also please consider migrating to the AD provider. The AD
|
| |
+ provider disabled referral support by default, so there's no need to
|
| |
+ disable referrals explicitly
|
| |
+ * When enumeration is enabled, or when the underlying storage has issues,
|
| |
+ the ``sssd_be`` process is being killed by ``SIGTERM`` or even ``SIGKILL``
|
| |
+
|
| |
+ * With huge directories, the ``sssd_be`` process takes a long time
|
| |
+ to store the entries to cache. The cache writes are blocking, so
|
| |
+ when ``sssd_be`` writes to the cache, it might be considered stuck
|
| |
+ (more on the actual mechanism below) You can increase the heartbeat
|
| |
+ interval by raising the value of the ``timeout`` option.
|
| |
+ * NOTE: The underlying mechanism changed with upstream version 1.14. In
|
| |
+ 1.13 and older, the main ``sssd`` process was sending `pings` to the
|
| |
+ worker processes and was expecting `pongs`. If the target process failed
|
| |
+ to reply three times, it was killed. Starting with upstream version
|
| |
+ 1.14, the ping-pongs were removed in favor of in-process `watchdog`.
|
| |
+ This change has consenquence for the admin in the sense that with
|
| |
+ the older versions, the most useful debug source was the ``sssd.log``
|
| |
+ file, with more recent versions, it's the per-process log file. In both
|
| |
+ cases, raising the ``timeout`` value works.
|
| |
+ * For configuration with ``id_provider=ldap`` and ``auth_provider=ldap``.
|
| |
+ retrieving user information works, but authetication does not
|
| |
+
|
| |
+ * Please note that user authentication is typically retrieved over
|
| |
+ unecrypted channel (unless ``ldap_id_use_start_tls`` is enabled), but
|
| |
+ authentication always happens over an ecrypted channel. Checking for
|
| |
+ certificate errors should be the first step. To test authentication
|
| |
+ manually, you can perform a base-search against the user entry together
|
| |
+ with ldapsearch's ``-Z`` option.
|
| |
+
|
| |
+ Common IPA provider issues
|
| |
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| |
+
|
| |
+ * In an IPA-AD trust setup, ``getent group $groupname`` doesn't display any group members of an AD group
|
| |
+ * In an IPA-AD trust setup, ``id $username`` doesn't display any groups for an AD user
|
| |
+
|
| |
+ * This is expected with very old SSSD and FreeIPA versions. In order to
|
| |
+ display the group members for groups and groups for user, you need to
|
| |
+ have at least SSSD 1.12 on the client and FreeIPA server 4.1 or newer
|
| |
+ at the same time
|
| |
+
|
| |
+ * In an IPA-AD trust setup, IPA users can be resolved, but AD trusted users can't
|
| |
+
|
| |
+ * The IPA client machines query the SSSD instance on the IPA server for AD users. If the client logs contain errors such as::
|
| |
+
|
| |
+ [ipa_s2n_exop_done] (0x0040): ldap_extended_operation result: Operations error(1), Failed to handle the request.
|
| |
+
|
| |
+ or::
|
| |
+
|
| |
+ [ipa_s2n_get_user_done] (0x0040): s2n exop request failed.
|
| |
+
|
| |
+ Check if AD trusted users be resolved on the server at least. Enable
|
| |
+ debugging for the SSSD instance on the IPA server and take a look at
|
| |
+ SSSD logs there. Chances are the SSSD on the server is misconfigured
|
| |
+ or maybe not running at all - make sure that all the requests towards
|
| |
+ the NSS responder can be answered on the server.
|
| |
+
|
| |
+ * In an IPA-AD trust setup, AD trust users cannot be resolved or secondary groups are missing on the IPA server
|
| |
+
|
| |
+ * This can be caused by AD permissions issues if the below errors are seen in the logs::
|
| |
+
|
| |
+ [sdap_ad_resolve_sids_done] (0x0020): Unable to resolve SID S-1-5-21-101891098-1139187330-4192773280-XXXXXX - will try next sid.
|
| |
+
|
| |
+ Validate permissions on the AD object printed in the logs. This can
|
| |
+ checked by manually performing ldapsearch with the same LDAP filter
|
| |
+ and kerberos credentials that SSSD uses(one-way trust uses keytab
|
| |
+ in ``/var/lib/sss/keytabs/`` and two-way trust uses host principal in
|
| |
+ ``/etc/krb5.keytab``).
|
| |
+
|
| |
+ Common AD provider issues
|
| |
+ ^^^^^^^^^^^^^^^^^^^^^^^^^
|
| |
+
|
| |
+ * How do I set up the AD provider?
|
| |
+
|
| |
+ * There is a dedicated page about AD provider setup
|
| |
+
|
| |
+ * Many users can't be displayed at all with ID mapping enabled and SSSD
|
| |
+ domain logs contain error message such as::
|
| |
+
|
| |
+ [sdap_idmap_sid_to_unix] (0x0080): Could not convert objectSID [S-1-5-21-101891098-1139187330-4192773280-XXXXXX]
|
| |
+
|
| |
+ If you are running an old (older than 1.13) version and XXXXXX is a
|
| |
+ number larger than 200000, then check the ``ldap_idmap_range_size``
|
| |
+ option. You'll likely want to increase its value. Keep in mind the
|
| |
+ largest ID value on a POSIX system is 2^32.
|
| |
+
|
| |
+ If you are running a more recent version, check that the
|
| |
+ ``subdomains_provider`` is set to ``ad`` (which is the default). Some
|
| |
+ users are setting the ``subdomains_provider`` to ``none`` to work around
|
| |
+ fail over issues, but this also causes the primary domain SID to be not
|
| |
+ read and therefore cannot map SIDs from the primary domain. Consider using
|
| |
+ the ``ad_enabled_domains`` option instead!
|
| |
+
|
| |
+ * The POSIX attributes disappear randomly after login
|
| |
+
|
| |
+ * SSSD looks the user's group membership in the Global Catalog to make
|
| |
+ sure even the cross-domain memberships are taken into account. Chances
|
| |
+ are the POSIX attributes are not replicated to the Global Catalog. You
|
| |
+ can disable the Global catalog lookups by disabling the ``ad_enable_gc``
|
| |
+ option, but you'll lose cross-domain memberships. Alternatively, modify
|
| |
+ the AD schema to replicate the POSIX attribute to the Global Catalog.
|
| |
+
|
| |
+ * After selecting a custom ``ldap_search_base``, the group membership no
|
| |
+ longer displays correctly.
|
| |
+
|
| |
+ * If you use a non-standard LDAP search bases, please
|
| |
+ disable the TokenGroups performance enhancement by setting
|
| |
+ ``ldap_use_tokengroups=False``. Otherwise, the AD provider would receive
|
| |
+ the group membership via a special call that is not restricted by the
|
| |
+ custom search base which causes unpredictable results
|
| |
+
|
| |
+ * Typically, users configure a custom ``ldap_search_base`` to limit the
|
| |
+ groups the user is a member of. Please see
|
| |
+ `this blog post <https://jhrozek.wordpress.com/2016/12/09/restrict-the-set-of-groups-the-user-is-a-member-of-with-sssd/>`_
|
| |
+ for more information on the subject.
|
| |
+
|
| |
+ * SSSD keeps connecting to a trusted domain that is not reachable
|
| |
+ and the whole daemon switches to offline mode as a result
|
| |
+
|
| |
+ * SSSD would connect to the forest root in order to discover all
|
| |
+ subdomains in the forest in case the SSSD client is enrolled with a member
|
| |
+ of the forest, not the forest root. This is because only the forest root
|
| |
+ knows all the subdomains, the forest member only knows about itself and
|
| |
+ the forest root. Also, SSSD by default tries to resolve all groups
|
| |
+ the user is a member of, from all domains. In case the SSSD client
|
| |
+ is behind a firewall preventing connection to a trusted domain,
|
| |
+ can set the ``ad_enabled_domains`` option to selectively enable only
|
| |
+ the reachable domains.
|
| |
+
|
| |
+ * A group my user is a member of doesn't display in the ``id`` output
|
| |
+
|
| |
+ * Cases like this are best debugged from an empty cache. Check if the
|
| |
+ group GID appears in the output of ``id -G`` first. In case the group is
|
| |
+ not present in the ``id -G`` output at all, there is something up with the
|
| |
+ initgroups part. Check the schema and look for anything strange during
|
| |
+ the initgr operation in SSSD back end logs. If the group is present
|
| |
+ in ``id -G`` output but not in ``id`` output (or a subsequent id output) then
|
| |
+ there's something wrong with resolving the group GIDs with ``getgrgid()``.
|
| |
+
|
| |
+ Troubleshooting Authentication, Password Change and Access Control
|
| |
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| |
+
|
| |
+ In order for authentication to be successfull, the user information must
|
| |
+ be accurately provided first. Before debugging authentication, please
|
| |
+ make sure the user information is resolvable with ``getent passwd $user`` or
|
| |
+ ``id $user``. Failing to retrieve the user info would also manifest in the
|
| |
+ secure logs or the journal with message such as::
|
| |
+
|
| |
+ pam_sss(sshd:account): Access denied for user admin: 10 (User not known to the underlying authentication module)
|
| |
+
|
| |
+ Authentication happens from PAM's ``auth`` stack and corresponds to SSSD's
|
| |
+ ``auth_provider``. Access control takes place in PAM ``account`` phase and
|
| |
+ is linked with SSSD's ``access_provider``. And lastly, password changes go
|
| |
+ through the ``password`` stack on the PAM side to SSSD's ``chpass_provider``.
|
| |
+
|
| |
+ If the user info can be retrieved, but authentication fails, the first place
|
| |
+ to look into is ``/var/log/secure`` or the system journal. Look for messages
|
| |
+ from ``pam_sss``. Please note that not all authentication requests come
|
| |
+ through SSSD. Notably, SSH key authentication and GSSAPI SSH authentication
|
| |
+ happen directly in SSHD and SSSD is only contacted for the ``account`` phase.
|
| |
+
|
| |
+ Troubleshooting general authentication problems
|
| |
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| |
+
|
| |
+ The PAM authentication flow follows this pattern:
|
| |
+
|
| |
+ 1. The PAM-aware application starts the PAM conversation. Depending on the
|
| |
+ PAM stack configuration, the ``pam_sss`` module would be contacted. To
|
| |
+ debug the authentication process, first check in the secure log or journal
|
| |
+ if ``pam_sss`` is called at all. If you don't see ``pam_sss`` mentioned,
|
| |
+ chances are your PAM stack is misconfigured. If you see ``pam_sss`` being
|
| |
+ contacted, enable debugging in pam responder logs
|
| |
+
|
| |
+ 2. SSSD's PAM responder receives the authentication request and in most
|
| |
+ cases forwards it to the back end. Please note that unlike identity
|
| |
+ requests, the authentication/access control is typically not cached and
|
| |
+ always contacts the server. This might manifest as a slowdown in some
|
| |
+ cases, but it's quite important, because the supplementary groups
|
| |
+ in Linux are only set during login time.
|
| |
+
|
| |
+ * The PAM responder logs should show the request being received from
|
| |
+ the pam stack and then forwarded to the back end.
|
| |
+ * If you see the authentication request getting to the PAM responder,
|
| |
+ but receiving an error from the back end, check the back end logs. An
|
| |
+ example error output might look like::
|
| |
+
|
| |
+ [sssd[pam]] [sss_dp_issue_request] (0x0400): Issuing request for [0x411d44:3:admin@ipa.example.com]
|
| |
+ [sssd[pam]] [sss_dp_get_account_msg] (0x0400): Creating request for [ipa.example.com][3][1][name=admin]
|
| |
+ [sssd[pam]] [sss_dp_internal_get_send] (0x0400): Entering request [0x411d44:3:admin@ipa.example.com]
|
| |
+ [sssd[pam]] [sss_dp_get_reply] (0x1000): Got reply from Data Provider - DP error code: 1 errno: 11 error message: Offline
|
| |
+ [sssd[pam]] [pam_check_user_dp_callback] (0x0040): Unable to get information from Data Provider Error: 1, 11, Offline
|
| |
+
|
| |
+ 3. The back end processes the request. This might include the equivalent
|
| |
+ of ``kinit`` done in the ``krb5_child`` process, an LDAP bind or
|
| |
+ consulting an access control list. After the back end request finishes,
|
| |
+ the result is sent back to the PAM responder.
|
| |
+
|
| |
+ * For Kerberos-based (that includes the IPA and AD providers)
|
| |
+ ``auth_provider``, look into the ``krb5_child.log`` file as
|
| |
+ well. Setting ``debug_level`` to 10 would also enable low-level
|
| |
+ Kerberos tracing information in that logfile. You can also simulate
|
| |
+ the authentication with ``kinit``.
|
| |
+ * If the back end's ``auth_provider`` is LDAP-based, you can simulate
|
| |
+ the authentication by performing a base-scoped bind as the user who
|
| |
+ is logging in::
|
| |
+
|
| |
+ ldapsearch -x -H ldap://master.ipa.example.com -b uid=admin,cn=users,cn=accounts,dc=ipa,dc=example,dc=com -s base -W
|
| |
+
|
| |
+ 4. The PAM responder receives the result and forwards it back to
|
| |
+ the ``pam_sss`` module. The error or status message is displayed in
|
| |
+ ``/var/log/secure`` or journal.
|
| |
+
|
| |
+
|
| |
+ Assorted common SSSD authentication problems
|
| |
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| |
+
|
| |
+ * How do I enable LDAP authentication over an unsecure connection?
|
| |
+
|
| |
+ * Not possible, sorry. SSSD requires the use of either TLS or LDAPS
|
| |
+ for LDAP authentication. Perimeter security is just not enough.
|
| |
+
|
| |
+ * There are no messages from ``pam_sss`` at all
|
| |
+
|
| |
+ * Your PAM stack is likely misconfigured. Use the ``authconfig`` tool
|
| |
+ if available together with ``--enablesssdauth``.
|
| |
+ * Alternatively, check that the authentication you are using is PAM-aware,
|
| |
+ because some authentication methods, like SSH public keys are handled
|
| |
+ directly in the SSHD and do not use PAM at all.
|
| |
+
|
| |
+ * I can ``su`` to an SSSD user from root, but not from a regular user, SSH doesn't work either
|
| |
+
|
| |
+ * If you su to another user from root, you typically bypass SSSD
|
| |
+ authentication completely by using the ``pam_rootok.so`` module. Your
|
| |
+ SSSD setup is likely broken, please log in as an ordinary user and
|
| |
+ continue debugging in this section
|
| |
+
|
| |
+ * I'm receiving ``System Error (4)`` in the authentication logs
|
| |
+
|
| |
+ * System Error is an "Unhandled Exception" during authentication. It can
|
| |
+ either be an SSSD bug or a fatal error during authentication. Either way,
|
| |
+ please bring up your issue on the
|
| |
+ `sssd-users mailing list <https://lists.fedorahosted.org/admin/lists/sssd-users.lists.fedorahosted.org/>`_
|
| |
+
|
| |
+ * I'm receiving ``Access denied for user $user: 6 (Permission denied)``
|
| |
+
|
| |
+ * Authentication went fine, but the user was denied access to the
|
| |
+ client machine. You can temporarily disable access control with setting
|
| |
+ ``access_provider=permit`` temporarily. Don't forget to reset the access
|
| |
+ provider to a stricter setting after finding out the root cause!
|
| |
+ * If disabling access control doesn't help, the account might be locked
|
| |
+ on the server side. Check the SSSD domain logs to find out more.
|
| |
+
|
| |
+ * I can't get my LDAP-based access control filter right for group
|
| |
+ access control using the memberOf attribute
|
| |
+
|
| |
+ * The LDAP-based access control is really tricky to get right and
|
| |
+ doesn't typically handle nested groups well. Use the `simple access
|
| |
+ provider
|
| |
+ <https://jhrozek.fedorapeople.org/sssd/1.14.0/man/sssd-simple.5.html>`_
|
| |
+ instead.
|
| |
+
|
| |
+ Common IPA provider issues
|
| |
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| |
+
|
| |
+ * In an IPA-AD trust setup, IPA users can log in, but AD users can't
|
| |
+
|
| |
+ * Unless you use a "legacy client" such as ``nss_ldap``, then IPA users authenticate against the IPA server, but AD users authenticate against the AD servers directly. Therefore, you can test the authentication directly with ``kinit``. Use ``KRB5_TRACE`` for extra tracing information.
|
| |
+
|
| |
+ * HBAC rules keep denying access.
|
| |
+
|
| |
+ * Use the ``ipa hbactest`` utility on the IPA server to see if the user is permitted access. The SSSD uses the same code as ``ipa hbactest``
|
| |
+
|
| |
+ * HBAC prevents access for a user from a trusted AD domain, where the HBAC rule is mapped to an IPA group via an AD group
|
| |
+
|
| |
+ * Make sure the group scope of the AD group mapped to the rule is not
|
| |
+ `domain-local`. Domain-local groups can't cross the trust boundary and
|
| |
+ cannot be therefore used for HBAC rules.
|
| |
+ * There was an irritating `SSSD bug
|
| |
+ <https://pagure.io/SSSD/sssd/issue/3382>`_ that manifested as SSSD not
|
| |
+ listing some of the groups in the SSSD logs during the HBAC check. Make
|
| |
+ sure you are running an SSSD version that includes the fix.
|
| |
