#7691 'ipa vault-retrieve' is failing with "ipa: ERROR: an internal error has occurred"
Closed: fixed 8 months ago Opened 10 months ago by abbra.

Ticket was cloned from Red Hat Bugzilla: Bug 1599939

Please note that this Bug is private and may not be accessible as it contains confidential Red Hat customer information.

Description of problem:
'ipa vault-retrieve' is failing with "ipa: ERROR: an internal error has occurred"
Retrieving data from a shared IDM vault as a secondary owner or member doesn't work.

Version-Release number of selected component (if applicable):
ipa-server-4.5.4-10.el7.x86_64

How reproducible:
Always(In customer's environment)

Steps to Reproduce:
1. As 'admin' user, add a shared vault.
2. Archive data into a vault.
3. Add new owner for the vault.
4. 'kinit' as the new owner and try to retrieve data from the vault

Actual results:
'ipa vault-retrieve' is failing with "ipa: ERROR: an internal error has occurred"

Expected results:
'ipa vault-retrieve' should not fail

Additional info:
I tried reproducing this issue with same ipa-server version but couldn't.
This looks like an environment specific or more like a configuration issue.

we could see this error in httpd error_log:
---
HTTPError: 404 Client Error: Not Found
ipa: INFO: [jsonserver_kerb] dmtest2@REALM
vaultconfig_show/1(version=u'2.228'): InternalError
---

Additional information from @abbra:

I think there a couple actions that could be done here:
- IPA would really benefit from handling 404 error when trying to login to KRA. If that happens, we should consider KRA missing and go search for a different server with this role.
- We need to find out why kra_is_enabled() returned True for the server in question as otherwise, we wouldn't try to get to log in to KRA service

A customer then reported that this particular IdM master has no KRA enabled (4 others do) but still we see kra_is_enabled() command reporting True:

[root@idm-admin ~]# ipa console
(Custom IPA interactive Python console)
>>> api.Command.kra_is_enabled()
{u'result': True, u'value': None, u'summary': None}

Metadata Update from @abbra:
- Custom field rhbz adjusted to https://bugzilla.redhat.com/show_bug.cgi?id=1599939

10 months ago

Adding info from BZ:

I see an issue in

    @property
    def kra_host(self):
        """
        :return:   host
                   as str

        Select our KRA host.
        """
        ldap2 = self.api.Backend.ldap2
        if host_has_service(api.env.ca_host, ldap2, "KRA"):
            return api.env.ca_host
        if api.env.host != api.env.ca_host:
            if host_has_service(api.env.host, ldap2, "KRA"):
                return api.env.host
        host = select_any_master(ldap2, "KRA")
        if host:
            return host
        else:
            return api.env.ca_host

Where at the end, when it doesn't find a suitable KRA server it returns a CA host. It might be the reason why it contacts itself. But that doesn't answer why it doesn't find a suitable server when there are 5. It may be another issue.

We need to log exceptions in:

def select_any_master(ldap2, service='CA'):
    """
    :param ldap2: connection to the local database
    :param service: The service for which we're looking for a master.
    :return:   host as str

    Select any host which is a master for a specified service.
    """
    base_dn = DN(('cn', 'masters'), ('cn', 'ipa'), ('cn', 'etc'),
                  api.env.basedn)
    filter_attrs = {
         'objectClass': 'ipaConfigObject',
         'cn': service,
         'ipaConfigString': 'enabledService',}
    query_filter = ldap2.make_filter(filter_attrs, rules='&')
    try:
        ent, _trunc = ldap2.find_entries(filter=query_filter, base_dn=base_dn)
        if len(ent):
            entry = random.choice(ent)
            return entry.dn[1].value
    except Exception:
        pass
    return None

As it probably hide the root cause.

The reproduction steps mention that the vault-retrieve is run by another user. I assume it is a standard user without any special permissions. In this case, the root cause is probably the fact the normal user cannot read Enabled services of IPA masters. IPA API has the same permissions as the user thus a search in select_any_master will return 0 entries. Thus it will wrongly default to master defined by api.env.ca_host which is a current master without KRA.

A workaround is to add permission "System: Read Status of Services on IPA Servers" to users which should be able to use Vault. Or a more specific one to not disclose some information.

@abbra api.Command.kra_is_enabled() will return True for KRA-enabled clusters and False for KRA-less clusters.

kra_is_enabled does:

(...)
        base_dn = DN(('cn', 'masters'), ('cn', 'ipa'), ('cn', 'etc'),
                     self.api.env.basedn)
        filter = '(&(objectClass=ipaConfigObject)(cn=KRA))'
(...)

which builds a non-host specific ldap query like:

# ldapsearch -Y GSSAPI -b cn=masters,cn=ipa,cn=etc,dc=laptop,dc=example,dc=org '(&(objectClass=ipaConfigObject)(cn=KRA))'

which will find KRA hosts if they exist in LDAP, even if the host this is run on is not a KRA master.

Right, so we need to review all code using kra_is_enabled() to see what they expect to receive. If they are checking for the KRA enabled in a cluster, that's one thing. If they expect KRA on the same machine, that's another one. We definitely need to make both queries possible and switch to use the one that is needed in a more explicit way.

So this is indeed an ACI issue.

Changing "pass" from select_any_master(ldap2, service='CA') into:
raise errors.RemoteRetrieveError(reason=_('Unable to determine master for {0}'.format(service)))"
generates as non-admin user:

# ipa vault-retrieve admin_vault --shared --out secret_exported.txt
ipa: ERROR: Unable to determine master for KRA

I tried the following search which is what select_any_master() does as a standard user:

#  ldapsearch -Y GSSAPI -b cn=masters,cn=ipa,cn=etc,dc=laptop,dc=example,dc=org '(&(objectClass=ipaConfigObject)(ipaConfigString=enabledService)(cn=KRA))'
SASL/GSSAPI authentication started
SASL username: testuser@LAPTOP.EXAMPLE.ORG
SASL SSF: 256
SASL data security layer installed.
# extended LDIF
#
# LDAPv3
# base <cn=masters,cn=ipa,cn=etc,dc=laptop,dc=example,dc=org> with scope subtree
# filter: (&(objectClass=ipaConfigObject)(ipaConfigString=enabledService)(cn=KRA))
# requesting: ALL
#

# search result
search: 4
result: 0 Success

# numResponses: 1

yet removing (ipaConfigString=enabledService) returns the only valid KRA server:

#  ldapsearch -Y GSSAPI -b cn=masters,cn=ipa,cn=etc,dc=laptop,dc=example,dc=org '(&(objectClass=ipaConfigObject)(cn=KRA))'
SASL/GSSAPI authentication started
SASL username: testuser@LAPTOP.EXAMPLE.ORG
SASL SSF: 256
SASL data security layer installed.
# extended LDIF
#
# LDAPv3
# base <cn=masters,cn=ipa,cn=etc,dc=laptop,dc=example,dc=org> with scope subtree
# filter: (&(objectClass=ipaConfigObject)(cn=KRA))
# requesting: ALL
#

# KRA, idm0.laptop.example.org, masters, ipa, etc, laptop.example.org
dn: cn=KRA,cn=idm0.laptop.example.org,cn=masters,cn=ipa,cn=etc,dc=laptop,dc=ex
 ample,dc=org
objectClass: nsContainer
objectClass: ipaConfigObject
objectClass: top
cn: KRA

# search result
search: 4
result: 0 Success

# numResponses: 2
# numEntries: 1

Metadata Update from @fcami:
- Custom field on_review adjusted to https://github.com/freeipa/freeipa/pull/2553

8 months ago

Metadata Update from @fcami:
- Issue marked as depending on: #7766

8 months ago

Metadata Update from @fcami:
- Issue assigned to fcami

8 months ago

Will revisit after https://github.com/freeipa/freeipa/pull/2144 is merged.
I'll check the implications of adding "System: Read Status of Services on IPA Servers" to all users. However, as noted above, the information is already available either through a custom LDAP search or by using DNS SRV records.

master:

  • a709da6 Add a shared-vault-retrieve test
  • dd0490e Add a "Find enabled services" ACI in 20-aci.update so that all users can find IPA servers and services. ACI suggested by Christian Heimes.

master:

  • 58053b2 TestBackupAndRestoreWithReplica needs 2 replicas

ipa-4-6:

  • e0207fa Add a shared-vault-retrieve test
  • 212215c Add a "Find enabled services" ACI in 20-aci.update so that all users can find IPA servers and services. ACI suggested by Christian Heimes.

ipa-4-7:

  • 8588591 Add a shared-vault-retrieve test
  • d57d97e Add a "Find enabled services" ACI in 20-aci.update so that all users can find IPA servers and services. ACI suggested by Christian Heimes.

Metadata Update from @fcami:
- Issue unmarked as depending on: #7766

8 months ago

Metadata Update from @frenaud:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

8 months ago

Login to comment on this ticket.

Metadata