#8493 Synchronize index LDIF and index update files
Closed: fixed 3 years ago by cheimes. Opened 3 years ago by cheimes.

Request for enhancement

FreeIPA has two files that deal with database indices

  • install/share/indices.ldif contains initial index definitions. The file is used very early in server and replica installer, shortly after DS instance is created
  • install/updates/20-indices.update contains index updates. The file is used at the very end of of server and replica installation process as well as by the server updater.

Any index that is defined only in 20-indices.update or modified in 20-indices.update impacts installation process in two ways

  • Since the update indexes are created very late, a missing index may negatively impact any LDAP query that would benefit from the index otherwise.
  • 389-DS has to perform a full DB read to create the new indexes or update modified indexes. This can take a long time.

I assume that it would be more efficient and faster to keep the files in sync so that 20-indices.update does not create any index tasks for ipa-server-install and ipa-replica-install. This may reduce installation time of a large replica, too.

https://pagure.io/freeipa/issue/8491


Perhaps it makes sense to get rid of indices.ldif and define the indices just in the update file? Update LDIF files works as soon as root autobind is configured. PR https://github.com/freeipa/freeipa/pull/5102 adds a convenient wrapper to apply update LDIF files.

Script to dump all indices in ULDIF format:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
#!/usr/bin/python3
import operator

from ipapython.dn import DN
from ipalib import api

INDEX_DN = DN("cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config")
INDEX_FILTER = "(objectClass=nsIndex)"

DEFAULT_ATTRS = ["cn", "objectClass", "nsSystemIndex"]
ADD_ATTRS = ["nsIndexType" "nsMatchingRule"]

def main():
    api.bootstrap(in_server=True)
    api.finalize()
    ldap2 = api.Backend.ldap2
    ldap2.connect()

    entries, truncated = ldap2.find_entries(
        filter=INDEX_FILTER, base_dn=INDEX_DN, attrs_list=["*"]
    )
    for entry in sorted(entries, key=operator.attrgetter("dn")):
        if entry.single_value['nsSystemIndex'] == 'true':
            continue
        print(f"dn: {entry.dn}")
        for name in DEFAULT_ATTRS:
            for value in sorted(entry.get(name, ())):
                print(f"default: {name}: {value}")
        for name in ADD_ATTRS:
            for value in sorted(entry.get(name, ())):
                print(f"only: {name}: {value}")
        if entry.single_value["cn"] == "ipaAnchorUUID":
            print("# see https://pagure.io/freeipa/issue/6975")
            print("remove: cn: ipaOriginalUid")
        print()

if __name__ == "__main__":
    main()

Initial LDIF + update LDIF define a total of 62 unique indices. None of the indices is flagged as system index.

$ grep -h 'cn=index' install/updates/20-indices.update install/updates/20-winsync_index.update install/updates/20-idoverride_index.update install/share/indices.ldif | sort | uniq 
dn: cn=accessRuleType,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=altSecurityIdentities,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=automountkey,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=automountMapName,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=carLicense,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=description,cn=index,cn=userroot,cn=ldbm database,cn=plugins,cn=config
dn: cn=displayname,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=fqdn,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=gidnumber,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=hostCategory,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=idnsName,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=ipaallowedtarget,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=ipaAnchorUUID,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=ipaassignedidview,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=ipaCertmapData,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=ipaConfigString,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=ipaEnabledFlag,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=ipaKrbAuthzData,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=ipakrbprincipalalias,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=ipalocation,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=ipaMemberCa,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=ipaMemberCertProfile,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=ipaNTSecurityIdentifier,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=ipaNTTrustPartner,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=ipaOriginalUid,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=ipasudorunas,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=ipasudorunasgroup,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=ipatokenradiusconfiglink,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=ipauniqueid,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=ipServicePort,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=krbCanonicalName,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=krbPasswordExpiration,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=krbPrincipalName,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=l,cn=index,cn=userroot,cn=ldbm database,cn=plugins,cn=config
dn: cn=macAddress,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=managedby,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=manager,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=memberallowcmd,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=member,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=memberdenycmd,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=memberHost,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=memberManager,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=memberof,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=memberservice,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=memberuid,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=memberUser,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=nsHardwarePlatform,cn=index,cn=userroot,cn=ldbm database,cn=plugins,cn=config
dn: cn=nsHostLocation,cn=index,cn=userroot,cn=ldbm database,cn=plugins,cn=config
dn: cn=nsOsVersion,cn=index,cn=userroot,cn=ldbm database,cn=plugins,cn=config
dn: cn=ntUniqueId,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=ntUserDomainId,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=ou,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=owner,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=secretary,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=seealso,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=serverhostname,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=sourcehost,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=title,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=uid,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=uidnumber,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=uniquemember,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
dn: cn=userCertificate,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config

Metadata Update from @cheimes:
- Custom field on_review adjusted to https://github.com/freeipa/freeipa/pull/5157

3 years ago

master:

ipa-4-8:

  • 1d7b108 Add ldap_update() helper to service class
  • d74bf64 Simplify LDAPUpdater
  • ff3bc4e Use single update LDIF for indices
  • b4b834f Add more indices

Metadata Update from @cheimes:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

3 years ago

Metadata Update from @abbra:
- Custom field changelog adjusted to Configuration of LDAP indices was moved into a single place. New indices were added to attributes related to trusted domains operations. Performance improvement is expected for Kerberos service tickets requested by users from trusted Active Directory domains.

3 years ago

Indices backported to ipa-4-6:
ipa-4-6:

Login to comment on this ticket.

Metadata