freeipa

Clone
Source Code
GIT

#8811 Error in the web interface "gecos: value #0 invalid per syntax: Invalid syntax."
Closed: worksforme a year ago by rjeffman. Opened 3 years ago by wanreport.

Closed: worksforme
Reopen Issue

Issue

Error in the web interface when creating a user with First name and Last name in Cyrillic.

Steps to Reproduce

1) go to the web interface https://example.test.net/ipa/ui/ with administrator rights
2) in the Users tab, click the Add button and create a user
, fill in the First name and Last name fields with Russian letters, fill in the remaining fields with random characters
3) click on the Add button and get an error

IPA Error 4208: InvalidSyntax
gecos: value #0 invalid per syntax: Invalid syntax.

1) go to the web interface https://example.test.net/ipa/ui/ with administrator rights
2) in the users tab, go to the settings of any user
3) in the gecos field, add any Russian letter and click on the save button, and an error appears

IPA Error 4208: InvalidSyntax
gecos: value #0 invalid per syntax: Invalid syntax.

Actual behavior

When creating a user in the web interface with First name and Last name in Cyrillic, or adding Russian letters to the user's gecos field, an error appears in the graphical shell

IPA Error 4208: InvalidSyntax
gecos: value #0 invalid per syntax: Invalid syntax.

an error appears in the logs:

[Tue Apr 20 12:04:45.469717 2021] [wsgi:error] [pid 229:tid 436] [remote 172.19.0.1:59648] ipa: INFO: [jsonserver_session] admin@*******.NET: user_add('test2', givenname='\xd1\x82\xd0\xb5\xd1\x81\xd1\x822', sn='\xd1\x82\xd0\xb5\xd1\x81\xd1\x822', userpassword='********', version='2.240'): InvalidSyntax

Version/Release/Distribution

Docker container fedora-34-4.9.3

Additional info:

This error is reproduced in different browsers (Firefox, Chrome, Chromium) on different versions
In FreeIPA, version: 4.8.3, this error is not observed
error_gecos.jpg

According to the log, givenname and sn are seen as bytes on the Python side but they should be a unicode string. Can you enable audit log in 389-ds to see what exactly get passed to the LDAP server?

the audit log in 389-Ds is enabled
when creating the user test6, the following events appeared in the logs

In the audit log

time: 20210421115024
dn: uid=test6,cn=users,cn=accounts,dc=****,dc=****
result: 21
changetype: add
givenName:: 0YLQtdGB0YI=
sn:: 0YLQtdGB0YI2
uid: test6
cn:: 0YLQtdGB0YIg0YLQtdGB0YI2
displayName:: 0YLQtdGB0YIg0YLQtdGB0YI2
initials:: 0YLRgg==
gecos:: 0YLQtdGB0YIg0YLQtdGB0YI2
krbPrincipalName: test6@****.****
objectClass: top
objectClass: person
objectClass: organizationalperson
objectClass: inetorgperson
objectClass: inetuser
objectClass: posixaccount
objectClass: krbprincipalaux
objectClass: krbticketpolicyaux
objectClass: ipaobject
objectClass: ipasshuser
objectClass: ipaSshGroupOfPubKeys
uidNumber: -1
loginShell: /bin/sh
homeDirectory: /home/test6
gidNumber: -1

in the access log

[21/Apr/2021:11:50:24.548459186 +0000] conn=88 op=0 RESULT err=0 tag=97 nentries=0 wtime=0.000442743 optime=0.003372390 etime=0.003812987 dn="uid=admin,cn=users,cn=accounts,dc=****,dc=****"
[21/Apr/2021:11:50:24.553036748 +0000] conn=88 op=1 SRCH base="cn=ipaconfig,cn=etc,dc=****,dc=****" scope=0 filter="(objectClass=*)" attrs=ALL
[21/Apr/2021:11:50:24.553992244 +0000] conn=88 op=1 RESULT err=0 tag=101 nentries=1 wtime=0.000221263 optime=0.000967842 etime=0.001186565
[21/Apr/2021:11:50:24.555322473 +0000] conn=88 op=2 SRCH base="uid=test6,cn=deleted users,cn=accounts,cn=provisioning,dc=****,dc=****" scope=0 filter="(objectClass=*)" attrs=""
[21/Apr/2021:11:50:24.555821637 +0000] conn=88 op=2 RESULT err=32 tag=101 nentries=0 wtime=0.000164415 optime=0.000522496 etime=0.000684501
[21/Apr/2021:11:50:24.556599489 +0000] conn=88 op=3 SRCH base="cn=UPG Definition,cn=Definitions,cn=Managed Entries,cn=etc,dc=****,dc=****" scope=0 filter="(objectClass=*)" attrs="* aci"
[21/Apr/2021:11:50:24.557247292 +0000] conn=88 op=3 RESULT err=0 tag=101 nentries=1 wtime=0.000147714 optime=0.000666211 etime=0.000811740
[21/Apr/2021:11:50:24.557921703 +0000] conn=88 op=4 SRCH base="cn=test6,cn=groups,cn=accounts,dc=****,dc=****" scope=0 filter="(objectClass=*)" attrs=""
[21/Apr/2021:11:50:24.558248292 +0000] conn=88 op=4 RESULT err=32 tag=101 nentries=0 wtime=0.000137159 optime=0.000350295 etime=0.000485107
[21/Apr/2021:11:50:24.561515618 +0000] conn=88 op=5 ADD dn="uid=test6,cn=users,cn=accounts,dc=****,dc=****"
[21/Apr/2021:11:50:24.595424407 +0000] conn=88 op=5 RESULT err=21 tag=105 nentries=0 wtime=0.000235579 optime=0.033980445 etime=0.034212952 - gecos: value #0 invalid per syntax
Edited 2 years ago by wanreport

Thanks.

So cn, sn, displayName, and gecos are all treated as OctetString instead of a DirectoryString, it seems.

Can you tell me how to fix it?

I tried to reproduce it with freeipa-4.9.3-1.fc33 and I failed.

With both IPA CLI and Web UI there is no problem, a user is created and givenname/sn/gecos set correctly when Russian words used for me.

Looks like there might be a problem with locale setting for httpd.service processes. Can you look into the httpd processes' environments and note LC_* or LANG variables set?

These are the parameters
/data/etc/systemd/system/httpd.service.d/ipa.conf 

# Do not edit. Created by IPA installer.

[Service]
Environment=KRB5CCNAME=/tmp/krb5cc-httpd
Environment=GSS_USE_PROXY=yes
Environment=KDCPROXY_CONFIG=/etc/ipa/kdcproxy/kdcproxy.conf
Environment=LC_ALL=C.UTF-8
ExecStartPre=/usr/libexec/ipa/ipa-httpd-kdcproxy

etc/locale.conf

LANG="C.UTF-8"

And what do you see in the environment of the actual httpd processes?

You can use something like this:

# for p in `systemd-cgls  /system.slice/httpd.service | grep FOREGROUND|cut -c3- | cut -d' ' -f1` ; do cat /proc/$p/cmdline && echo ", PID $p:" && (cat /proc/$p/environ | tr '\000' '\n'|grep  -E '(LANG|LC)') && echo; done

/usr/sbin/httpd-DFOREGROUND, PID 531080:
LANG=C
LC_ALL=C.UTF-8

/usr/sbin/httpd-DFOREGROUND, PID 672816:
LANG=C
LC_ALL=C.UTF-8

(wsgi:kdcproxy)-DFOREGROUND, PID 672817:
LANG=C
LC_ALL=C.UTF-8

(wsgi:kdcproxy)-DFOREGROUND, PID 672818:
LANG=C
LC_ALL=C.UTF-8

(wsgi:ipa)     -DFOREGROUND, PID 672819:
LANG=C
LC_ALL=C.UTF-8

(wsgi:ipa)     -DFOREGROUND, PID 672820:
LANG=C
LC_ALL=C.UTF-8

(wsgi:ipa)     -DFOREGROUND, PID 672821:
LANG=C
LC_ALL=C.UTF-8

(wsgi:ipa)     -DFOREGROUND, PID 672822:
LANG=C
LC_ALL=C.UTF-8

/usr/sbin/httpd-DFOREGROUND, PID 672823:
LANG=C
LC_ALL=C.UTF-8

/usr/sbin/httpd-DFOREGROUND, PID 672824:
LANG=C
LC_ALL=C.UTF-8

/usr/sbin/httpd-DFOREGROUND, PID 672825:
LANG=C
LC_ALL=C.UTF-8

/usr/sbin/httpd-DFOREGROUND, PID 673089:
LANG=C
LC_ALL=C.UTF-8

what other information do you need to provide?

/usr/sbin/httpd-DFOREGROUND, PID 1536:
LANG=C
LC_ALL=C.UTF-8

/usr/sbin/httpd-DFOREGROUND, PID 1538:
cat: /proc/1538/environ: Permission denied
(wsgi:kdcproxy)-DFOREGROUND, PID 1539:
cat: /proc/1539/environ: Permission denied
(wsgi:kdcproxy)-DFOREGROUND, PID 1540:
cat: /proc/1540/environ: Permission denied
(wsgi:ipa)     -DFOREGROUND, PID 1541:
cat: /proc/1541/environ: Permission denied
(wsgi:ipa)     -DFOREGROUND, PID 1542:
cat: /proc/1542/environ: Permission denied
(wsgi:ipa)     -DFOREGROUND, PID 1543:
cat: /proc/1543/environ: Permission denied
(wsgi:ipa)     -DFOREGROUND, PID 1544:
cat: /proc/1544/environ: Permission denied
/usr/sbin/httpd-DFOREGROUND, PID 1545:
cat: /proc/1545/environ: Permission denied
/usr/sbin/httpd-DFOREGROUND, PID 1546:
cat: /proc/1546/environ: Permission denied
/usr/sbin/httpd-DFOREGROUND, PID 1547:
cat: /proc/1547/environ: Permission denied
/usr/sbin/httpd-DFOREGROUND, PID 2093:
cat: /proc/2093/environ: Permission denied

What environment is this? Did you run the commands as root?

This is a docker container from the official repository. Built on images freeipa/freeipa-server: fedora-34-4.9.3
I ran the command as root

Ok, so this is not a problem with the code but rather an environmental issue with locale settings in that environment. Nothing we should be fixing in IPA itself but may be you need to review your docker environment.
I'd suggest by comparing locale-specific packages first with a normal Fedora VM. Perhaps some packages are missing and thus locale support is incomplete which trips Python to not treat the resulting environment as UTF-8.

@abbra What are the expected locales? And why would Python code care about the locales in the first place?

This has been brought to freeipa-container repository as https://github.com/freeipa/freeipa-container/issues/398.

I was able to reproduce the problem on host (no containers) on Fedora 34 with freeipa-server-4.9.3-2.fc34.x86_64.

I was also able to reproduce the problem on host Fedora 33 with freeipa-server-4.9.3-1.fc33.x86_64.

On both those hosts, the environment is

LANG=C
LC_ALL=C.UTF-8

They were a fresh install (internal references J:5303458 and J:5303459).

@vstinner, could you please help us with this strange behavior?

@vstinner, could you please help us with this strange behavior?

Hi. I understand that this issue is about the encoding used by FreeIPA to encode/decode data. Which function do you use to encode/decode? Which encoding do you use?

Usually, the most important encoding is: sys.getfilesystemencoding().

We do use sys.getfilesystemencoding() to verify that the server or client side runs in an environment with UTF-8 encoding. If it is not, we don't even start.

When encoding and decoding JSON content we use UTF-8 explicitly with two helpers: json_decode_binary and json_encode_binary: https://pagure.io/freeipa/blob/master/f/ipalib/rpc.py#_439 and https://pagure.io/freeipa/blob/master/f/ipalib/rpc.py#_402

On the server side marshal and unmarshal is done with these two methods: https://pagure.io/freeipa/blob/master/f/ipaserver/rpcserver.py#_520

The binary value from this report would be in options:

[Tue Apr 20 12:04:45.469717 2021] [wsgi:error] [pid 229:tid 436] [remote 172.19.0.1:59648] ipa: INFO: [jsonserver_session] admin@*******.NET: user_add('test2', givenname='\xd1\x82\xd0\xb5\xd1\x81\xd1\x822', sn='\xd1\x82\xd0\xb5\xd1\x81\xd1\x822', userpassword='********', version='2.240'): InvalidSyntax

E.g. givenname and sn are keys in the options dictionary taken out of transformed JSON content:

        options = dict((str(k), v) for (k, v) in options.items())

I was able to reproduce the issue in the Web UI and when adding new users via command line:

[root@hosts ~]# ipa user-add --first="Hélène" --last="TEST" helene.test
ipa: ERROR: gecos: value #0 invalid per syntax : syntaxe invalide.

Locales seem correctly set (i.e. same as above in https://pagure.io/freeipa/issue/8811#comment-728841 and https://pagure.io/freeipa/issue/8811#comment-728851)

I replaced that in the container deployed from the freeipa / freeipa-server: fedora-32 image with freeipa 4.9.3 the error "gecos: value # 0 invalid per syntax: Invalid syntax." absent.

@wanreport could you please detail what did you replace?

I deployed a new container from the freeipa / freeipa-server image: fedora-32.
1) docker pull freeipa / freeipa-server: fedora-32
2) created a container using docker run
3) check:

[root@ipatest /]# ipa user-add
First name: пользователь
Last name: пользователь2
User login [ппользователь2]: user-test
----------------------
Added user "user-test"
----------------------
  User login: user-test
  First name: пользователь
  Last name: пользователь2
  Full name: пользователь пользователь2
  Display name: пользователь пользователь2
  Initials: пп
  Home directory: /home/user-test
  GECOS: пользователь пользователь2
  Login shell: /bin/sh
  Principal name: user-test@********
  Principal alias: user-test@*******
  Email address: user-test@********
  UID: 1601400011
  GID: 1601400011
  Password: False
  Member of groups: ipausers
  Kerberos keys available: False
[root@ipatest /]# kinit admin
Password for admin@*******: 
[root@ipatest /]# ipa -v ping 
-------------------------------------------
IPA server version 4.9.3. API version 2.240
-------------------------------------------
[root@ipatest /]# cat /etc/fedora-release 
Fedora release 32 (Thirty Two)
[root@ipatest /]#

I assume the error "gecos: value # 0 is invalid for syntax: invalid syntax." occurs in containers created from images freeipa / freeipa-server: fedora-34 and freeipa / freeipa-server: fedora-33

According to a discussion on 389-ds list it seems to be something that changed in 389-DS behavior.

I tried to reproduce the issue with:
- Fedora 32 (389-ds-base-1.4.3.23-1.fc32.x86_64)
- Fedora 33 (389-ds-base-1.4.4.15-1.fc33.x86_64)
- Fedora 34 (389-ds-base-2.0.5-2.fc34.x86_64)

The issue is not reproduced on Fedora 32, but is reproduced in Fedora 33 and 34, it might be something added in 1.4.4 series that make into 2.0.

As of RFC 2307, section 3, the 'gecos' field is a IA5String, and should only contain ASCII 7-bit characters, so I'm unsure how (or if) it should be fixed.
Edited 2 years ago by rjeffman

Some sort of transliteration to US-ASCII might be needed, equivalent of

$ echo 'пользователь пользователь2' | LC_CTYPE=POSIX iconv -f utf-8 -t us-ascii//TRANSLIT
pol`zovatel` pol`zovatel`2

@rjeffman in 389-ds gecos is defined as a DirectoryString:

attributeTypes: ( 1.3.6.1.1.1.1.2 NAME 'gecos' DESC 'Standard LDAP attribute type' SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 SINGLE-VALUE X-ORIGIN 'RFC 2307' )

I wrote in one of original comments about that:

So cn, sn, displayName, and gecos are all treated as OctetString instead of a DirectoryString, it seems.

DirectoryString is UTF-8 and it needs no transliteration to US-ASCII.

Metadata Update from @rjeffman:
- Issue assigned to rjeffman
2 years ago

@abbra, the schema on Fedora 32 is, as you mention:

attributeTypes: ( 1.3.6.1.1.1.1.2 NAME 'gecos' DESC 'Standard LDAP attribute t
 ype' SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 SINGLE-VALUE X-ORIGIN 'RFC 2307' )

But for Fedora 33 and 34 I get a different SYNTAX:

attributeTypes: ( 1.3.6.1.1.1.1.2 NAME 'gecos' DESC 'The GECOS field; the comm
 on name' EQUALITY caseIgnoreIA5Match SUBSTR caseIgnoreIA5SubstringsMatch SYNT
 AX 1.3.6.1.4.1.1466.115.121.1.26 SINGLE-VALUE )

The new syntax 1.3.6.1.4.1.1466.115.121.1.26 (IA5String) is consistent with RFC 2307, but it does not allow non-ASCII 7-bit characters.

BTW, this is the search I did:

ldapsearch -X admin -b "cn=schema" -s base -D cn=admin,ou=...,dc=ipa,dc=test objectClasses attributeTypes | grep gecos -A2

I found the root cause: between 1.4.3.4 and 1.4.3.5 389-ds did a consolidation of RFC2307-related schemes into a single one. This consolidation did change gecos and few other attributes' definitions to be more strict.

commit 0683bcde1b667b6d0ca6e8d1ef605f17c51ea2f7
Author: William Brown <william@blackhats.net.au>
Date:   Fri Mar 27 09:48:57 2020 +1000

    Ticket 50933 - rfc2307compat.ldif

    Bug Description: rfc2307 is the original schema for posix and other related
    attributes. rfc2307bis was a draft propsed by a member of the openldap team
    that fixed a number of deficiencies in rfc2307. However, rfc2307bis is not
    completely forward compatible - replacing them may introduce possible data
    errors or other subtle issues.

    In the interest of allowing easier openldap to 389 migrations
    ( https://pagure.io/389-ds-base/issue/50544 ) I propose a rfc2307compat,
    which is a forward compatible version combining rfc2307 and rfc2307bis. This
    would allow items from both to be considered "valid' without changing the
    semantics of either.

    Fix Description: This adds rfc2307compat.ldif, which is a forward compatabile
    expression of both rfc2307 and rfc2307bis, with the knowledge that 389 ds
    does not enforce structural/auxillary rules.

    https://pagure.io/389-ds-base/issue/50933

    Author: William Brown <william@blackhats.net.au>

    Review by: tbordaz (Thanks!)

More details can be found in https://github.com/389ds/389-ds-base/issues/3986

@tbordaz @mreynolds any chance we can revert IA5String to DirectoryString for gecos? In IPA case we probably have thousands of deployments where non-ASCII is used in gecos already.

@abbra I'm not sure we can change it. Changing it will probably cause replication schema conflicts and break replication. @tbordaz can probably confirm this as he's our replication schema expert.

Anyway we had IPA test this patch over a year ago before we merged it, so why is this problem only being found now?

Maybe we need to go back to your old idea of having configurable schema sets?

@mreynolds we didn't have a test that used non-ASCII for gecos/usernames. However, I would argue that this is one of RFC failures that would be best to address with DirectoryString change. 389-ds did have DirectoryString for gecos since 2005 so literally all FreeIPA deployments are using it.
I am not sure about the replication schema conflicts, though. IA5String storage-wise is a subset of DirectoryString -- can we make it compatible in the schema somehow?

When a modified definittion is present in the topology, others instances will try to learn it at the condition the new definition does not break their own data. Their data (gecos) are IA5 so they are also valid as DirectoryString and I would expect new DirectoryString definition to win and to be propagated.
Has this change already been tested ?

When a modified definittion is present in the topology, others instances will try to learn it at the condition the new definition does not break their own data. Their data (gecos) are IA5 so they are also valid as DirectoryString and I would expect new DirectoryString definition to win and to be propagated.

As long as changing it to DirectoryString does not break replication then I am absolutely fine making this change.

Has this change already been tested ?

This must be done first :-)

Metadata Update from @frenaud:
- Custom field rhbz adjusted to https://bugzilla.redhat.com/show_bug.cgi?id=1981281
2 years ago

Issue fixed on 389 ds side: https://github.com/389ds/389-ds-base/issues/4972
The fix is included in 389-ds-base 2.0.11, available on fedora 34+: https://directory.fedoraproject.org/docs/389ds/releases/release-2-0-11.html

@wanreport could you check if 389 ds update works for you?

I've upgraded from fedora 33 to 34, which has 389-ds-base 2.0.12, I'm still seeing the same issue. Does the change work only on newly created users? Are extra steps needed to update existing users?

Can you provide the schema definition for the gecos attribute?

# ldapsearch -x -o ldif-wrap=no -LLL -b cn=schema attributetypes | grep -i gecos
attributetypes: ( 1.3.6.1.1.1.1.2 NAME 'gecos' DESC 'The GECOS field; the common name' EQUALITY caseIgnoreMatch SUBSTR caseIgnoreSubstringsMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 SINGLE-VALUE )

If you see SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 it means the attribute is stored with a DirectoryString syntax which should support non-ASCII characters.
If you see SYNTAX 1.3.6.1.4.1.1466.115.121.1.26, IA5String syntax is used and it means that the update didn't update the schema.

I'm seeing:

$ ldapsearch -x -o ldif-wrap=no -LLL -b cn=schema attributetypes | grep -i gecos
attributetypes: ( 1.3.6.1.1.1.1.2 NAME 'gecos' DESC 'The GECOS field; the common name' EQUALITY caseIgnoreIA5Match SUBSTR caseIgnoreIA5SubstringsMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.26 SINGLE-VALUE X-ORIGIN 'user defined' )

@g5pw , F34 (https://koji.fedoraproject.org/koji/buildinfo?buildID=1867857) reverts the gecos definition to DirectoryString. (you may check that looking at /usr/share/dirsrv/schema/10rfc2307compat.ldif). Does it exists a /etc/dirsrv/slapd-instance/schema/99user.ldif file containing the old definition ?

Does it exists a /etc/dirsrv/slapd-instance/schema/99user.ldif file containing the old definition ?

Yes, it does, however I have no recollection of creating the file manually
Edited 2 years ago by g5pw

Under Fedora 37 with FreeIPA 4.10.1 and 389-DHS 2.2.6 this issue is not reproduced.

Metadata Update from @rjeffman:
- Issue close_status updated to: worksforme
- Issue status updated to: Closed (was: Open)
a year ago

If the resolution is not enough, please reopen the issue with a reproducer for the issue.

Login to comment on this ticket.

Metadata
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
https://bugzilla.redhat.com/show_bug.cgi?id=1981281
None
None
None
Attachments 1
error_gecos.jpg
Attached 3 years ago View Comment
Powered by Pagure 5.13.3
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.