#7337 FreeIPA install fails on ARM (64/32) based computers
Closed: insufficientinfo 3 years ago Opened 3 years ago by mandragor2017.

Request for enhancement

As an admin i want to run freeipa on my raspi 3 or nanopi ARM computer with a fedora 27 image on it, which was installed with offical fedora tools

Issue

The installation fails with any option chosen
a) due to problems with slow computers (timeouts)
b) due to an issue with mod_wsgi
c) logfile totally points into a wrong directory

Root cause is that on the serverside the environment variables KRB5CCNAME etc. are not set for the server threads of wsgi:ipa. Not that you can see the environment variables being set in the /proc/<id>environ if you look them up. But for a specific thread starting a process on the server side (like using "ipa env --server") the environment variables are not there in the "environ" variable they use and it ends with a 500 Error and complaing in the /var/log/httpd/error.log that KRBCCNAME is missing.

Nothiing i tried for one week could make it work. I even to hardcode them into the server code, but then another exception occured, which indicates that further parameters are not set correctly.

Maybe the mod_wsgi is broken, but you can see the behaviour as soon as you try to print the environment variables in the httpd error log (they are simply misingl, the same install (4.6.1) works on a normal linux x86_64 machine.
I am giving up after one week of endless tries to make it work on arm.
It all boils down to the mod_wsgi <-> wsgi.py interaction (missing environment variables). Also it seems that the transfer of paramter is somehow wrong ( the content of the POST is used a sever method selection, which indicates for me that there is something serious wrong with the compilation of the C-Code of mod_wsgi).
Tried current fedora mod_wsgi , fedora 28 mod_wsgi, self-compiled mod_wsgi nothing changed the behaviour, again it works fine on x86_64, but neither on arm32 nor on arm64 .

For the installatin itself, it also turns of due to various dbus failures, only patching it to use "wait infinite" in the dbus area, actually made it work on a slow ARM.

Steps to Reproduce

  1. Ty to install on ipa on a fresh fedora 27 installed ARM computer
  2. Any option it always fails
    3.

Actual behavior

  1. Due to high load the install fails early (fixed with the crude patch)
  2. When the patched version runs, the installation fails for the ipa client, which actually is caused by some not working mod_wsgi / server interaction

ipa-server-install\
--no-ntp\
--ssh-trust-dns\
--mkhomedir\
--domain=$MY_DOMAIN\
--realm=$MY_REALM\
--ds-password=$MY_PASSWORD\
--admin-password=$MY_PASSWORD\
--setup-dns\
--auto-reverse\
--reverse-zone=168.192.in-addr.arpa\
--forwarder=$IP1\
--forwarder=$IP2\
--allow-zone-overlap\
--unattended\
-d

Expected behavior

install should work, especially mod_wsgi should work

Version/Release/Distribution

$ rpm -q freeipa-server freeipa-client ipa-server ipa-client 389-ds-base pki-ca krb5-server
freeipa-server-4.6.2-3.fc27.armv7hl
freeipa-client-4.6.2-3.fc27.armv7hl
package ipa-server is not installed
package ipa-client is not installed
389-ds-base-1.3.7.8-1.fc27.armv7hl
pki-ca-10.5.3-1.fc27.noarch
krb5-server-1.15.2-4.fc27.armv7hl

(but you can also use the the 64 bit arm it doesn'tchange anything)
python3-mod_wsgi-4.5.20-2.fc28.armv7hl
... and the regular one and a self compiled one...

Additional info:

Any additional information, configuration, data or log snippets that is needed for reproduction or investigation of the issue.
2017-12-26T13:56:41Z DEBUG Logging to /var/log/ipaclient-install.log
2017-12-26T13:56:41Z DEBUG ipa-client-install was invoked with arguments [] and options: {'unattended': True, 'principal': None, 'prompt_password': False, 'on_master': True, 'ca_cert_files': None, 'no_ac': False, 'force': False, 'configure_firefox': False, 'firefox_dir': None, 'keytab': None, 'mkhomedir': True, 'force_join': False, 'ntp_servers': None, 'no_ntp': False, 'force_ntpd': False, 'nisdomain': None, 'no_nisdomain': False, 'ssh_trust_dns': True, 'no_ssh': False, 'no_sshd': False, 'no_sudo': False, 'no_dns_sshfp': False, 'kinit_attempts': None, 'request_cert': False, 'ip_addresses': None, 'all_ip_addresses': False, 'fixed_primary': False, 'permit': False, 'enable_dns_updates': False, 'no_krb5_offline_passwords': False, 'preserve_sssd': False, 'no_sssd': False, 'automount_location': None, 'domain_name': 'fritz.box', 'servers': ['raspi3.fritz.box'], 'realm_name': 'FRITZ.BOX', 'host_name': 'raspi3.fritz.box', 'verbose': False, 'quiet': False, 'log_file': None, 'uninstall': False}
2017-12-26T13:56:41Z DEBUG IPA version 4.6.2-3.fc27
2017-12-26T13:56:41Z DEBUG Loading Index file from '/var/lib/ipa-client/sysrestore/sysrestore.index'
2017-12-26T13:56:41Z DEBUG Starting external process
2017-12-26T13:56:41Z DEBUG args=/usr/sbin/selinuxenabled
2017-12-26T13:56:41Z DEBUG Process finished, return code=1
2017-12-26T13:56:41Z DEBUG stdout=
2017-12-26T13:56:41Z DEBUG stderr=
2017-12-26T13:56:41Z WARNING Using existing certificate '/etc/ipa/ca.crt'.
2017-12-26T13:56:41Z DEBUG [IPA Discovery]
2017-12-26T13:56:41Z DEBUG Starting IPA discovery with domain=fritz.box, servers=['raspi3.fritz.box'], hostname=raspi3.fritz.box
2017-12-26T13:56:41Z DEBUG Server and domain forced
2017-12-26T13:56:41Z DEBUG [Kerberos realm search]
2017-12-26T13:56:41Z DEBUG Kerberos realm forced
2017-12-26T13:56:41Z DEBUG [LDAP server check]
2017-12-26T13:56:41Z DEBUG Verifying that raspi3.fritz.box (realm FRITZ.BOX) is an IPA server
2017-12-26T13:56:41Z DEBUG Init LDAP connection to: ldap://raspi3.fritz.box:389
2017-12-26T13:56:41Z DEBUG Search LDAP server for IPA base DN
2017-12-26T13:56:41Z DEBUG Check if naming context 'dc=fritz,dc=box' is for IPA
2017-12-26T13:56:42Z DEBUG Naming context 'dc=fritz,dc=box' is a valid IPA context
2017-12-26T13:56:42Z DEBUG Search for (objectClass=krbRealmContainer) in dc=fritz,dc=box (sub)
2017-12-26T13:56:42Z DEBUG Found: cn=FRITZ.BOX,cn=kerberos,dc=fritz,dc=box
2017-12-26T13:56:42Z DEBUG Discovery result: Success; server=raspi3.fritz.box, domain=fritz.box, kdc=raspi3.fritz.box, basedn=dc=fritz,dc=box
2017-12-26T13:56:42Z DEBUG Validated servers: raspi3.fritz.box
2017-12-26T13:56:42Z DEBUG will use discovered domain: fritz.box
2017-12-26T13:56:42Z DEBUG Using servers from command line, disabling DNS discovery
2017-12-26T13:56:42Z DEBUG will use provided server: raspi3.fritz.box
2017-12-26T13:56:42Z DEBUG will use discovered realm: FRITZ.BOX
2017-12-26T13:56:42Z DEBUG will use discovered basedn: dc=fritz,dc=box
2017-12-26T13:56:42Z INFO Client hostname: raspi3.fritz.box
2017-12-26T13:56:42Z DEBUG Hostname source: Provided as option
2017-12-26T13:56:42Z INFO Realm: FRITZ.BOX
2017-12-26T13:56:42Z DEBUG Realm source: Discovered from LDAP DNS records in raspi3.fritz.box
2017-12-26T13:56:42Z INFO DNS Domain: fritz.box
2017-12-26T13:56:42Z DEBUG DNS Domain source: Forced
2017-12-26T13:56:42Z INFO IPA Server: raspi3.fritz.box
2017-12-26T13:56:42Z DEBUG IPA Server source: Provided as option
2017-12-26T13:56:42Z INFO BaseDN: dc=fritz,dc=box
2017-12-26T13:56:42Z DEBUG BaseDN source: From IPA server ldap://raspi3.fritz.box:389
2017-12-26T13:56:42Z DEBUG Loading Index file from '/var/lib/ipa-client/sysrestore/sysrestore.index'
2017-12-26T13:56:42Z DEBUG Loading StateFile from '/var/lib/ipa-client/sysrestore/sysrestore.state'
2017-12-26T13:56:42Z INFO Skipping synchronizing time with NTP server.
2017-12-26T13:56:42Z DEBUG Backing up system configuration file '/etc/sssd/sssd.conf'
2017-12-26T13:56:42Z DEBUG -> Not backing up - '/etc/sssd/sssd.conf' doesn't exist
2017-12-26T13:56:42Z INFO New SSSD config will be created
2017-12-26T13:56:42Z DEBUG Backing up system configuration file '/etc/nsswitch.conf'
2017-12-26T13:56:42Z DEBUG Saving Index File to '/var/lib/ipa-client/sysrestore/sysrestore.index'
2017-12-26T13:56:42Z INFO Configured sudoers in /etc/nsswitch.conf
2017-12-26T13:56:42Z INFO Configured /etc/sssd/sssd.conf
2017-12-26T13:56:42Z DEBUG Initializing principal host/raspi3.fritz.box@FRITZ.BOX using keytab /etc/krb5.keytab
2017-12-26T13:56:42Z DEBUG using ccache /etc/ipa/.dns_ccache
2017-12-26T13:56:42Z DEBUG Attempt 1/5: success
2017-12-26T13:56:43Z DEBUG Starting external process
2017-12-26T13:56:43Z DEBUG args=/usr/bin/certutil -d /tmp/tmp1tnft5qd -N -f /tmp/tmp1tnft5qd/pwdfile.txt -f /tmp/tmp1tnft5qd/pwdfile.txt
2017-12-26T13:56:43Z DEBUG Process finished, return code=0
2017-12-26T13:56:43Z DEBUG stdout=
2017-12-26T13:56:43Z DEBUG stderr=
2017-12-26T13:56:43Z DEBUG Starting external process
2017-12-26T13:56:43Z DEBUG args=/usr/bin/certutil -d /tmp/tmp1tnft5qd -A -n CA certificate 1 -t C,, -a -f /tmp/tmp1tnft5qd/pwdfile.txt
2017-12-26T13:56:43Z DEBUG Process finished, return code=0
2017-12-26T13:56:43Z DEBUG stdout=
2017-12-26T13:56:43Z DEBUG stderr=
2017-12-26T13:56:43Z DEBUG Error reading client session data: 'NoneType' object has no attribute 'decode'
2017-12-26T13:56:43Z DEBUG failed to find session_cookie in persistent storage for principal 'host/raspi3.fritz.box@FRITZ.BOX'
2017-12-26T13:56:44Z INFO trying https://raspi3.fritz.box/ipa/json
2017-12-26T13:56:44Z DEBUG Created connection context.rpcclient_3026908336
2017-12-26T13:56:44Z INFO [try 1]: Forwarding 'schema' to json server 'https://raspi3.fritz.box/ipa/json'
2017-12-26T13:56:44Z DEBUG New HTTP connection (raspi3.fritz.box)
2017-12-26T13:56:45Z DEBUG HTTP connection destroyed (raspi3.fritz.box)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/init.py", line 118, in get_package
plugins = api._remote_plugins
AttributeError: 'API' object has no attribute '_remote_plugins'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 720, in single_request
response.msg)
xmlrpc.client.ProtocolError: <ProtocolError for raspi3.fritz.box/ipa/json: 500 Internal Server Error>
2017-12-26T13:56:46Z DEBUG Destroyed connection context.rpcclient_3026908336
2017-12-26T13:56:46Z DEBUG File "/usr/lib/python3.6/site-packages/ipapython/admintool.py", line 174, in execute
return_value = self.run()
File "/usr/lib/python3.6/site-packages/ipapython/install/cli.py", line 319, in run
cfgr.run()
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 364, in run
self.execute()
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 388, in execute
for _nothing in self._executor():
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 430, in __runner
exc_handler(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 459, in _handle_execute_exception
self._handle_exception(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 449, in _handle_exception
six.reraise(exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 420, in __runner
step()
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 417, in <lambda>
step = lambda: next(self.__gen)
File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
six.reraise(
exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
value = gen.send(prev_value)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 654, in _configure
next(executor)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 430, in __runner
exc_handler(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 459, in _handle_execute_exception
self._handle_exception(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 517, in _handle_exception
self.__parent._handle_exception(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 449, in _handle_exception
six.reraise(exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 514, in _handle_exception
super(ComponentBase, self)._handle_exception(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 449, in _handle_exception
six.reraise(
exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 420, in __runner
step()
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 417, in <lambda>
step = lambda: next(self.__gen)
File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
six.reraise(exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
value = gen.send(prev_value)
File "/usr/lib/python3.6/site-packages/ipapython/install/common.py", line 66, in _install
for _nothing in self._installer(self.parent):
File "/usr/lib/python3.6/site-packages/ipaclient/install/client.py", line 3622, in main
install(self)
File "/usr/lib/python3.6/site-packages/ipaclient/install/client.py", line 2344, in install
_install(options)
File "/usr/lib/python3.6/site-packages/ipaclient/install/client.py", line 2686, in _install
api.finalize()
File "/usr/lib/python3.6/site-packages/ipalib/plugable.py", line 738, in finalize
self.do_if_not_done('load_plugins')
File "/usr/lib/python3.6/site-packages/ipalib/plugable.py", line 425, in __do_if_not_done
getattr(self, name)()
File "/usr/lib/python3.6/site-packages/ipalib/plugable.py", line 618, in load_plugins
for package in self.packages:
File "/usr/lib/python3.6/site-packages/ipalib/__init
.py", line 949, in packages
ipaclient.remote_plugins.get_package(self),
File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/init.py", line 126, in get_package
plugins = schema.get_package(server_info, client)
File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/schema.py", line 547, in get_package
schema = Schema(client)
File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/schema.py", line 395, in init
fingerprint, ttl = self._fetch(client, ignore_cache=read_failed)
File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/schema.py", line 420, in _fetch
schema = client.forward(u'schema',
*kwargs)['result']
File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 1180, in forward
raise NetworkError(uri=server, error=e.errmsg)

2017-12-26T13:56:46Z DEBUG The ipa-client-install command failed, exception: NetworkError: cannot connect to 'https://raspi3.fritz.box/ipa/json': Internal Server Error
2017-12-26T13:56:46Z ERROR cannot connect to 'https://raspi3.fritz.box/ipa/json': Internal Server Error
2017-12-26T13:56:46Z ERROR The ipa-client-install command failed. See /var/log/ipaclient-install.log for more information

Log file locations: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/config-files-logs.html
Troubleshooting guide: https://www.freeipa.org/page/Troubleshooting

ipa.patch


debugged a little bit further, new learning: mod_wsgi is not guily. I replaced mod_wsgi with mod_proxy_uwsg and uwsgi. Same results. but now i at least can provide data, what exactly fails.
It seems that KRBCCNAME becomes garbled in the non-workign result. Remark also the dumps are not complete dumps KRB5CCNAME is nowhere in the non-working request. From what i understand from wsgi, the this VARNAME should exactly be there, where the garbled data shows up.
This garbled data is not produced by mod_wsgi. It comes from httpd (?) or some module there.
I have no idea.

A Working request to uwsgi with as hexdump (on a intel x86_64 box, called freeipa.fritz.box)

00000000 00 68 08 00 09 00 55 4e 49 51 55 45 5f 49 44 1b |.h....UNIQUE_ID.|
00000010 00 57 6b 4e 6f 58 35 74 6a 4e 53 43 46 75 53 53 |.WkNoX5tjNSCFuSS|
00000020 6d 74 41 77 4d 54 41 41 41 41 42 63 0a 00 53 43 |mtAwMTAAAABc..SC|
00000030 52 49 50 54 5f 55 52 4c 09 00 2f 69 70 61 2f 6a |RIPT_URL../ipa/j|
00000040 73 6f 6e 0a 00 53 43 52 49 50 54 5f 55 52 49 22 |son..SCRIPT_URI"|
00000050 00 68 74 74 70 73 3a 2f 2f 66 72 65 65 69 70 61 |.https://freeipa|
00000060 2e 66 72 69 74 7a 2e 62 6f 78 2f 69 70 61 2f 6a |.fritz.box/ipa/j|
00000070 73 6f 6e 08 00 47 53 53 5f 4e 41 4d 45 0f 00 61 |son..GSS_NAME..a|
00000080 64 6d 69 6e 40 46 52 49 54 5a 2e 42 4f 58 16 00 |dmin@FRITZ.BOX..|
00000090 47 53 53 5f 53 45 53 53 49 4f 4e 5f 45 58 50 49 |GSS_SESSION_EXPI|
000000a0 52 41 54 49 4f 4e 0a 00 31 35 31 34 34 35 32 33 |RATION..15144523|
000000b0 31 39 0a 00 4b 52 42 35 43 43 4e 41 4d 45 29 00 |19..KRB5CCNAME).|
000000c0 46 49 4c 45 3a 2f 76 61 72 2f 72 75 6e 2f 69 70 |FILE:/var/run/ip|
000000d0 61 2f 63 63 61 63 68 65 73 2f 61 64 6d 69 6e 40 |a/ccaches/admin@|
000000e0 46 52 49 54 5a 2e 42 4f 58 05 00 48 54 54 50 53 |FRITZ.BOX..HTTPS|
000000f0 02 00 6f 6e 0b 00 53 53 4c 5f 54 4c 53 5f 53 4e |..on..SSL_TLS_SN|

Non working request on an ARM Box (raspi3.fritz.box)

00000000 00 71 08 00 09 00 55 4e 49 51 55 45 5f 49 44 1b |.q....UNIQUE_ID.|
00000010 00 57 6b 4e 34 44 48 43 32 50 75 45 51 75 49 45 |.WkN4DHC2PuEQuIE|
00000020 6f 68 79 49 57 47 51 41 41 41 46 63 0a 00 53 43 |ohyIWGQAAAFc..SC|
00000030 52 49 50 54 5f 55 52 4c 09 00 2f 69 70 61 2f 6a |RIPT_URL../ipa/j|
00000040 73 6f 6e 0a 00 53 43 52 49 50 54 5f 55 52 49 21 |son..SCRIPT_URI!|
00000050 00 68 74 74 70 73 3a 2f 2f 72 61 73 70 69 33 2e |.https://raspi3.|
00000060 66 72 69 74 7a 2e 62 6f 78 2f 69 70 61 2f 6a 73 |fritz.box/ipa/js|
00000070 6f 6e 08 00 47 53 53 5f 4e 41 4d 45 0f 00 61 64 |on..GSS_NAME..ad|
00000080 6d 69 6e 40 46 52 49 54 5a 2e 42 4f 58 16 00 47 |min@FRITZ.BOX..G|
00000090 53 53 5f 53 45 53 53 49 4f 4e 5f 45 58 50 49 52 |SS_SESSION_EXPIR|
000000a0 41 54 49 4f 4e 0a 00 31 35 31 34 34 30 38 37 31 |ATION..151440871|
000000b0 31 1c 00 01 20 a0 e3 06 a0 5a e0 12 1c a0 e1 07 |1... ....Z......|
000000c0 b0 cb e0 32 1e 81 e1 12 23 a0 e1 04 10 8d e5 29 |...2....#......)|
000000d0 00 46 49 4c 45 3a 2f 76 61 72 2f 72 75 6e 2f 69 |.FILE:/var/run/i|
000000e0 70 61 2f 63 63 61 63 68 65 73 2f 61 64 6d 69 6e |pa/ccaches/admin|
000000f0 40 46 52 49 54 5a 2e 42 4f 58 05 00 48 54 54 50 |@FRITZ.BOX..HTTP|

ok found a "workaround" if you set in the file
/etc/httpd/conf.d/ipa.conf
in the section for the "/ipa" Location
GssapiDelegCcacheEnvVar KRB5CCNAME

Then the garbled name is not taken, but the KRB5CCNAME appears...

This is not a fix, but a workaround for an compiler issue

Installs now.

(but other issue (startup of services is wrong certain services do not wait until others are available) --> the whole idea on ARM seems to be very challenging...)

What is your use-case for running IPA on a Pi? It is severely underpowered (as you've seen) for something as heavy-weight as IPA. 2GB is the minimum recommended RAM for IPA.

You mention a compiler issue. Is there a bug filed for that?

Note that timeouts can be mitigated with https://pagure.io/freeipa/issue/6268

I'm not sure what the IPA team should be tracking here other than being aware of the compiler issue.

We are closing this bug because we have not received sufficient information to make progress. Please feel free to open this bug again when you are able to provide the required information we requested.

Metadata Update from @rcritten:
- Issue close_status updated to: insufficientinfo
- Issue status updated to: Closed (was: Open)

3 years ago

Login to comment on this ticket.

Metadata
Attachments 1
Attached 3 years ago View Comment