#9002 Nightly failure in test_fips.py::TestInstallFIPS::test_basic::setup
Closed: fixed 2 months ago by frenaud. Opened 3 years ago by frenaud.

The nightly test test_fips.py::TestInstallFIPS::test_basic is failing in rawhide, see PR #1195 with the following logs and report:

cls = <class 'ipatests.test_integration.test_fips.TestInstallFIPS'>
mh = <pytest_multihost.plugin.MultihostFixture object at 0x7f874ff29ff0>

    @classmethod
    def install(cls, mh):
>       super(TestInstallFIPS, cls).install(mh)

test_integration/test_fips.py:28: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test_integration/base.py:83: in install
    cls.enable_fips_mode()
test_integration/base.py:65: in enable_fips_mode
    host.enable_userspace_fips()
pytest_ipa/integration/host.py:106: in enable_userspace_fips
    enable_userspace_fips(self)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

host = <ipatests.pytest_ipa.integration.host.Host master.ipa.test (master)>

    def enable_userspace_fips(host):
        # create /etc/system-fips
        host.put_file_contents(SYSTEM_FIPS, "# userspace fips\n")
        # fake Kernel FIPS mode with bind mount
        host.run_command(["mkdir", "-p", FIPS_OVERLAY_DIR])
        host.put_file_contents(FIPS_OVERLAY, "1\n")
        host.run_command(
            ["chcon", "-t", "sysctl_crypto_t", "-u", "system_u", FIPS_OVERLAY]
        )
        host.run_command(
            ["mount", "--bind", FIPS_OVERLAY, paths.PROC_FIPS_ENABLED]
        )
        # set crypto policy to FIPS mode
        host.run_command(["update-crypto-policies", "--show"])
        host.run_command(["update-crypto-policies", "--set", "FIPS"])
        # sanity check
        assert is_fips_enabled(host)
        result = host.run_command(
            ["openssl", "md5", "/dev/null"], raiseonerr=False
        )
>       assert result.returncode == 1
E       assert 0 == 1
E         +0
E         -1

Test scenario: the test is faking userspace fips mode with commands equivalent to the following:

echo -e "#userspace fips\n" > /etc/system-fips
mkdir -p /var/tmp/userspace-fips
echo -e "1\n" > /var/tmp/userspace-fips/fips-enabled
chcon -t sysctl_crypto_t -u system_u /var/tmp/userspace-fips/fips-enabled
mount --bind /var/tmp/userspace-fips/fips-enabled /proc/sys/crypto/fips_enabled
update-crypto-policies --set FIPS

and then it checks that md5 cannot be used with openssl md5 /dev/null, expecting a result code =1 (error).

Apparently this fake FIPS mode does not disable md5 on rawhide, with openssl-3.0.0-1.fc36.x86_64 (it used to work with openssl-1.1.1k-2.fc35.x86_64).


Metadata Update from @mpolovka:
- Issue assigned to mpolovka

3 years ago

test failure seen in testing_master_latest PR 1734 Report

In RHEL 9 we suggest trying a different approach to userspace FIPS mode emulation in OpenSSL

openssl.cnf like this should work

openssl_conf = openssl_init

# Comment out the next line to ignore configuration errors
config_diagnostics = 1

[openssl_init]
providers = provider_sect
alg_section = algorithm_sect

[provider_sect]
fips = fips_sect
base = base_sect

[base_sect]
activate = 1

[fips_sect]
activate = 1

[algorithm_sect]
default_properties = fips=yes

test failure seen in testing_master_latest PR 1743 Report

Crypto team tells me that openssl 3.x in Fedora has no been yet updated to include FIPS patches on par with RHEL 9. This means we cannot use the same method for 'userspace FIPS' testing as it was with openssl 1.x and as works in RHEL 8/9. An alternative is what @dbelyavs provided -- it would need to be done on top of existing 'fake FIPS' setup. The latter is still needed for the other libraries (gnutls, etc).

An alternative is to run FIPS pipelines against CentOS 9 Stream which has the same openssl version as in RHEL. The downside is that we cannot test and guarantee Fedora in FIPS mode at all.

Reproducible in testing_master_latest PR 1760 Report

Reproducible in testing_master_latest PR 1768 Report

Reproducible in testing_master_latest PR 1778 Report

Reproducible in testing_master_latest PR 1786 Report

Could the issue be cause by a regression in Rawhide's OpenSSL build? RHEL's OpenSSL supports OPENSSL_FORCE_FIPS_MODE=1 env var and /proc/sys/crypto/fips_enabled file. Fedora seems to ignore both the proc file and the env var.

RHEL 9.0

# rpm -qa openssl
openssl-3.0.1-20.el9_0.x86_64
# OPENSSL_FORCE_FIPS_MODE=1 openssl md5 < /dev/null
Error setting digest
000C1650DC7F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:349:Global default library context, Algorithm (MD5 : 97), Properties ()
000C1650DC7F0000:error:03000086:digital envelope routines:evp_md_init_internal:initialization error:crypto/evp/digest.c:237:

Fake FIPS mode:

# echo 1 > /fips_enabled
# mount --bind /fips_enabled /proc/sys/crypto/fips_enabled
# cat /proc/sys/crypto/fips_enabled
1
# openssl md5 < /dev/null
Error setting digest
001C27E02E7F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:349:Global default library context, Algorithm (MD5 : 97), Properties ()
001C27E02E7F0000:error:03000086:digital envelope routines:evp_md_init_internal:initialization error:crypto/evp/digest.c:237

strace

# strace -P /proc/sys/crypto/fips_enabled openssl md5
openat(AT_FDCWD, "/proc/sys/crypto/fips_enabled", O_RDONLY) = 3
read(3, "1\n", 2)                       = 2
close(3)                                = 0
Error setting digest
...

Fedora Rawhide

# rpm -qa openssl
openssl-3.0.3-1.fc37.x86_64
# OPENSSL_FORCE_FIPS_MODE=1 openssl md5 < /dev/null
MD5(stdin)= d41d8cd98f00b204e9800998ecf8427e

Fake fips mode

# echo 1 > /fips_enabled
# mount --bind /fips_enabled /proc/sys/crypto/fips_enabled
# cat /proc/sys/crypto/fips_enabled 
1
# openssl md5 < /dev/null
MD5(stdin)= d41d8cd98f00b204e9800998ecf8427e
# strace -P /proc/sys/crypto/fips_enabled openssl md5 < /dev/null
MD5(stdin)= d41d8cd98f00b204e9800998ecf8427e
+++ exited with 0 +++

Unfortunately, Fedora currently does not support this mode at all. I recommend using the solution from https://pagure.io/freeipa/issue/9002#comment-799339

We have plans to sync Fedora FIPS patches with RHEL, but it will not happen until RHEL patches will be sort of finalized.

reproduced in testing_master_latest_selinux report

Reproducible in testing_master_latest PR 1813 Report

Reproducible in testing_master_pki report

Reproducible in testing_master_latest PR 1821 Report

Reproducible in testing_master_latest PR 1838 Report

Reproducible in testing_master_latest PR 1856 Report

Reproducible in testing_master_latest PR 1866 Report

Reproducible in testing_master_389ds PR 1891 Report

Reproducible in testing_master_latest PR 1887 Report

Reproducible in testing_master_latest PR Report

Reproducible in testing_master_389ds PR 1910 Report

Reproducible in `testing_master_latest PR 1906 Report

Reproducible in testing_master_latest PR 1916 Report

Reproducible in testing_master_latest PR 1927 Report

Reproducible in testing_master_latest PR 1937 Report

Reproducible in testing_master_pki PR 1937 Report

Reproducible in testing_master_latest PR 1965 Report

Reproducible in testing_master_latest PR 1975 Report

Reproducible in testing_master_latest PR 1985 Report

Reproducible in testing_master_latest PR 1995 Report

Reproducible in testing_master_latest PR 2014 Report

Reproducible in testing_master_latest PR 2024 Report

On rawhide, the failure is slightly different since the update of openssl to openssl-3.0.5-3:

        result = host.run_command(
            ["openssl", "md5", "/dev/null"], raiseonerr=False
        )
        assert result.returncode == 1
>       assert "EVP_DigestInit_ex:disabled for FIPS" in result.stderr_text
E       AssertionError: assert 'EVP_DigestInit_ex:disabled for FIPS' in 'Error setting digest\n0022EA66477F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:349:Global default library context, Algorithm (MD5 : 97), Properties ()\n0022EA66477F0000:error:03000086:digital envelope routines:evp_md_init_internal:initialization error:crypto/evp/digest.c:252:\n'
E        +  where 'Error setting digest\n0022EA66477F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:349:Global default library context, Algorithm (MD5 : 97), Properties ()\n0022EA66477F0000:error:03000086:digital envelope routines:evp_md_init_internal:initialization error:crypto/evp/digest.c:252:\n' = <pytest_multihost.transport.SSHCommand object at 0x7f5339df0290>.stderr_text

See PR #2026 with the following report.

This update of openssl brings the same set of patches as on RHEL.

The last error message is completely normal and matching the intended FIPS behavior.

Reproducible in testing_master_latest PR 2034 Report

Reproducible in testing_master_latest PR 2043 Report

Reproducible in testing_master_latest PR 2053 Report

master:

  • b8947b8 ipatests: update the fake fips mode expected message
  • 6bd9d15 cert utilities: MAC verification is incompatible with FIPS mode
  • 17a5d5b FIPS setup: fix typo filtering camellia encryption
  • be21cab Spec file: bump krb5_kdb_version on rawhide

ipa-4-9:

  • 1d01692 ipatests: update the fake fips mode expected message
  • 42381eb cert utilities: MAC verification is incompatible with FIPS mode
  • f2a337c FIPS setup: fix typo filtering camellia encryption
  • f2b4d01 Spec file: bump krb5_kdb_version on rawhide

ipa-4-10:

  • 68f6574 ipatests: update the fake fips mode expected message
  • c853cfd cert utilities: MAC verification is incompatible with FIPS mode
  • dfba6eb FIPS setup: fix typo filtering camellia encryption
  • 2904b15 Spec file: bump krb5_kdb_version on rawhide

Metadata Update from @frenaud:
- Issue tagged with: fips

5 months ago

Metadata Update from @frenaud:
- Custom field on_review adjusted to https://github.com/freeipa/freeipa/pull/7684
- Issue assigned to frenaud (was: mpolovka)

2 months ago

master:

  • 50e8c4a ipatests: simulate FIPS mode and install replica

Metadata Update from @frenaud:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

2 months ago

Log in to comment on this ticket.

Metadata