Hi,
I am not able to reproduce the issue with the same packages. Is there anything special about your system (running in FIPS mode, or with a non-default crypto policy?)

I am using nss-3.69.0-1.fc34.x86_64 and openssl-1.1.1k-1.fc34.x86_64.

Hi,
# rpm -q nss openssl
nss-3.69.0-1.fc34.x86_64
openssl-1.1.1k-1.fc34.x86_64

It is a nearly (some tries to install freeip were made) fresh install of fedora 34.

I tried again.
Fresh install Fedora 34
set hostname

timedatectl set-timezone "Europe/Berlin" 

dnf install vim bind-utils freeipa-server freeipa-server-trust-ad samba

ipa-server-install

[16/28]: requesting RA certificate from CA [error] CalledProcessError: CalledProcessError(Command ['/usr/bin/openssl', 'pkcs12', '-nokeys', '-clcerts', '-in', '/root/ca-agent.p12', '-out', '/var/lib/ipa/tmpqufrrmkl', '-passin', 'file:/tmp/tmp2i5z_6gw'] returned non-zero exit status 1: 'Error outputting keys and certificates\n140243523540800:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:crypto/evp/evp_enc.c:643:\n140243523540800:error:23077074:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 cipherfinal error:crypto/pkcs12/p12_decr.c:62:\n140243523540800:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:crypto/pkcs12/p12_decr.c:93:\n') CalledProcessError(Command ['/usr/bin/openssl', 'pkcs12', '-nokeys', '-clcerts', '-in', '/root/ca-agent.p12', '-out', '/var/lib/ipa/tmpqufrrmkl', '-passin', 'file:/tmp/tmp2i5z_6gw'] returned non-zero exit status 1: 'Error outputting keys and certificates\n140243523540800:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:crypto/evp/evp_enc.c:643:\n140243523540800:error:23077074:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 cipherfinal error:crypto/pkcs12/p12_decr.c:62:\n140243523540800:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:crypto/pkcs12/p12_decr.c:93:\n')

The same procedure on Fedora 32 works flawlessly.
It seems there ist a version problem with openssl.
Envelope-Errors occur, when versions do not match.

But:

# rpm -q openssl openssl-libs
openssl-1.1.1k-1.fc34.x86_64
openssl-libs-1.1.1k-1.fc34.x86_64

From the logs:

2021-08-29T13:56:16Z DEBUG   [16/28]: requesting RA certificate from CA
2021-08-29T13:56:16Z DEBUG Starting external process
2021-08-29T13:56:16Z DEBUG args=['/usr/bin/openssl', 'pkcs7', '-inform', 'DER', '-print_certs', '-out', '/var/lib/ipa/tmpl1rxr1wp']
2021-08-29T13:56:16Z DEBUG Process finished, return code=0
2021-08-29T13:56:16Z DEBUG stdout=
2021-08-29T13:56:16Z DEBUG stderr=
2021-08-29T13:56:16Z DEBUG Starting external process
2021-08-29T13:56:16Z DEBUG args=['/usr/bin/openssl', 'pkcs12', '-nokeys', '-clcerts', '-in', '/root/ca-agent.p12', '-out', '/var/lib/ipa/
tmp9ct3d74c', '-passin', 'file:/tmp/tmpo72jz3br']
2021-08-29T13:56:17Z DEBUG Process finished, return code=1
2021-08-29T13:56:17Z DEBUG stdout=
2021-08-29T13:56:17Z DEBUG stderr=Error outputting keys and certificates
140287509026624:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:crypto/evp/evp_enc.c:643:
140287509026624:error:23077074:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 cipherfinal error:crypto/pkcs12/p12_decr.c:62:
140287509026624:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:crypto/pkcs12/p12_decr.c:93:

I am really unable to reproduce this error. The pkcs12 file /root/ca-agent.p12 is generated by Dogtag installer using pk12util command, and read using openssl pkcs12.
Is it possible that another SSL engine is installed on the machine? If pk12util and openssl uses a different engine this may explain the issue, but I'm only guessing.

Do you have any update on this? I'm having the exact same issue.
Also on a fresh install of fedora 34

Edit:
this error also occurs on a newly installed fedora 33 instance.
I can confirm that it works fine on a fedora 32 instance.

The issue looks similar to Bug 1975406 - IPA installation fails during pki-tomcatd setup
It was fixed by pki 11.0.0-beta1, with this commit: https://github.com/dogtagpki/pki/commit/9a37dbf59f80282571245db3ebea11a1eb7daf8a

Can you check if updating pki solves your issue?

We have lost traction on this issue, closing.