It would appear that the server doesn't like the format of the request.

I'd suggest adding three -v flags to the certmonger helper in the CA you defined and run certmonger with -d 3 (or higher).

The certmonger client log is availble at https://pastebin.com/UcDmeUGt . I removed all the cert data as a precaution. I could not find out how to make the enroll request through the helper program.

I also tested the old SSCEP client with the server I am trying to reach and that client was able to enroll successfully. I compared the wireshark captures of certmonger and sscep client requests
- In both cases HTTP GetCA request is successful.
- The PKIOperation request is also successful in both cases (HTTP 200 OK) but in the certmonger case there is no cert returned while in the sscep client the content type is application/x-pki-message.

I have also decoded the message we send with GET PKIOperation (first through URL decode and then PKCS decoder https://certlogik.com/decoder/). The only difference I found out is that if you decode the PKIOperation request from certmonger I see

PKCS#7 Detailed Information
    Certificate:
    Data:
        Version: 1 (0x0)

while the sscep client request shows

PKCS#7 Detailed Information
    Certificate:
    Data:
        Version: 3 (0x2)

Could this be the reason for server returning empty reply to certmonger?

I don't know. The server log you included complains about the ASN.1 format of the request so maybe, but it doesn't seem to be complaining specifically about the PKCS#7 version.

Is there a particular part of the request message that I can look into to see if it is anyway different from the request message that succeeds? I am not familiar with PKCS or ASN.1 standards... Any hints would be appreciated

Update:
I got feedback from the server guys where I am trying to connect. Apparently my request with certmonger is using "DES" which is not allowed by the server and it only supports DES3. How can I set this option?

It depends on the version of certmonger you have. This was fixed upstream in version 0.79.6.

There is no CLI to modify an existing certmonger CA so you'll need to modify the CA file directly. Start by shutting down certmonger.

Find the CA in /var/lib/certmonger/cas/

Edit the file and add:
scep_cipher=DES3

Restart certmonger then resubmit the request.

Adding scep_cipher=DES3 fixed the issue. Now I am able to get x-pki-message (presumably the cert) over HTTP but certmonger fails with new error

2019-05-20 15:39:10 [16165] Request4('TASK1') moved to state 'SUBMITTING'
2019-05-20 15:39:10 [16165] Will revisit Request4('TASK1') on traffic from 14.
2019-05-20 15:39:11 [16165] Certificate submission attempt complete.
2019-05-20 15:39:11 [16165] Child status = 3.
2019-05-20 15:39:11 [16165] Child output:
"Error: failed to verify signature on server response.
"
2019-05-20 15:39:11 [16165] Error: failed to verify signature on server response.
2019-05-20 15:39:11 [16165] Certificate not (yet?) issued.
2019-05-20 15:39:11 [16165] Request4('TASK1') moved to state 'CA_UNREACHABLE'
2019-05-20 15:39:11 [16165] Will revisit Request4('TASK1') in 604800 seconds.

Sounds like doesn't know what CA is signing the response. You may need some combination of -I and -R when creating the SCEP CA. If, for example, you have a Sub CA issuing the certs you'd want to use:

-I /etc/pki/sub.pem -v -R /etc/pki/root.pem

Is this normal for enrollment process? I did not get any file beforehand containing RA certifying chain (-I file) when using sscep client.

So -R is the cert you get when using add-scep-ca command.No problem.
For -I does the CA have to provide it manually through some separate channel?

I can't speak to how sscep validates the signature.

I'm not sure I understand the rest.

-R is only needed if you are doing SCEP over https, it is the chain for the SCEP server itself
-I contains the chain necessary to validate the response

So once again I talked to the scep server people. According to them

"the RootCA in generation V2 and in the new generation V3 the chain e.g. RootCA-SubCA#1 is sent by the PKI if you send the “getca” command.
Sscep then will divide the chain into separate ca.crt-x-files."

So I am already getting the chain using add-scep-ca (GetCA) command.
Is the verification failing because how certmonger handles the chain as compared to sscep client does it?

Right now certmonger requires you to explicitly provide the certs needed for validation.

using the CA cert with -I solved the issue.Thanks

one last question. You mentioned the scep_cipher issue was fixed in 0.79.6.
I used certmonger 0.79.7 but still have to manually edit the CA file to add scep_cipher=DES3 to make the scep request work.
Is it a known issue or just me?

Strange, it should have defaulted to AES256.

If you added the -v's to the helper in the CA config and set the certmonger debug to 3 you should see logging related to the selected cipher and digest like:

SCEP cipher set from configuration to: X
Server supports X, using that.
Could not determine supported CA capabilities, using cipher AES256.
SCEP digest set from configuration to: X
Could not determine supported CA capabilities, using digest SHA256

I started certmonger with
sudo certmonger -n -d 5

Then I used the command
sudo getcert add-ca -c XXX -e"/usr/libexec/certmonger/scep-submit -u XXX -v"

I saw no logging related to the scep cipher/digest in the certmonger window. Also no such information in /var/log/syslog.

But the CA config was correctly available in /var/lib/certmonger/cas with entries like ca_type,ca_external_helper,ca_encryption_cert etc but no scep_cipher.

What does the CA look like? It should be something like:

/usr/libexec/certmonger/scep-submit -vvv http://something ...

Do you mean inside the CA file ?
It is

ca_external_helper=/usr/libexec/certmonger/scep-submit -u http://[removed]/scepv2/default.aspx -vvv

Ok that looks right. You should see the logging in the syslog when you issue a request for a cert using this CA.

In /var/log/syslog I only see this when i add-ca with the above helper (-vvv)

Jun 7 17:28:01 s-VirtualBox scep-submit: GET http://[nameremoved]/scepv2 /default.aspx?operation=GetCACaps
Jun 7 17:28:01 s-VirtualBox scep-submit:
Jun 7 17:28:01 s-VirtualBox scep-submit: GET http://[nameremoved]/scepv2/default.aspx?operation=GetCACert
Jun 7 17:28:01 salman-VirtualBox scep-submit: MIIFeTCCA2GgA[Valid CA Cert HERE]==
Jun 7 17:28:01 s-VirtualBox scep-submit: GET http://[nameremoved]/scepv2/default.aspx?(null)
Jun 7 17:28:01 s-VirtualBox scep-submit: 0�#005y0�#003a�#003#002#001#002#002#001#0010#015#006#011*�H��#015#001#001#013#005

No mention of scep cipher/digest

I'm not sure why it isn't logged. I can try to reproduce.

Thank you... i am using certmonger 0.79.7 ...let me know if you need any more info...

It is working for me in Fedora 29 certmonger-0.79.7-1.fc29.x86_64

Jun 25 14:06:30 ipa certmonger[4817]: 2019-06-25 14:06:30 [4991] Could not determine supported CA capabilities, using cipher AES256.
Jun 25 14:06:30 ipa certmonger[4817]: 2019-06-25 14:06:30 [4991] Could not determine supported CA capabilities, using digest SHA256.
Jun 25 14:06:30 ipa certmonger[4817]: 2019-06-25 14:06:30 [4991] Generating PKCSREQ pkiMessage.

This is the helper configuration:

ca_external_helper=/usr/libexec/certmonger/scep-submit -vvv -u http://ipa.example.test:8080/ca/cgi-bin/pkiclient.exe -I /etc/ipa/ca.crt

/etc/sysconfig/certmonger contains
OPTS=-d3

It is very strange. I tried again on a yocto built firmware with certmonger and again I am unable to get an automatic DES3 added to the CA file from the server. There is no syslog available on it so could not check the log messages. Previously I was using ubuntu 16.04

Maybe I should give it a try on Fedora 29

also is it possible that it can be an issue on the server side and not certmonger ? I don't have any other scep server to test so I can't be sure.