#5028 FreeIPA project's mirroring hook fails constantly today
Closed: Invalid 3 years ago by pingou. Opened 3 years ago by abbra.

I did a number of pushes today and twice the mirroring hooks failed with:

git push origin bbe990128811c99f777aa04bbd39ea0bbc29611d:master
remote: Has commit access: True        
remote: Has commit access: True        
remote: Sending notification emails to: pvoborni@redhat.com,tdudlak@redhat.com,frenaud@redhat.com,rcritten@redhat.com        
remote: Exception 'UnicodeEncodeError' raised. Please report this as a bug to        
remote: https://github.com/git-multimail/git-multimail/issues        
remote: with the information below:        
remote: 
remote: git-multimail version 1.4.0        
remote: Python version 3.6.8 (default, Aug 18 2020, 08:33:21)         
remote: [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]        
remote: Traceback (most recent call last):        
remote:   File "/usr/lib/python3.6/site-packages/pagure/hooks/files/git_multimail_upstream.py", line 4567, in main        
remote:     run_as_post_receive_hook(environment, mailer)        
remote:   File "/usr/lib/python3.6/site-packages/pagure/hooks/files/git_multimail_upstream.py", line 3898, in run_as_post_receive_hook        
remote:     push.send_emails(mailer, body_filter=environment.filter_body)        
remote:   File "/usr/lib/python3.6/site-packages/pagure/hooks/files/git_multimail_upstream.py", line 3814, in send_emails        
remote:     rev.recipients,        
remote:   File "/usr/lib/python3.6/site-packages/pagure/hooks/files/git_multimail_upstream.py", line 2297, in send        
remote:     self.smtp.sendmail(self.envelopesender, to_addrs, msg)        
remote:   File "/usr/lib64/python3.6/smtplib.py", line 855, in sendmail        
remote:     msg = _fix_eols(msg).encode('ascii')        
remote: UnicodeEncodeError: 'ascii' codec can't encode character '\xe7' in position 1829: ordinal not in range(128)        
remote: 
remote: 
remote: Pushing to github...        
remote: fatal: cannot run /opt/git-ssh-key.sh: No such file or directory        
remote: fatal: cannot run /opt/git-ssh-key.sh: No such file or directory        
remote: fatal: unable to fork        
remote: Push to github finished        
remote: Traceback (most recent call last):        
remote:   File "/usr/lib/python3.6/site-packages/pagure/hooks/__init__.py", line 392, in run_project_hooks        
remote:     changes=changes,        
remote:   File "/usr/lib/python3.6/site-packages/pagure/hooks/__init__.py", line 111, in runhook        
remote:     changes=changes,        
remote:   File "/usr/lib/python3.6/site-packages/pagure/hooks/mail.py", line 135, in post_receive        
remote:     raise Exception("git_multimail failed")        
remote: Exception: git_multimail failed        
remote: Sending to redis to log activity and send commit notification emails        
remote: * Publishing information for 1 commits        
remote:   - to fedora-message        
remote:   - to stomp        
remote:   - to mqtt        
remote:   - to web-hooks        
remote: Running the default hook        
remote: The ticket hook only runs on the ticket git repository.        
remote: git_multimail failed        
remote: Running legacy hook post-receive.ghpush. Please ask your admin to port this to the new plugin format, as the current system will cease functioning in a future Pagure release        
To ssh://pagure.io/freeipa.git
   ca6fc689b..bbe990128  bbe990128811c99f777aa04bbd39ea0bbc29611d -> master
→ 0

This prevents FreeIPA upstream from working as we are dependent on the mirroring to Github for CI and pull request reviews.


Metadata Update from @ngompa:
- Issue assigned to pingou
- Issue priority set to: High
- Issue tagged with: IDM, bug

3 years ago

Let's see if I can find something

Yes, I did manually push the changes from upstream.

Ok so the issue about:

remote: Pushing to github...        
remote: fatal: cannot run /opt/git-ssh-key.sh: No such file or directory        
remote: fatal: cannot run /opt/git-ssh-key.sh: No such file or directory        
remote: fatal: unable to fork      

looks to be coming from the git hook used to mirror to github being a hand-written script that was not saved anywhere...

I've opened an infra ticket asking if we have ways to retrieve that script: https://pagure.io/fedora-infrastructure/issue/9460 but we may not actually need it, you have also the mirroring hook from pagure itself, which should be sufficient/achieve the same.
Could you check that the mirroring still work or which part is not?

I tried to push updates manually for one of pull requests and still got the same issue with github hook:

remote: Pushing to github...        
remote: fatal: cannot run /opt/git-ssh-key.sh: No such file or directory        
remote: fatal: cannot run /opt/git-ssh-key.sh: No such file or directory        
remote: fatal: unable to fork        
remote: Push to github finished        

after that I did manual push to synchronize the trees.

So turns out this was again the service that was not properly running.

I've opened https://pagure.io/fedora-infrastructure/issue/9461 asking that we add some monitoring to the pagure services.

Sorry for the troubles!

Metadata Update from @pingou:
- Issue close_status updated to: Invalid
- Issue status updated to: Closed (was: Open)

3 years ago

Login to comment on this ticket.

Metadata