fedora-infrastructure

#9376 PDC seems down
Closed: Fixed by pingou. Opened by ankursinha.

Closed: Fixed
Reopen Issue

Describe what you would like us to do:

pdc.fedoraproject.org seems down. We get the "Sorry! This service is unavailable" page. I checked the status page but this doesn't seem to be outage related. So I thought I'd report it here.

This stops bodhi from working:

https://pdc.fedoraproject.org/rest_api/v1/component-branches/?active=true&critical_path=true&fields=global_component&name=f33&page_size=100&type=rpm&global_component=arbor". The status code was "503". The error was

(it didn't complete the error bit)

When do you need this to be done by? (YYYY/MM/DD)

Whenever possible please.

Is this related to pull requests not being mergeable? 

 Your task failed: Remote hook declined the push: 2020-10-06 09:52:36,455 [INFO] pagure.lib.git_auth: Looking for backend: distgit 2020-10-06 09:52:36,455 [INFO] pagure.lib.git_auth: Looking for backend: distgit 2020-10-06 09:52:36,463 [DEBUG] pagure.lib.git_auth: Found the following installed helpers {'test_auth': EntryPoint.parse('test_auth = pagure.lib.git_auth:GitAuthTestHelper'), 'gitolite2': EntryPoint.parse('gitolite2 = pagure.lib.git_auth:Gitolite2Auth'), 'gitolite3': EntryPoint.parse('gitolite3 = pagure.lib.git_auth:Gitolite3Auth'), 'pagure': EntryPoint.parse('pagure = pagure.lib.git_auth:PagureGitAuth'), 'pagure_authorized_keys': EntryPoint.parse('pagure_authorized_keys = pagure.lib.git_auth:PagureGitAuth'), 'distgit': EntryPoint.parse('distgit = dist_git_auth:DistGitAuth')} 2020-10-06 09:52:36,463 [DEBUG] pagure.lib.git_auth: Found the following installed helpers {'test_auth': EntryPoint.parse('test_auth = pagure.lib.git_auth:GitAuthTestHelper'), 'gitolite2': EntryPoint.parse('gitolite2 = pagure.lib.git_auth:Gitolite2Auth'), 'gitolite3': EntryPoint.parse('gitolite3 = pagure.lib.git_auth:Gitolite3Auth'), 'pagure': EntryPoint.parse('pagure = pagure.lib.git_auth:PagureGitAuth'), 'pagure_authorized_keys': EntryPoint.parse('pagure_authorized_keys = pagure.lib.git_auth:PagureGitAuth'), 'distgit': EntryPoint.parse('distgit = dist_git_auth:DistGitAuth')} 2020-10-06 09:52:36,533 [DEBUG] pagure.lib.git_auth: Returning helper <class 'dist_git_auth.DistGitAuth'> from backend key 'distgit' 2020-10-06 09:52:36,533 [DEBUG] pagure.lib.git_auth: Returning helper <class 'dist_git_auth.DistGitAuth'> from backend key 'distgit' 2020-10-06 09:52:36,569 [INFO] pagure_auth: Unspecified ref refs/heads/f33 is blocked Unspecified ref refs/heads/f33 is blocked Denied push for ref 'refs/heads/f33' for user 'churchyard' All changes have been rejected

I saw this issue when submitting update (either through fedpkg or from bodhi web interface), getting errors like

Builds : Unable to create update. Bodhi failed to get a resource from PDC at the following URL "https://pdc.fedoraproject.org/rest_api/v1/component-branches/?active=true&critical_path=true&fields=global_component&name=f33&page_size=100&type=rpm&global_component=openssh". The status code was "503". The error was "".

and

Could not execute update: Could not generate update request: Unable to create update. Bodhi failed to get a resource from PDC at the following URL "https://pdc.fedoraproject.org/rest_api/v1/component-branches/?active=true&critical_path=true&fields=global_component&name=f33&page_size=100&type=rpm&global_component=openssh". The status code was "503". The error was "".

Is this related to pull requests not being mergeable?

yup :(

Cannot run new build...

$ fedpkg build
Could not execute build: The following error occurred while trying to get the active release branches in PDC: 
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
  <head>
...

Update: Investigation as to the root cause is still ongoing

The problem appears to be between the proxies and the pdc server as the pdc server is running and processing direct api requests correctly.

It is still not known what is causing this connection issue.

fedpkg scratch-build works but not fedpkg build.

Also pushing to f33 dist-git branch fails:

$ git push
Total 0 (delta 0), reused 0 (delta 0)
remote: Unspecified ref refs/heads/f33 is blocked
remote: Denied push for ref 'refs/heads/f33' for user 'ppisar'
remote: All changes have been rejected
To ssh://pkgs.fedoraproject.org/rpms/perl-Plack-Test-Agent
 ! [remote rejected] f33 -> f33 (pre-receive hook declined)
error: failed to push some refs to 'ssh://ppisar@pkgs.fedoraproject.org/rpms/perl-Plack-Test-Agent'

Dist-git server probably asks PDC whether a branch is active of blocked.

I've updated status.fedoraproject.org

Alrigth, we found the issue. Haproxy got an incorrect configuration file for PDC. It has been fixed in https://pagure.io/fedora-infra/ansible/c/c0f7fa3e8cd2f603fdc871b39395e95db59fab6b?branch=master and all the proxies have been restarted.

So we should be all back in business now :)

status.fedoraproject.org has been reset for koji and pkgs.

Let us know if you see any more issues and sorry folks for the issue to begin with :(

Metadata Update from @pingou:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

I was able to submit update and build successfully. Looks good now. Thanks!

Works for me as well.

Works,
Thanks
Metadata
None
None
None
None
Lowest
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.