#4384 Pagure creates repo in DB/UI even if it wasn't able to really init the repo in repospanner
Closed: Fixed 4 years ago by pingou. Opened 5 years ago by arrfab.

Issue description

When creating a repo through pagure (webui or api call) that should be created in repospanner region, but that repospanner itself is down/unreachable, pagure still creates the repo in DB and so visible in WebUI.
That means that there is no way to even force to create it correctly when repospanner is down, and the task to create the repo in pagure worker isn't kept.
That means that there is currently no way to heal that state ?

How to reproduce

  • disable repospanner
  • create repo in pagure (webui or api, doesn't make a diff)
  • repo is created in pagure but not in repospanner
  • pagure goes 500 for each request and so impossible do anything with that repo

Example :

Mar 27 12:27:14 git.dev.lon1.centos.org celery[382]: Traceback (most recent call last):
Mar 27 12:27:14 git.dev.lon1.centos.org celery[382]: File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 238, in trace_task
Mar 27 12:27:14 git.dev.lon1.centos.org celery[382]: R = retval = fun(*args, **kwargs)
Mar 27 12:27:14 git.dev.lon1.centos.org celery[382]: File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 416, in __protected_call__
Mar 27 12:27:14 git.dev.lon1.centos.org celery[382]: return self.run(*args, **kwargs)
Mar 27 12:27:14 git.dev.lon1.centos.org celery[382]: File "/usr/lib/python2.7/site-packages/pagure/lib/tasks_utils.py", line 36, in decorated_function
Mar 27 12:27:14 git.dev.lon1.centos.org celery[382]: return function(self, session, *args, **kwargs)
Mar 27 12:27:14 git.dev.lon1.centos.org celery[382]: File "/usr/lib/python2.7/site-packages/pagure/lib/tasks.py", line 267, in create_project
Mar 27 12:27:14 git.dev.lon1.centos.org celery[382]: ignore_existing_repo,
Mar 27 12:27:14 git.dev.lon1.centos.org celery[382]: File "/usr/lib/python2.7/site-packages/pagure/lib/git.py", line 2612, in create_project_repos
Mar 27 12:27:14 git.dev.lon1.centos.org celery[382]: project, region, templ, ignore_existing, repotype
Mar 27 12:27:14 git.dev.lon1.centos.org celery[382]: File "/usr/lib/python2.7/site-packages/pagure/lib/git.py", line 2544, in _create_project_repo
Mar 27 12:27:14 git.dev.lon1.centos.org celery[382]: regioninfo["admin_cert"]["key"],
Mar 27 12:27:14 git.dev.lon1.centos.org celery[382]: File "/usr/lib/python2.7/site-packages/requests/api.py", line 108, in post
Mar 27 12:27:14 git.dev.lon1.centos.org celery[382]: return request('post', url, data=data, json=json, **kwargs)
Mar 27 12:27:14 git.dev.lon1.centos.org celery[382]: File "/usr/lib/python2.7/site-packages/requests/api.py", line 50, in request
Mar 27 12:27:14 git.dev.lon1.centos.org celery[382]: response = session.request(method=method, url=url, **kwargs)
Mar 27 12:27:14 git.dev.lon1.centos.org celery[382]: File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 464, in request
Mar 27 12:27:14 git.dev.lon1.centos.org celery[382]: resp = self.send(prep, **send_kwargs)
Mar 27 12:27:14 git.dev.lon1.centos.org celery[382]: File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
Mar 27 12:27:14 git.dev.lon1.centos.org celery[382]: r = adapter.send(request, **kwargs)
Mar 27 12:27:14 git.dev.lon1.centos.org celery[382]: File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
Mar 27 12:27:14 git.dev.lon1.centos.org celery[382]: raise ConnectionError(err, request=request)
Mar 27 12:27:14 git.dev.lon1.centos.org celery[382]: ConnectionError: ('Connection aborted.', error(111, 'Connection refused'))

Impossible then to create the repo through api, as it exists (from Pagure side), so goes 500 :
https://git.dev.centos.org/sig-core/repospanner_down.
And if repospanner is available again, of course repospanner itself answers 404 as nothing was ever created :

[Wed Mar 27 12:35:20.250088 2019] [:error] [pid 25397] Traceback (most recent call last):
[Wed Mar 27 12:35:20.250115 2019] [:error] [pid 25397]   File "/usr/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
[Wed Mar 27 12:35:20.250146 2019] [:error] [pid 25397]     response = self.full_dispatch_request()
[Wed Mar 27 12:35:20.250154 2019] [:error] [pid 25397]   File "/usr/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
[Wed Mar 27 12:35:20.250161 2019] [:error] [pid 25397]     rv = self.handle_user_exception(e)
[Wed Mar 27 12:35:20.250167 2019] [:error] [pid 25397]   File "/usr/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
[Wed Mar 27 12:35:20.250173 2019] [:error] [pid 25397]     reraise(exc_type, exc_value, tb)
[Wed Mar 27 12:35:20.250179 2019] [:error] [pid 25397]   File "/usr/lib/python2.7/site-packages/flask/app.py", line 1473, in full_dispatch_request
[Wed Mar 27 12:35:20.250186 2019] [:error] [pid 25397]     rv = self.preprocess_request()
[Wed Mar 27 12:35:20.250192 2019] [:error] [pid 25397]   File "/usr/lib/python2.7/site-packages/flask/app.py", line 1666, in preprocess_request
[Wed Mar 27 12:35:20.250198 2019] [:error] [pid 25397]     rv = func()
[Wed Mar 27 12:35:20.250204 2019] [:error] [pid 25397]   File "/usr/lib/python2.7/site-packages/pagure/flask_app.py", line 354, in set_request
[Wed Mar 27 12:35:20.250210 2019] [:error] [pid 25397]     flask.g.branches = sorted(flask.g.repo_obj.listall_branches())
[Wed Mar 27 12:35:20.250216 2019] [:error] [pid 25397] GitError: Error received from repoSpanner: 404

So what's the way to ensure that pagure_worker fixes those request and doesn't create in pagure the repo if it wasn't able to really init it in repospanner region ?
Next question is : how to solve that (I'll open a new RFE to allow deleting a project through api, as I have hundreds of repos in that state for now)


Metadata Update from @pingou:
- Issue tagged with: bug

5 years ago

I confirm that following patch (https://pagure.io/pagure/c/a3b1fe3ec6dcec2b46b60f96b1c84f30468cebda.patch) solves this.
I disabled repospanner, tried to create project and go directly "connection refused" and so no project/repo was created in pagure (Expected).
Thank you ! I guess it will be merge soon and so available in next release. Feel free to change status on that issue when it will be merged (or ask me to even close it once it landed in the tree)

Metadata Update from @pingou:
- Issue assigned to pingou
- Issue set to the milestone: 5.5

4 years ago

Login to comment on this ticket.

Metadata