Adding to list of stg problems.

I'd like to raise priority on this issue. This is blocking our work on packit where we:
1. can't fork a src.fp.o repo via API
2. we need another API token of the forked repo

These two points make a terrible user experience for API users.

I'd appreciate it very much if you fixed this by the end of the week.

@pingou should I file another ticket to run the updated playbook against prod?

There are multiple problems going on here and @puiterwijk and @pingou are working through the many code changes needed because it seems limited to a subset of forks and yours is in that subset. I can not give a deadline at this time for it to happen.

Okay, I understand, thanks for clarification.

Would it be possible to apply the playbook change which @pingou did in https://pagure.io/fedora-infrastructure/issue/7623 in production?

This was applied to src.stg.fedoraproject.org 12 days ago. I am not sure what it has to do with this issue

I am not sure what it has to do with this issue

That I cannot verify that the fix which was applied 12 days ago resolves the issue.

I am running this test against stg: https://github.com/packit-service/packit/blob/master/tests/integration/test_pagure.py#L14

If it passes, it means the issue is resolved and we only need one pagure token for the whole process.

Is possible to create the fork now, but it stays read-only.

the create fork API call still raises an error:

>               raise APIError(output['error'])
E               libpagure.exceptions.APIError: Remote hook declined the push: Performing pre-check...
E               Pre-check results in
E               Welcome to repoSpanner 0.5+2.9764a01d4983a6e19df68ba0d734fb708a6e06fe.el7.infra, node fedora01.rpms.stg.fedoraproject.org
E               Delta resolving finished
E               Validating objects...
E               Objects validated
E               Finishing hook runner preparation...
E               Telling hook runner to grab new contents...
E               Running pre-receive hook...
E               Traceback (most recent call last):
E               File "/tmp/repospanner_hook_runner_638326501/hookrun/pre-receive", line 72, in <module>
E               pull_request,
E               File "/usr/lib/python2.7/site-packages/pagure/hooks/__init__.py", line 355, in run_project_hooks
E               repodir=repodir,
E               File "/usr/lib/python2.7/site-packages/dist_git_auth.py", line 174, in check_acl
E               user = get_user(session, username)
E               File "/usr/lib/python2.7/site-packages/pagure/lib/query.py", line 131, in get_user
E               raise pagure.exceptions.PagureException('No user "%s" found' % key)
E               pagure.exceptions.PagureException: No user "pagure" found
E               Hook returned error
E               ERR Pre-receive hook refused push

Update, there seems to be a problem in the forking code in pagure working with repoSpanner. Patrick is working on fixes to deal with this.

This now works fine for me, with one minor (cosmetic) issue:

➜ ~ git clone ssh://kevin@pkgs.stg.fedoraproject.org/forks/kevin/rpms/driconf.git
Cloning into 'driconf'...
remote: Welcome to repoSpanner 0.5+4.aa81c3cf0e30d34a9a6da128c7eccfd0a34612e3.el7.infra, node fedora01.rpms.stg.fedorapro
ject.org
remote: Building packfile
remote: Packfile built, sending
remote: Packfile sent
Receiving objects: 100% (159/159), 77.61 KiB | 17.00 KiB/s, done.
warning: remote HEAD refers to nonexistent ref, unable to checkout.

You can then checkout master and all is well.

➜ driconf git:(master) ✗ git commit -s -a [61/1881]
[master 6c799a7] Whitespace change
1 file changed, 1 insertion(+)
➜ driconf git:(master) git push
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Delta compression using up to 8 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 2.50 KiB | 2.50 MiB/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: Performing pre-check...
remote: Pre-check results in
remote: Welcome to repoSpanner 0.5+4.aa81c3cf0e30d34a9a6da128c7eccfd0a34612e3.el7.infra, node fedora01.rpms.stg.fedorapro
ject.org
remote: Delta resolving finished
remote: Validating objects...
remote: Objects validated
remote: Finishing hook runner preparation...
remote: Telling hook runner to grab new contents...
remote: Running pre-receive hook...
remote: Protected namespaces: ['rpms', 'modules', 'container']
remote: Blocking unspecified refs: False
remote: Blacklists: [<_sre.SRE_Pattern object at 0x7f4317de7330>]
remote: User: User: 791 - name kevin
remote: User groups: set([u'cvsadmin', u'packager', u'infra-sig', u'provenpackager', u'sysadmin-main'])
remote: Committer: True
remote: SIG memberships: set([])
remote: RCM: False
remote: Committer push to unprotected
remote: Pre-receive hook done
remote: Running update hook...
remote: Protected namespaces: ['rpms', 'modules', 'container']
remote: Blocking unspecified refs: False
remote: Blacklists: [<_sre.SRE_Pattern object at 0x7fe4332c1330>]
remote: User: User: 791 - name kevin
remote: User groups: set([u'cvsadmin', u'packager', u'infra-sig', u'provenpackager', u'sysadmin-main'])
remote: Committer: True
remote: SIG memberships: set([])
remote: RCM: False
remote: Committer push to unprotected
remote: Update hook done
remote: Syncing objects...
remote: Objects synced
remote: Requesting push...
remote: Push results in
remote: Running post-receive hook...
remote: Sending to redis to log activity and send commit notification emails
remote: * Publishing information for 1 commits
remote: - to fedora-message
remote: Post-receive hook done
To ssh://pkgs.stg.fedoraproject.org/forks/kevin/rpms/driconf.git
9671ff0..6c799a7 master -> master

Can you all please retry (with new forks))?

I was able to create fork and it doesn't look like read-only.

However, my script, which is using API key is unable to push to repository with following error:

E           Cmd('git') failed due to: exit code(128)
E             cmdline: git push --porcelain fork master
E             stderr: 'fatal: Could not read from remote repository.
E           
E           Please make sure you have the correct access rights
E           and the repository exists.'

I wanted to test it manually, but I don't see any way to add SSH key (I have it in my FAS account already) and the fork is giving me only HTTPS URL for cloning.

You can't push to src.fedoraproject.org with API key.
For that you will either need to use an SSH key, which will get synced from FAS, or an OpenID Connect token.
For the latter, run standard "fedpkg push", though this (for now) will require a graphical interface.

So I think that for now you'll want to just use an SSH key.

Do note, however, that this means the original issue of this ticket has been fixed, so thanks for confirming that.

@puiterwijk Glad to help.

I added the SSH key half year ago, so I don't understand why src.stg.fp.o can't see it.

Nevermind, I found out that I didn't had SSH key on FAS staging instance.

Great, then I'd say that this is now resolved.
If you hit any other issues, feel free to open a new ticket.