@kevin it seems like permission in directory have been changed ??

The problem is that we upgraded to rhel 8.8 and the new git version does not trust repos with different ownership.

This is a CVE they had: https://lore.kernel.org/git/xmqqv8veb5i6.fsf@gitster.g/

I've run the command to add a exception. Can you please try and resync/push another commit and let us see if that fixed it?

Same issue:

Output from the push (2023-05-22T18:24:43.257879):
  stdout: 
  stderr: fatal: detected dubious ownership in repository at '/srv/git/repositories/freeipa.git'
To add an exception for this directory, call:

    git config --global --add safe.directory /srv/git/repositories/freeipa.git

Hum, try now? I might have updated the wrong user...

No change:

Output from the push (2023-05-23T01:50:16.745740):
  stdout: 
  stderr: fatal: detected dubious ownership in repository at '/srv/git/repositories/freeipa.git'
To add an exception for this directory, call:

    git config --global --add safe.directory /srv/git/repositories/freeipa.git

I manually sync'ed master, ipa-4-10 and ipa-4-9 branches but the issue is still there.

Latest commit also failed to synchronize:

Output from the push (2023-05-23T13:27:33.682682):
  stdout: 
  stderr: fatal: detected dubious ownership in repository at '/srv/git/repositories/freeipa.git'
To add an exception for this directory, call:

    git config --global --add safe.directory /srv/git/repositories/freeipa.git

@kevin we still struggle. Could this be prioritized?

Yes, sorry for the troubles.

Can one of you join #fedora-admin (libera.chat) / #admin:fedoraproject.org (matrix) and we could try and do some more real time debugging?

Alternately I can have some more folks look at it and see what I might be missing.

I am already on IRC and keep asking for few days, nobody answers.

Odd. I don't see any queries about this. ;(

Anyhow, I tried something further, can you try another action to get it to sync?

Current state is

Output from the push (2023-05-24T17:00:34.493129):
  stdout: 
  stderr: fatal: detected dubious ownership in repository at '/srv/git/repositories/freeipa.git'
To add an exception for this directory, call:

    git config --global --add safe.directory /srv/git/repositories/freeipa.git

I cannot manually trigger a sync other than pushing into the repository but I don't have anything to push at this moment. If you know how to trigger a sync, please tell me.

Yeah, I don't know either. ;(

@pingou might know...

we pushed another update that should have triggered sync and it caused a failure too:

Output from the push (2023-05-24T18:11:23.275684):
  stdout: 
  stderr: fatal: detected dubious ownership in repository at '/srv/git/repositories/freeipa.git'
To add an exception for this directory, call:

    git config --global --add safe.directory /srv/git/repositories/freeipa.git

I've now just downgraded git to the 8.7 version. Hopefully that will work to unstick this until we can figure out where to fix the new one.

With the downgrade now we have working mirroring. Thank you. There are also issues with host signatures but this is due to Github problems:

Output from the push (2023-05-24T21:58:53.105448):
  stdout: 
  stderr: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s.
Please contact your system administrator.
Add correct host key in /srv/mirror/.ssh/known_hosts to get rid of this message.
Offending RSA key in /srv/mirror/.ssh/known_hosts:1
Password authentication is disabled to avoid man-in-the-middle attacks.
Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
Everything up-to-date

Can you clean up known_hosts as well, please?

Can you clean up known_hosts as well, please?

I just did that, does that help?

That particular issue is gone. However, the original issue is back:

Output from the push (2023-05-25T08:05:48.561652):
  stdout: 
  stderr: fatal: detected dubious ownership in repository at '/srv/git/repositories/freeipa.git'
To add an exception for this directory, call:

    git config --global --add safe.directory /srv/git/repositories/freeipa.git

this looks like somebody again upgraded git on the host

ok, so it was marked as a security update and out automation upgraded it again. ;(

I've re-downgraded it and excluded it from automation. ;(

ok. I would like to try setting this globally in /etc/gitconfig and re-upgrading.

Can we schedule some time next week to do that and test?

Hi @kevin
last commit was successfully mirrored from pagure to github. There is just a warning in the sync status:

Output from the push (2023-05-31T07:17:53.088532):
  stdout: 
  stderr: Warning: the ECDSA host key for 'github.com' differs from the key for the IP address '140.82.114.4'
Offending key for IP in /srv/mirror/.ssh/known_hosts:11
Matching host key in /srv/mirror/.ssh/known_hosts:25
To github.com:freeipa/freeipa.git
   359e1a3d9..2be07242b  master -> master

Yes, I would expect it to be working right now, it's still downgraded. ;)

I want to try and re-upgrade it and test, but I suppose I can try and do that with staging...

@kevin I set up https://stg.pagure.io/test_mirroring in staging pagure for testing, reproduced the issue and gave you admin rights in case you would like to play with it.

I was playing with this for some time today and at the end I was able to solve it.
The https://stg.pagure.io/test_mirroring is now mirrored to https://github.com/Zlopez/test_mirroring.

Here is what I tried:

1. sudo -u git git config --global --add safe.directory /srv/git/repositories/test_mirroring.git - that didn't work
2. sudo -u paguremirroring git config --global --add safe.directory /srv/git/repositories/test_mirroring.git - that didn't work, but it didn't show the error when trying git status in the directory as paguremirroring user
3. sudo -u paguremirroring git config --global --add safe.directory '*' - that didn't work as well
4. Creating /etc/gitconfig with the following

[safe]
        directory = *

The last thing solved it. But I'm not sure if this is a good solution as it sets every git directory as safe for every user on the machine.

Rather than using safe.directory * I would suggest:

sudo git config --system --add safe.directory /srv/git/repositories/test_mirroring.git

I.e. replacing --global with --system. The former writes to ~/.gitconfig instead of $PWD/.git/config while the latter writes to /etc/gitconfig.

I forgot about the --system flag, but it's basically what I did. I first tried to set it up only for specific users, this is why I used --global flag.

Yeah, though removing the * was the other important part of my suggestion. Noting the use of --system helps prevent others from thinking they need to write /etc/gitconfig directly.

As you said, using safe.directory * is a big hammer. It's certainly valid if it's safe to trust that all repos on the system will always be safe.

I don't know if that's true or not and would imagine being more selective is better. :)

So, a few things:

• The git repos on pagure are owned by 'git:git'.
• The pagure wsgi app runs as user git
• The mirroring celery tasks run as user 'paguremirroring'
• we need this to work for any project that enables mirroring, we can't just add them manually.

So, possible solutions:

1. We could adjust pagure to add the above config for a project when mirroring is enabled?

2. We could just globally allow, which I think would be safe to do in this case.

3. Some other solution to make paguremirroring user allowed/bypassing this error?

4. Something else clever. :)

I actually tried to set safe directory for both git and paguremirroring users, they both were able to run git status without having the error (the paguremirroring had this issue before enabling it). But this didn't help with actual mirroring.

I'm +1 for enabling this globally, I think we are safe to do this in case of pagure.

Yeah, same... can you make a PR to do this in staging and we can confirm it works there, then push to prod after freeze?

Did the change on staging in 2 PRs (https://pagure.io/fedora-infra/ansible/pull-request/1575 and https://pagure.io/fedora-infra/ansible/pull-request/1576). The second PR was just for fixing syntax error.

Added another commit to https://stg.pagure.io/test_mirroring and it showed up in https://github.com/Zlopez/test_mirroring. So I can confirm that this solution is working.

So now we can either wait with creating PR after freeze or create it now and just label it with post-freeze tag.

I'd say creating a 'post-freeze' tagged pr would be good...

PR created, let's now wait till the end of freeze :-)

The safe directories are now enabled on production pagure. Mirroring should work again now on RHEL 8.8.

@kevin When you have some time, you can upgrade it.

Upgraded. Will watch it for a bit and make sure it's working before closing this.

No new errors... but I am not sure that anything that is mirrored has pushed any commits. Will check back tomorrow.

I think it works for FreeIPA:

Output from the push (2023-09-26T09:31:04.217893):
  stdout: 
  stderr: Warning: the ECDSA host key for 'github.com' differs from the key for the IP address '140.82.112.3'
Offending key for IP in /srv/mirror/.ssh/known_hosts:13
Matching host key in /srv/mirror/.ssh/known_hosts:25
To github.com:freeipa/freeipa.git
   496e3ace8..4af05dde4  master -> master

Yep. :) Thanks everyone.