Hello, looking at src.fp.o push messages - they have very different structure from the normal Pagure messages. I.e.
https://apps.fedoraproject.org/datagrepper/id?id=2017-b7779d69-3e1d-46ec-bb05-00e345c884f5&is_raw=true&size=extra-large
vs.
https://apps.fedoraproject.org/datagrepper/id?id=2017-5655addf-60b3-4630-ba33-2da4b4cfe53f&is_raw=true&size=extra-large
I wonder why is that and if those message formats could be unified. The reason is that you might want to write a script (we do for COPR) that listens both to src.fp.o and pagure.io git.receive topics and which then can process those messages in the exactly same way. Additionally, the src.fp.o message misses some useful information like url_path to the repo for easy composition of clone URL (instead we need to compose it from namespace and repo name) and start and end commit (instead there is rev field). Additonally git.receive messages for forked repos does not seem to be emitted at all.
url_path
rev
I think these two messages types should be pretty much the same. There are some additional fields like path in src.fp.o message but this instead is information that should be internal to the DistGit server.
path
So the main issue is that the messages are not emitted for forks but the message format (and mainly its sameness across Pagure instances) could be improved as well. Is that possible? I understand there might be already some consumers for the messages, which might complicate the matters but it would be really nice and useful to have those messages unified.
Here is the mentioned script: https://pagure.io/copr/copr/c/c520baa3aaa6f2c5b468a1a18b46e746b2c574ca?branch=master
Note that the hook is stored in ansible so it would be easy to adjust but we'll need to be careful to keep it backward compatible.
@clime can you propose a change to the hook that would perhaps emit things in a backward compatible, but yet more standard way? or what can we do to move this forward?
Well, we agreed with @pingou (in https://pagure.io/fedora-infrastructure/issue/6612 and on IRC), it would be probably the best to keep the current messages while also enabling the Pagure ones (the only fedmsgs sent originally were custom ones created by a script in Infrastructure playbooks).
I have checked how it behaves right now for src.fp.o: and the behavior is a little bit curious. For main (unforked) repos, the original fedmsg (org.fedoraproject.prod.git.receive) is sent upon push. But for forked repos, the new Pagure one is sent (org.fedoraproject.prod.pagure.git.receive) and the original is not sent.
My explanation for this is that the original webhooks are being set only for main repos and they override the Pagure webhooks there.
Anyway, the current behavior is alright for me (I really just needed some message being sent for forked repos). We could also enable the Pagure messages for the main repos so that listening scripts will need to listen only to org.fedoraproject.prod.pagure.git.receive and that's it. I don't exactly need this though (my script will need to listen to two events but it will at least work) so it's just a suggestion.
org.fedoraproject.prod.pagure.git.receive
I think I have fixed all the hooks to be the same now.
Can you check it?
Huh, sorry, I didn't get notified about this message :(.
Right now, only DistGit messages are sent for pushes into any DistGit repo (forked/unforked). That's actually not that great for COPR because:
1) COPR fedmsg listener currently counts on the Pagure-default messages to be sent for forks so it is not working correctly right now 2) the DistGit messages are fired for each individual commit which spawns a lot of (unnecessary) builds in COPR. Instead, Pagure messages groups commits in one push together and only one event is always fired.
Also see: https://pagure.io/fedora-infrastructure/issue/6791
I would like the Pagure default messages to be sent for both main repos and forked repos but also keep the original DistGit messages for main repos for backward compatibility. Would this be possible? I can send a patch.
@clime do you have time to work on this today (ie: before freeze)?
Huh, yes sure. But I would need to make sure we agree on the messages being actually sent on push. I wanted to keep it for discussion for the next meeting.
I don't have clear-cut solution right now :) but I could work something out if required.
I think we're ok with doing: - pagure.git.receive on all repos (main + forks) - git.receive only on main repos
I think we're ok with doing:
pagure.git.receive on all repos (main + forks) git.receive only on main repos
Ok, I will try to have a look.
I believe this is now fixed :)
https://apps.fedoraproject.org/datagrepper/raw?topic=org.fedoraproject.prod.pagure.git.receive and https://apps.fedoraproject.org/datagrepper/raw?topic=org.fedoraproject.prod.git.receive show messages for all repos
Metadata Update from @pingou: - Issue close_status updated to: Fixed
@kevin one thing I didn't do is fixing the cron job going through all the repos, shall I?
Metadata Update from @pingou: - Issue status updated to: Open (was: Closed)
@clime please let me know if you don't see the expected behavior :)
hm I'm not sure this actually works as we want, I am getting some email about the fedmsg failing to access the cert to sign the messages. However, we still see some in datagrepper, so some messages are going through. I'll need to check this further.
So to avoid the issue report at: https://pagure.io/fedora-infrastructure/issue/6866 I turned off the pagure fedmsg hook for now
Is this actually fixed? I just pushed something to a fork and I see messages in both git.receive and pagure.git.receive.
It should be fixed but it seems this git repo isn't using the appropriate git hook
Ok, I think I see, we have two scripts setting up/fixing the hooks and they are stepping on each other's toes.
I'll send a FBR to fix this
This is marked as fixed but I think it was decided that it wasn't fixed? Should it be reopened?
@mavit as far as I can see it is fixed, but I could be missing something. Are you seeing commits that aren't showing up in fedmsg?
My symptom was that commit https://src.fedoraproject.org/rpms/squeezelite/c/d1d5b7bd48d76af47244868ae413b2ac279b5a51?branch=master didn’t (within the time that I waited) trigger a build of https://copr.fedorainfracloud.org/coprs/mavit/squeezelite/package/squeezelite/ when I expected that it should. I saw the comments in this bug and didn’t look any more closely into the cause of my symptoms.
Can you try another commit and confirm it's working or not?
Commit https://src.fedoraproject.org/rpms/keepass/c/eb6741be7d30f22766316955534f4b849d533589?branch=master generated message https://apps.fedoraproject.org/datagrepper/id?id=2019-14f13837-7b5f-4b7d-bc46-55d235c64457&is_raw=true&size=extra-large which didn't trigger a build of https://copr.fedorainfracloud.org/coprs/mavit/keepass/package/keepass/.
@mavit: i assume this is an issue on copr side: can you, please, quickly report the issue at pagure.io/copr/copr? Thank you!
Since this is a copr side issue, closing this upstream... if there's a problem on our side, do re-open or file a new issues on it.
Thanks!
Metadata Update from @kevin: - Issue status updated to: Open (was: Closed)
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.