#4636 Event hooks on src.f.o sometime not called, is the event hook processor stuck ?
Closed: Invalid 6 months ago by fbo. Opened 9 months ago by fbo.

Hello,

That's not the first time that I see the event hook are not called. This morning again, it worked and I received my last event at 07:50:30 UTC, then no more calls.

Last call payload:

2019-10-08 07:50:30,878 DEBUG zuul.RPCClient: Submitting job pagure:src.fedoraproject.org:payload with data {'payload': {'topic': 'pull-request.comment.added', 'msg_id': '2019-c1a4b407-0ef3-4ef6-b049-32107bd14e6b', 'i': 2, 'msg': {'pullrequest': {'initial_comment': 'Depends-on: https://pagure.io/fedora-zuul-jobs/pull-request/6', 'user': {'fullname': 'Fabien Boucher', 'name': 'fbo'}, 'closed_at': None, 'id': 8, 'commit_start': '8ac634d66b7e756e2e4c3711717a7a7d0563385e', 'project': {'user': {'fullname': 'Fabien Boucher', 'name': 'fbo'}, 'close_status': [], 'date_created': '1501874074', 'tags': [], 'namespace': 'rpms', 'url_path': 'rpms/python-gear', 'id': 18095, 'parent': None, 'custom_keys': [], 'fullname': 'rpms/python-gear', 'priorities': {}, 'access_users': {'commit': ['tdecacqu'], 'admin': ['pabelanger', 'zuul'], 'ticket': [], 'owner': ['fbo']}, 'access_groups': {'commit': [], 'admin': [], 'ticket': []}, 'name': 'python-gear', 'date_modified': '1570520978', 'milestones': {}, 'description': 'The python-gear rpms'}, 'date_created': '1568636672', 'commit_stop': '8ac634d66b7e756e2e4c3711717a7a7d0563385e', 'threshold_reached': False, 'cached_merge_status': 'MERGE', 'remote_git': None, 'uid': '10751ac8709241f09d53ab2c4c8ac997', 'last_updated': '1570521026', 'assignee': None, 'tags': [], 'branch': 'master', 'title': 'Test bump for testing rpminspect job', 'comments': [{'editor': None, 'comment': 'recheck', 'edited_on': None, 'commit': None, 'line': None, 'id': 30486, 'parent': None, 'user': {'fullname': 'Fabien Boucher', 'name': 'fbo'}, 'tree': None, 'date_created': '1568637571', 'reactions': {}, 'filename': None, 'notification': False}, {'editor': None, 'comment': 'recheck', 'edited_on': None, 'commit': None, 'line': None, 'id': 30487, 'parent': None, 'user': {'fullname': 'Fabien Boucher', 'name': 'fbo'}, 'tree': None, 'date_created': '1568637747', 'reactions': {}, 'filename': None, 'notification': False}, {'editor': None, 'comment': 'recheck', 'edited_on': None, 'commit': None, 'line': None, 'id': 30488, 'parent': None, 'user': {'fullname': 'Fabien Boucher', 'name': 'fbo'}, 'tree': None, 'date_created': '1568637793', 'reactions': {}, 'filename': None, 'notification': False}, {'editor': None, 'comment': 'recheck', 'edited_on': None, 'commit': None, 'line': None, 'id': 30489, 'parent': None, 'user': {'fullname': 'Fabien Boucher', 'name': 'fbo'}, 'tree': None, 'date_created': '1568638151', 'reactions': {}, 'filename': None, 'notification': False}, {'editor': None, 'comment': 'recheck', 'edited_on': None, 'commit': None, 'line': None, 'id': 30530, 'parent': None, 'user': {'fullname': 'Fabien Boucher', 'name': 'fbo'}, 'tree': None, 'date_created': '1568707287', 'reactions': {}, 'filename': None, 'notification': False}, {'editor': None, 'comment': 'recheck', 'edited_on': None, 'commit': None, 'line': None, 'id': 30531, 'parent': None, 'user': {'fullname': 'Fabien Boucher', 'name': 'fbo'}, 'tree': None, 'date_created': '1568707539', 'reactions': {}, 'filename': None, 'notification': False}, {'editor': None, 'comment': 'recheck', 'edited_on': None, 'commit': None, 'line': None, 'id': 30591, 'parent': None, 'user': {'fullname': 'Fabien Boucher', 'name': 'fbo'}, 'tree': None, 'date_created': '1568799538', 'reactions': {}, 'filename': None, 'notification': False}, {'editor': None, 'comment': 'Build failed.\n\n- [rawhide-rpm-scratch-build ](https://fedora.softwarefactory-project.io/zuul/t/fedora-staging/build/9ab14cb3fea54ed4b5358233ae610f31) : SUCCESS in 7m 17s\n- [rawhide-rpm-test ](https://fedora.softwarefactory-project.io/zuul/t/fedora-staging/build/5e09339aa8f442afb6f598fab0771ae6) : SUCCESS in 3m 19s\n- [artifact-rpm-lint ](https://fedora.softwarefactory-project.io/zuul/t/fedora-staging/build/a3f1a1d26e3a44d185ec8352086163c5) : SUCCESS in 2m 28s\n- [artifact-rpminspect ](https://fedora.softwarefactory-project.io/zuul/t/fedora-staging/build/1d61914c2dc6410595a673bf28f08c6b) : FAILURE in 3m 07s\n', 'edited_on': None, 'commit': None, 'line': None, 'id': 30592, 'parent': None, 'user': {'fullname': 'Fabien Boucher', 'name': 'fbo'}, 'tree': None, 'date_created': '1568800536', 'reactions': {}, 'filename': None, 'notification': False}, {'editor': None, 'comment': 'recheck', 'edited_on': None, 'commit': None, 'line': None, 'id': 30830, 'parent': None, 'user': {'fullname': 'Fabien Boucher', 'name': 'fbo'}, 'tree': None, 'date_created': '1569399238', 'reactions': {}, 'filename': None, 'notification': False}, {'editor': None, 'comment': 'Build failed.\n\n- [rawhide-rpm-scratch-build ](https://fedora.softwarefactory-project.io/zuul/t/fedora-staging/build/eee7e79cc5be4301bcae78c1c6bf2c3e) : SUCCESS in 8m 22s\n- [rawhide-rpm-test ](https://fedora.softwarefactory-project.io/zuul/t/fedora-staging/build/53d85df0e9a7432d962eb475b97b6a96) : SUCCESS in 3m 21s\n- [artifact-rpm-lint ](https://fedora.softwarefactory-project.io/zuul/t/fedora-staging/build/1bb537183573402c9ef5cb74d2e80324) : SUCCESS in 3m 04s\n- [artifact-rpminspect ](https://fedora.softwarefactory-project.io/zuul/t/fedora-staging/build/588ae1b42c144e8f88e692f0f75d9048) : FAILURE in 3m 45s\n', 'edited_on': None, 'commit': None, 'line': None, 'id': 30831, 'parent': None, 'user': {'fullname': 'Fabien Boucher', 'name': 'fbo'}, 'tree': None, 'date_created': '1569400238', 'reactions': {}, 'filename': None, 'notification': False}, {'editor': None, 'comment': 'recheck', 'edited_on': None, 'commit': None, 'line': None, 'id': 31808, 'parent': None, 'user': {'fullname': 'Fabien Boucher', 'name': 'fbo'}, 'tree': None, 'date_created': '1570521026', 'reactions': {}, 'filename': None, 'notification': False}], 'repo_from': {'user': {'fullname': 'Fabien Boucher', 'name': 'fbo'}, 'close_status': [], 'date_created': '1560345137', 'tags': [], 'namespace': 'rpms', 'url_path': 'fork/fbo/rpms/python-gear', 'id': 36500, 'parent': {'user': {'fullname': 'Fabien Boucher', 'name': 'fbo'}, 'close_status': [], 'date_created': '1501874074', 'tags': [], 'namespace': 'rpms', 'url_path': 'rpms/python-gear', 'id': 18095, 'parent': None, 'custom_keys': [], 'fullname': 'rpms/python-gear', 'priorities': {}, 'access_users': {'commit': ['tdecacqu'], 'admin': ['pabelanger', 'zuul'], 'ticket': [], 'owner': ['fbo']}, 'access_groups': {'commit': [], 'admin': [], 'ticket': []}, 'name': 'python-gear', 'date_modified': '1570520978', 'milestones': {}, 'description': 'The python-gear rpms'}, 'custom_keys': [], 'fullname': 'forks/fbo/rpms/python-gear', 'priorities': {}, 'access_users': {'commit': [], 'admin': [], 'ticket': [], 'owner': ['fbo']}, 'access_groups': {'commit': [], 'admin': [], 'ticket': []}, 'name': 'python-gear', 'date_modified': '1560345137', 'milestones': {}, 'description': 'The python-gear rpms'}, 'closed_by': None, 'branch_from': 'test-rpminspect', 'status': 'Open', 'updated_on': '1570521026'}, 'project_fullname': 'rpms/python-gear', 'agent': 'fbo', 'pagure_instance': 'https://src.fedoraproject.org/'}, 'timestamp': 1570521029}}

Since this last event call I no longer receive event call on my side.
It happened previously at least 3 times and it was fixed somehow as the day after it started to work again.

This time I'm opening an issue as it is an problem regarding the workflow I want to implement as it requires the stability of events hook system.

I don't remember it happened with pagure.io.

Let me know if you need more information

Thanks a lot in advance for your help


Today It started to work again. Is there any logs server side that could help to know if the hook service get stuck sometime ?

Hello,

I never received the hook calls, that should have been generated by the comments added to the PRs below the Friday 18, Monday 21.

https://src.fedoraproject.org/rpms/python-zuul-sphinx/pull-request/2
https://src.fedoraproject.org/rpms/nodepool/pull-request/1

This morning it started to work again w/o any change on tooling/config.

Could it be possible to check what is happening server side ?

@fbo could it be the case of 1 commit vs multiple commits pushed?

We just removed an old hotfix that was deployed on src.fp.o and was breaking when someone pushed multiple commits at once. So it could be that since it was failing earlier in the script (fedmsg notifs are sent before the web-hooks' ones), it never made it to the web-hooks.

If that was the issue, it should be fixed :)

No I don't think that was related. My test case is based on the pr commented event (but seems to affect all events) It seems to be more an infra issue. As @jlanda mentioned in https://pagure.io/fedora-infrastructure/issue/8497 the right and up to date issue for this is on fedora-infrastructure project: https://pagure.io/fedora-infrastructure/issue/8320.
So I'm closing this one.

Metadata Update from @fbo:
- Issue close_status updated to: Invalid
- Issue status updated to: Closed (was: Open)

6 months ago

Login to comment on this ticket.

Metadata