PR#130: Drop support for fedmsg and replace by fedora-messaging - taskotron/resultsdb

taskotron / resultsdb

#130 Drop support for fedmsg and replace by fedora-messaging

Merged 5 years ago by jskladan. Opened 5 years ago by cverna.

taskotron/ cverna/resultsdb replace_fedmsg into develop

Drop support for fedmsg and replace by fedora-messaging

Clement Verna • 5 years ago

cd57fc0

conf/settings.py.example

file modified

-6

		`@@ -58,12 +58,6 @@`
		`# default, but you could create your own.`
		`# Supported values: 'dummy', 'stomp', 'fedmsg'`
		`MESSAGE_BUS_PLUGIN = 'fedmsg'`
		`- # You can pass extra arguments to your message bus plugin here. For instance,`
		- # the fedmsg plugin expects an extra `modname` argument that can be used to
		`- # configure the topic, like this:`
		`- # <topic_prefix>.<environment>.<modname>.<topic>`
		`- # e.g. org.fedoraproject.prod.taskotron.result.new`
		`- MESSAGE_BUS_KWARGS = {'modname': 'resultsdb'}`

		`## Alternatively, you could use the 'stomp' messaging plugin.`
		`#MESSAGE_BUS_PLUGIN = 'stomp'`

requirements.txt

file modified

+1 -1

		`@@ -7,7 +7,7 @@`
		`# A note for maintainers: Please keep this list in sync and in the same order`
		`# as the spec file.`

		`- fedmsg >= 0.16.2`
		`+ fedora-messaging`
		`alembic >= 0.8.3`
		`Flask >= 0.10.1`
		`# FIXME: Flask-RESTful 0.3.6 breaks tests, see https://phab.qa.fedoraproject.org/T961`

resultsdb.spec

file modified

+2 -3

		`@@ -11,8 +11,7 @@`
		`BuildArch: noarch`

		`%if 0%{?fedora}`
		`- Requires: fedmsg`
		`- Requires: python3-fedmsg`
		`+ Requires: python3-fedora-messaging`
		`Requires: python3-alembic`
		`Requires: python3-flask`
		`Requires: python3-flask-restful`
		`@@ -21,7 +20,7 @@`
		`Requires: python3-six`
		`Requires: python3-sqlalchemy`
		`%else`
		`- Requires: fedmsg >= 0.16.2`
		`+ Requires: python-fedora-messaging`
		`Requires: python-alembic >= 0.8.3`
		`Requires: python-flask >= 0.10.1`
		`Requires: python-flask-restful >= 0.2.11`

resultsdb/messaging.py

file modified

+26 -5

		`@@ -22,7 +22,8 @@`

		`import pkg_resources`

		`- import fedmsg`
		`+ from fedora_messaging.api import Message, publish`
		`+ from fedora_messaging.exceptions import PublishReturned, ConnectionException`

		`from resultsdb import db`
		`from resultsdb.models.results import Result, ResultData`
		`@@ -83,7 +84,7 @@`
		`if datum.key in ('item', 'type',)`
		`)`
		`task['name'] = result.testcase.name`
		`- msg = {`
		`+ body = {`
		`'task': task,`
		`'result': {`
		`'id': result.id,`
		`@@ -95,9 +96,18 @@`
		`}`

		`if include_job_url: # only in the v1 API`
		`- msg['result']['job_url'] = result.groups[0].ref_url if result.groups else None`
		`+ body['result']['job_url'] = result.groups[0].ref_url if result.groups else None`

		`- fedmsg.publish(topic='result.new', modname='taskotron', msg=msg)`
		`+ try:`
		`+ msg = Message (`
		`+ topic='taskotron.result.new',`
		`+ body=body`
		`+ )`
		`+ publish(msg)`
		`+ except PublishReturned as e:`
		`+ log.error('Fedora Messaging broker rejected message {}: {}'.format(msg.id, e))`
		`+ except ConnectionException as e:`
		`+ log.error('Error sending message {}: {}'.format(msg.id, e))`


		`def create_message(result):`
		`@@ -138,7 +148,18 @@`
		`""" A fedmsg plugin, used to publish to the fedmsg bus. """`

		`def publish(self, message):`
		`- fedmsg.publish(topic='result.new', modname=self.modname, msg=message)`
		`+`
		`+ try:`
		`+ msg = Message(`
		`+ topic='{}.result.new'.format(self.modname),`
		`+ body=message`
		`+ )`
		`+ publish(msg)`
		`+ except PublishReturned as e:`
		`+ log.error('Fedora Messaging broker rejected message {}: {}'.format(msg.id, e))`
		`+ except ConnectionException as e:`
		`+ log.error('Error sending message {}: {}'.format(msg.id, e))`
		`+`


		`class StompPlugin(MessagingPlugin):`

cverna commented 5 years ago

This commit replaces fedmsg by fedora-messaging a library
that uses AMQP protocol. More details can be found here
https://fedoraproject.org/wiki/Infrastructure_2020/Fedora_Messaging.

Signed-off-by: Clement Verna cverna@tutanota.com

jskladan commented 5 years ago

Looks good to me, I'll merge this shortly. Thanks for the great work!

Pull-Request has been merged by jskladan

5 years ago

kparal commented 5 years ago

Josef, which Taskotron projects do we need to modify to make sure it still works well with the new ResultsDB?

jskladan commented 5 years ago

@kparal Thing is, the content of the messages (or the logic behind spawning them) is not changing, only the "delivery framework" is, so the question is not "what should we change in order to work with new resultsdb", but "what should we switch from fedmsg to fedora_messaging"

I don't know that any of the bits in the taskotron stack consume the fedmessages ATM, other than trigger (and even that IMO does not consume the resultsdb messages). And that will need to be changed not because of the ResultsDB change, but to use the fedora_messaging instead of fedmsg.

Maybe openqa consumes resultsdb messages? But then again, the bit that receives the messages from the void needs to be changed, not the parts that use the content of them (AFAIK, @cverna could you confirm, please?)

cverna commented 5 years ago

@jskladan and @kparal that should not affect other systems, there is a bridge running between fedora-messaging and fedmsg that replicates the messages on the bus. So anything sent by fedora-messaging will still be available on the fedmsg bus.

We are planning to look at openqa you can see the status of the migration here (https://github.com/orgs/fedora-infra/projects/2). Note that this is focused on the application we need for rawhide gating.