PR#728: Use a much less memory hungry lambda - fedora-badges

fedora-badges

#728 Use a much less memory hungry lambda

Merged 4 years ago by misc. Opened 4 years ago by misc.

misc/fedora-badges fix_rules into master

Use a much less memory hungry lambda

Michael Scherer • 4 years ago

4ddc017

file modified

+1 -1

		`@@ -31,7 +31,7 @@`
		`- "%(msg.owner)s" # The user that was created.`
		`rows_per_page: 9999999`
		`operation:`
		`- lambda: len([msg for msg in query.all() if msg.msg['new'] is 1])`
		`+ lambda: sum(1 for msg in query.all() if msg.msg['new'] is 1)`
		`condition:`
		`greater than or equal to: 1`

rules/koji-if-you-build-it-02.yml

file modified

+1 -1

		`@@ -31,7 +31,7 @@`
		`- "%(msg.owner)s" # The user that was created.`
		`rows_per_page: 9999999`
		`operation:`
		`- lambda: len([msg for msg in query.all() if msg.msg['new'] is 1])`
		`+ lambda: sum(1 for msg in query.all() if msg.msg['new'] is 1)`
		`condition:`
		`greater than or equal to: 10`

rules/koji-if-you-build-it-03.yml

file modified

+1 -1

		`@@ -31,7 +31,7 @@`
		`- "%(msg.owner)s" # The user that was created.`
		`rows_per_page: 9999999`
		`operation:`
		`- lambda: len([msg for msg in query.all() if msg.msg['new'] is 1])`
		`+ lambda: sum(1 for msg in query.all() if msg.msg['new'] is 1)`
		`condition:`
		`greater than or equal to: 50`

rules/koji-if-you-build-it-04.yml

file modified

+1 -1

		`@@ -31,7 +31,7 @@`
		`- "%(msg.owner)s" # The user that was created.`
		`rows_per_page: 9999999`
		`operation:`
		`- lambda: len([msg for msg in query.all() if msg.msg['new'] is 1])`
		`+ lambda: sum(1 for msg in query.all() if msg.msg['new'] is 1)`
		`condition:`
		`greater than or equal to: 250`

rules/koji-if-you-build-it-05.yml

file modified

+1 -1

		`@@ -31,7 +31,7 @@`
		`- "%(msg.owner)s" # The user that was created.`
		`rows_per_page: 9999999`
		`operation:`
		`- lambda: len([msg for msg in query.all() if msg.msg['new'] is 1])`
		`+ lambda: sum(1 for msg in query.all() if msg.msg['new'] is 1)`
		`condition:`
		`greater than or equal to: 1000`

rules/koji-its-over-9000.yml

file modified

+1 -1

		`@@ -23,7 +23,7 @@`
		`- "%(msg.owner)s" # The user that was created.`
		`rows_per_page: 9999999`
		`operation:`
		`- lambda: len([msg for msg in query.all() if msg.msg['new'] is 1])`
		`+ lambda: sum(1 for msg in query.all() if msg.msg['new'] is 1)`
		`condition:`
		`greater than or equal to: 9001`

rules/koji-the-tower-of-babel.yml

file modified

+1 -1

		`@@ -23,7 +23,7 @@`
		`- "%(msg.owner)s" # The user that was created.`
		`rows_per_page: 9999999`
		`operation:`
		`- lambda: len([msg for msg in query.all() if msg.msg['new'] is 1])`
		`+ lambda: sum(1 for msg in query.all() if msg.msg['new'] is 1)`
		`condition:`
		`greater than or equal to: 10000`

rules/koji-what-goes-up-01.yml

file modified

+1 -1

		`@@ -31,7 +31,7 @@`
		`- "%(msg.owner)s" # The user that was created.`
		`rows_per_page: 9999999`
		`operation:`
		`- lambda: len([msg for msg in query.all() if msg.msg['new'] is 3])`
		`+ lambda: sum(1 for msg in query.all() if msg.msg['new'] is 3)`
		`condition:`
		`greater than or equal to: 1`

rules/koji-what-goes-up-02.yml

file modified

+1 -1

		`@@ -31,7 +31,7 @@`
		`- "%(msg.owner)s" # The user that was created.`
		`rows_per_page: 9999999`
		`operation:`
		`- lambda: len([msg for msg in query.all() if msg.msg['new'] is 3])`
		`+ lambda: sum(1 for msg in query.all() if msg.msg['new'] is 3)`
		`condition:`
		`greater than or equal to: 5`

rules/koji-what-goes-up-03.yml

file modified

+1 -1

		`@@ -31,7 +31,7 @@`
		`- "%(msg.owner)s" # The user that was created.`
		`rows_per_page: 9999999`
		`operation:`
		`- lambda: len([msg for msg in query.all() if msg.msg['new'] is 3])`
		`+ lambda: sum(1 for msg in query.all() if msg.msg['new'] is 3)`
		`condition:`
		`greater than or equal to: 20`

rules/koji-what-goes-up-04.yml

file modified

+1 -1

		`@@ -31,7 +31,7 @@`
		`- "%(msg.owner)s" # The user that was created.`
		`rows_per_page: 9999999`
		`operation:`
		`- lambda: len([msg for msg in query.all() if msg.msg['new'] is 3])`
		`+ lambda: sum(1 for msg in query.all() if msg.msg['new'] is 3)`
		`condition:`
		`greater than or equal to: 100`

rules/koji-what-goes-up-05.yml

file modified

+1 -1

		`@@ -31,7 +31,7 @@`
		`- "%(msg.owner)s" # The user that was created.`
		`rows_per_page: 9999999`
		`operation:`
		`- lambda: len([msg for msg in query.all() if msg.msg['new'] is 3])`
		`+ lambda: sum(1 for msg in query.all() if msg.msg['new'] is 3)`
		`condition:`
		`greater than or equal to: 250`

misc commented 4 years ago

Since [msg for msg in query.all() if msg.msg['new'] is 1] is
constructing a gigantic list in memory with all decoded json
message, we are hitting memory limit on a regular basis. I suspect
something changed on fedmsg side recently that did start
to make the msg take a lot more memory since fedbadge crash
every 10 to 15 minutes each time a build is made.

Increasing memory is not enough, since that consume more than 20G
(I did test that). So the simplest solution is to not
build a list to count the items, but just count them with sum.

The same pattern should be applied to others rules.

misc commented 4 years ago

For people who want to see the effect on memory, I did a hotfix on the server, so you can take a look at the graph:

https://admin.fedoraproject.org/collectd/bin/index.cgi?hostname=badges-backend01.phx2.fedoraproject.org&plugin=memory&timespan=3600&action=show_selection&ok_button=OK

Look around 9h on the 31 of march 2020.

misc commented 4 years ago

Ok so I still see some memory error, but much less. I do wonder if that's caused by others queries or something, as they are also reusing the same pattern.

misc commented 4 years ago