#728 Use a much less memory hungry lambda
Merged 2 years ago by misc. Opened 2 years ago by misc.
misc/fedora-badges fix_rules  into  master

Use a much less memory hungry lambda
Michael Scherer • 2 years ago  
@@ -31,7 +31,7 @@ 

          - "%(msg.owner)s"  # The user that was created.

          rows_per_page: 9999999

        operation:

-         lambda: len([msg for msg in query.all() if msg.msg['new'] is 1])

+         lambda: sum(1 for msg in query.all() if msg.msg['new'] is 1)

        condition:

          greater than or equal to: 1 

  

@@ -31,7 +31,7 @@ 

          - "%(msg.owner)s"  # The user that was created.

          rows_per_page: 9999999

        operation:

-         lambda: len([msg for msg in query.all() if msg.msg['new'] is 1])

+         lambda: sum(1 for msg in query.all() if msg.msg['new'] is 1)

        condition:

          greater than or equal to: 10

  

@@ -31,7 +31,7 @@ 

          - "%(msg.owner)s"  # The user that was created.

          rows_per_page: 9999999

        operation:

-         lambda: len([msg for msg in query.all() if msg.msg['new'] is 1])

+         lambda: sum(1 for msg in query.all() if msg.msg['new'] is 1)

        condition:

          greater than or equal to: 50

  

@@ -31,7 +31,7 @@ 

          - "%(msg.owner)s"  # The user that was created.

          rows_per_page: 9999999

        operation:

-         lambda: len([msg for msg in query.all() if msg.msg['new'] is 1])

+         lambda: sum(1 for msg in query.all() if msg.msg['new'] is 1)

        condition:

          greater than or equal to: 250 

  

@@ -31,7 +31,7 @@ 

          - "%(msg.owner)s"  # The user that was created.

          rows_per_page: 9999999

        operation:

-         lambda: len([msg for msg in query.all() if msg.msg['new'] is 1])

+         lambda: sum(1 for msg in query.all() if msg.msg['new'] is 1)

        condition:

          greater than or equal to: 1000 

  

file modified
+1 -1
@@ -23,7 +23,7 @@ 

          - "%(msg.owner)s"  # The user that was created.

          rows_per_page: 9999999

        operation:

-         lambda: len([msg for msg in query.all() if msg.msg['new'] is 1])

+         lambda: sum(1 for msg in query.all() if msg.msg['new'] is 1)

        condition:

          greater than or equal to: 9001

  

@@ -23,7 +23,7 @@ 

          - "%(msg.owner)s"  # The user that was created.

          rows_per_page: 9999999

        operation:

-         lambda: len([msg for msg in query.all() if msg.msg['new'] is 1])

+         lambda: sum(1 for msg in query.all() if msg.msg['new'] is 1)

        condition:

          greater than or equal to: 10000

  

@@ -31,7 +31,7 @@ 

          - "%(msg.owner)s"  # The user that was created.

          rows_per_page: 9999999

        operation:

-         lambda: len([msg for msg in query.all() if msg.msg['new'] is 3])

+         lambda: sum(1 for msg in query.all() if msg.msg['new'] is 3)

        condition:

          greater than or equal to: 1 

  

@@ -31,7 +31,7 @@ 

          - "%(msg.owner)s"  # The user that was created.

          rows_per_page: 9999999

        operation:

-         lambda: len([msg for msg in query.all() if msg.msg['new'] is 3])

+         lambda: sum(1 for msg in query.all() if msg.msg['new'] is 3)

        condition:

          greater than or equal to: 5

  

@@ -31,7 +31,7 @@ 

          - "%(msg.owner)s"  # The user that was created.

          rows_per_page: 9999999

        operation:

-         lambda: len([msg for msg in query.all() if msg.msg['new'] is 3])

+         lambda: sum(1 for msg in query.all() if msg.msg['new'] is 3)

        condition:

          greater than or equal to: 20

  

@@ -31,7 +31,7 @@ 

          - "%(msg.owner)s"  # The user that was created.

          rows_per_page: 9999999

        operation:

-         lambda: len([msg for msg in query.all() if msg.msg['new'] is 3])

+         lambda: sum(1 for msg in query.all() if msg.msg['new'] is 3)

        condition:

          greater than or equal to: 100

  

@@ -31,7 +31,7 @@ 

          - "%(msg.owner)s"  # The user that was created.

          rows_per_page: 9999999

        operation:

-         lambda: len([msg for msg in query.all() if msg.msg['new'] is 3])

+         lambda: sum(1 for msg in query.all() if msg.msg['new'] is 3)

        condition:

          greater than or equal to: 250

  

Since [msg for msg in query.all() if msg.msg['new'] is 1] is
constructing a gigantic list in memory with all decoded json
message, we are hitting memory limit on a regular basis. I suspect
something changed on fedmsg side recently that did start
to make the msg take a lot more memory since fedbadge crash
every 10 to 15 minutes each time a build is made.

Increasing memory is not enough, since that consume more than 20G
(I did test that). So the simplest solution is to not
build a list to count the items, but just count them with sum.

The same pattern should be applied to others rules.

For people who want to see the effect on memory, I did a hotfix on the server, so you can take a look at the graph:

https://admin.fedoraproject.org/collectd/bin/index.cgi?hostname=badges-backend01.phx2.fedoraproject.org&plugin=memory&timespan=3600&action=show_selection&ok_button=OK

Look around 9h on the 31 of march 2020.

Ok so I still see some memory error, but much less. I do wonder if that's caused by others queries or something, as they are also reusing the same pattern.

So, those have been in use since 1 week, I am going to merge

Pull-Request has been merged by misc

2 years ago