Websites strongly recommend to archive old messages ASAP, as we are using datagrepper for many things in getfedora.org. Unfortunately the build fails too often, just because datagrepper is not talking to the web builder (500).

Also, developing websites becomes very annoying, because it takes minutes to make a local build (when it has success).
Example:

https://apps.fedoraproject.org/datagrepper/raw?topic=org.fedoraproject.prod.releng.atomic.twoweek.complete&start=14
41402109&rows_per_page=1&page=1

The link works in a browser, but is too slow when building the website.

Could we use statscache here? have it cache this query?

Strangely, this query seems to take 2 seconds here... perhaps it's cached somehow right now?

It would be nice to still be able to query the older data if you were not in a hurry/wanted complete answers. Perhaps we could do some performance tuning here...

Yes, we did performance tuning (ralph did) on datagrepper, and we also cache the data. But the first time you build it locally, or if the cache is too old, you rebuild it. For developing we do not need up to date data indeed, right.
Maxamillion did a change on our script and we now call datagrepper less often, and it's working better now. Until today, because....

"File "build/atomic_vars.py", line 77, in get_page
raise IOError("Failed to talk to %r %r" % (response.url, response))
IOError: Failed to talk to u'https://apps.fedoraproject.org/datagrepper/raw?topic=org.fedoraproject.prod.releng.atomic.twoweek.complete&start=1441402109&rows_per_page=1&page=1' "

huh, is that actually a timeout there? or is it something else happening on the datagrepper side?

A 500 error sounds like something different than just a time out issue.

Accessing the URL by end seems to return a traceback with:

File \"/usr/lib/python2.7/site-packages/datagrepper/app.py\", line 228, in raw
 page = int(flask.request.args.get('page', 1))
ValueError: invalid literal for int() with base 10: \"1'\"\n"}

Hm, nevermind, opening the URL in pagure lead to a trailing ' to be added to the url which lead to the error above

Accessing https://apps.fedoraproject.org/datagrepper/raw?topic=org.fedoraproject.prod.releng.atomic.twoweek.complete&start=1441402109&rows_per_page=1&page=1 by hand takes a while here but returns a 200 OK.

The 500 is when mod_wsgi aborts the process for too long of a wait.

I think that the website stuff needs to be ported to statscache, as that's an ideal place to cache the last of some kind of message.

Yes, opening it by hand takes a while, but works. Probably Kevin and Patrick are right, it would be the best to port it to statscache.

So currently this query takes like 1.3 seconds.

Do we still need to do something here?

We added a bunch of indexes and other tweaks and I think things are going ok for now.

We can revisit the archiving if we need to down the road.

We are also going to look at moving this database to ssd's at some point which might help it for a while longer at least.

:minidisc: