@pingou has asked that we prioritize this above any other greenwave/waiverdb issues for the moment.

We should do this as two pull requests imho. First - just do the resultsdb part. Do the waiverdb part in a second PR and release so we can break the problem down.

I'm not quite clear about the last bullet point of solution 2. Please see below.

Any time it sees a message bus message about a new result in resultsdb, it should construct the
cache key used to store results about a given item, and then ask the cache to delete the value
associated with that cache key.

what is the point of asking the cache to delete the value associated with that cache key? If the cache key does not exist, there should be nothing to delete. And as I understand, it would just ask the cache to update the value with the new result for that cache key. Btw, where are we going to cache the results? I guess it might make sense to put the cached results into database so that Greenwave will be able to query the cached results when answering a question?

Any subsequent questions posed to greenwave would force a cache refresh and return >up-to-date decisions.

How could a cache get refreshed? I'm totally lost here and I guess it would be easier to use a real life example to explain this whole idea.

OK - for a start, see #84. I introduced a no-op cache there.

The big missing piece is a fedmsg consumer that would invalidate particular cache keys when resultsdb gets new results. It would look something like this:

def consume(self, msg):
    if is_resultsdb_message(msg):
        task = msg['msg']['task']
        del task['name']
        # here, task is {"item": "nodejs-ansi-black-0.1.1-1.fc28", "type": "koji_build" }
        key = greenwave.cache.cache_key_generator(
            greenwave.resources.retrieve_results, task)
        current_app.cache.delete(key)

I don't want to submit a commit/patch with that yet until #51 is fixed up and merged.. since presumably it is going to introduce a consumer too. I don't want to duplicate work there.

Do you see how it would work?

• At request time, a user requests a decision from greenwave about some subject. Greenwave gets results from resultsdb for some items, and caches them in some dogpile.cache backend (say, on disk or in memcached). Every subsequent request for a decision about the same subject will cause greenwave to pull the results from the cache instead of actually going out to resultsdb.
• Now, when a new result for item=foo-1.2.3 comes in to resultsdb, resultsdb will publish a message to the bus. Our (currently unwritten) CacheInvalidatorConsumer will receive that message, construct the cache key, and delete the old value from the cache if it is there. (If it is not there, then nothing happens). Now, the next request to the greenwave frontend will go to get the cached value from the cache -- but it won't be there, it was deleted. The greenwave frontend should then proceed to actually query resultsdb for the latest set of 1000 results.. and store them in cache for all future queries.
• Rinse and repeat. The backend may invalidate the cache dozens of times for a single item. Every time a decision is requested involving that item, the frontend will re-fill the cache with the latest data.

Thanks @ralph. I'm on board. I will try to fix up #51

OK, I'll get cracking on the invalidator shortly!

Invalidator posted in #86.

I see this is assigned to the milestone 0.2 and I see a 0.3 got released earlier today. Does that mean there is something we can poke at ? :)

Nearly... We have a build done. @mjia was trying to deploy it on stage, but something went wrong in Openshift and we can't see what. I think we need @codeblock to grant us access to the project in Openshift to debug further.

@pingou , Greenwave 0.3 is now in stg and you can try this out there.

https://greenwave-web-greenwave.app.os.stg.fedoraproject.org/api/v1.0/version

Let us know how it goes, thanks.

FYI - I ran a modified version of the script from fedora-infrastructure#6363 against greenwave stg and everything seemed fine (although we still need to hook up a few things before moving anything to prod).