When we do several test days shortly afterwards, we often see a problem that our wiki redirects [1] return an old content randomly. Some users are redirected to the current test day, some users are redirected to an old (yesterday's) test day.
I've been trying to track down the issue and I've find out that it depends which proxy server is (randomly) chosen to return the wiki content.
This is a series of commands executed immediately one after each other:
{{{ $ wget -S 'https://fedoraproject.org/wiki/Test_Day:Current' 2>&1 | grep -E '(Location|Proxy)' Location: https://fedoraproject.org/wiki/Test_Day:2013-04-18_FreeIPA?rd=Test_Day:Current ProxyTime: D=1013 ProxyServer: proxy03.fedoraproject.org Location: https://fedoraproject.org/wiki/Test_Day:2013-04-18_FreeIPA?rd=Test_Day:Current [following] ProxyTime: D=949 ProxyServer: proxy03.fedoraproject.org $ wget -S 'https://fedoraproject.org/wiki/Test_Day:Current' 2>&1 | grep -E '(Location|Proxy)' Location: https://fedoraproject.org/wiki/Test_Day:2013-04-17_Power_Management?rd=Test_Day:Current ProxyTime: D=1106 ProxyServer: proxy01.phx2.fedoraproject.org Location: https://fedoraproject.org/wiki/Test_Day:2013-04-17_Power_Management?rd=Test_Day:Current [following] ProxyTime: D=1157 ProxyServer: proxy01.phx2.fedoraproject.org $ wget -S 'https://fedoraproject.org/wiki/Test_Day:Current' 2>&1 | grep -E '(Location|Proxy)' Location: https://fedoraproject.org/wiki/Test_Day:2013-04-17_Power_Management?rd=Test_Day:Current ProxyTime: D=553 ProxyServer: proxy04.fedoraproject.org Location: https://fedoraproject.org/wiki/Test_Day:2013-04-17_Power_Management?rd=Test_Day:Current [following] ProxyTime: D=709 ProxyServer: proxy04.fedoraproject.org $ wget -S 'https://fedoraproject.org/wiki/Test_Day:Current' 2>&1 | grep -E '(Location|Proxy)' Location: https://fedoraproject.org/wiki/Test_Day:2013-04-17_Power_Management?rd=Test_Day:Current ProxyTime: D=1002 ProxyServer: proxy06.fedoraproject.org Location: https://fedoraproject.org/wiki/Test_Day:2013-04-17_Power_Management?rd=Test_Day:Current [following] ProxyTime: D=964 ProxyServer: proxy06.fedoraproject.org $ wget -S 'https://fedoraproject.org/wiki/Test_Day:Current' 2>&1 | grep -E '(Location|Proxy)' Location: https://fedoraproject.org/wiki/Test_Day:2013-04-17_Power_Management?rd=Test_Day:Current ProxyTime: D=887 ProxyServer: proxy04.fedoraproject.org Location: https://fedoraproject.org/wiki/Test_Day:2013-04-17_Power_Management?rd=Test_Day:Current [following] ProxyTime: D=965 ProxyServer: proxy04.fedoraproject.org $ wget -S 'https://fedoraproject.org/wiki/Test_Day:Current' 2>&1 | grep -E '(Location|Proxy)' Location: https://fedoraproject.org/wiki/Test_Day:2013-04-18_FreeIPA?rd=Test_Day:Current ProxyTime: D=1286 ProxyServer: proxy03.fedoraproject.org Location: https://fedoraproject.org/wiki/Test_Day:2013-04-18_FreeIPA?rd=Test_Day:Current [following] ProxyTime: D=1130 ProxyServer: proxy03.fedoraproject.org $ wget -S 'https://fedoraproject.org/wiki/Test_Day:Current' 2>&1 | grep -E '(Location|Proxy)' Location: https://fedoraproject.org/wiki/Test_Day:2013-04-17_Power_Management?rd=Test_Day:Current ProxyTime: D=1252 ProxyServer: proxy01.phx2.fedoraproject.org Location: https://fedoraproject.org/wiki/Test_Day:2013-04-17_Power_Management?rd=Test_Day:Current [following] ProxyTime: D=799 ProxyServer: proxy01.phx2.fedoraproject.org $ wget -S 'https://fedoraproject.org/wiki/Test_Day:Current' 2>&1 | grep -E '(Location|Proxy)' Location: https://fedoraproject.org/wiki/Test_Day:2013-04-18_FreeIPA?rd=Test_Day:Current ProxyTime: D=1046 ProxyServer: proxy08.fedoraproject.org Location: https://fedoraproject.org/wiki/Test_Day:2013-04-18_FreeIPA?rd=Test_Day:Current [following] ProxyTime: D=1239 ProxyServer: proxy08.fedoraproject.org }}}
As you can see, at that time, these proxies were returning a current content: proxy03.fedoraproject.org proxy08.fedoraproject.org
And these proxies were returning an outdated content: proxy01.phx2.fedoraproject.org proxy04.fedoraproject.org * proxy06.fedoraproject.org
This is not just a problem with wiki redirects, but generally with any wiki content. It often happens that I edit a wiki content, but my colleague still sees the old content until he hits Ctrl+F5 in Firefox to refresh the page without cache.
Can we make sure all the proxies serve current content? Why are some of them up-to-date and some of them outdated?
[1] https://fedoraproject.org/wiki/Test_Day:Current
Yeah, so this is very likely caching we have in place... it could be 2 possible places. Either varnish (on the proxies) or memcached (which the wiki on our app servers uses).
Could you relpeat your test and also add "app" to the grep? It should have a header showing which particular app server processed the request:
AppServer: app04.phx2.fedoraproject.org
Are all the requests that show old data on any particular app server? If so, thats pointing to an issue with a particular app server/memcached. If not, then it's varnish...
Also, in mediawiki you can manually tell it to purge the memcached cache for a page, just add '?action=purge' to the end of the page url and it should flush any cache it has.
http://example.org/wiki/Main_Page?action=purge
Also, please add the Varnish headers, as they will show us if it was cached content being returned.
Sorry, I did not receive any email about your comments, so I'm replying late. I just changed Test_Day:Current contents, here we go:
{{{ $ wget -S 'https://fedoraproject.org/wiki/Test_Day:Current' 2>&1 | grep -E '(Location|Proxy|App|Varnish)' Location: https://fedoraproject.org/wiki/Test_Day:2013-06-06_FreeIPA_Two_Factor_Authentication?rd=Test_Day:Current AppTime: D=232571 AppServer: app01.phx2.fedoraproject.org X-Varnish: 1479552164 1479552013 ProxyTime: D=1260 ProxyServer: proxy09.fedoraproject.org Location: https://fedoraproject.org/wiki/Test_Day:2013-06-06_FreeIPA_Two_Factor_Authentication?rd=Test_Day:Current [following] AppTime: D=221575 AppServer: app02.phx2.fedoraproject.org X-Varnish: 1479552166 1479552014 ProxyTime: D=952 ProxyServer: proxy09.fedoraproject.org $ wget -S 'https://fedoraproject.org/wiki/Test_Day:Current' 2>&1 | grep -E '(Location|Proxy|App|Varnish)' Location: https://fedoraproject.org/wiki/Test_Day:2013-06-06_FreeIPA_Two_Factor_Authentication?rd=Test_Day:Current AppTime: D=251166 AppServer: app02.phx2.fedoraproject.org X-Varnish: 1982866583 1982866323 ProxyTime: D=1223 ProxyServer: proxy03.fedoraproject.org Location: https://fedoraproject.org/wiki/Test_Day:2013-06-06_FreeIPA_Two_Factor_Authentication?rd=Test_Day:Current [following] AppTime: D=178417 AppServer: app04.phx2.fedoraproject.org X-Varnish: 1982866585 1982866326 ProxyTime: D=992 ProxyServer: proxy03.fedoraproject.org $ wget -S 'https://fedoraproject.org/wiki/Test_Day:Current' 2>&1 | grep -E '(Location|Proxy|App|Varnish)' Location: https://fedoraproject.org/wiki/Test_Day:2013-06-04_Network_Manager?rd=Test_Day:Current AppTime: D=222456 AppServer: app02.phx2.fedoraproject.org X-Varnish: 1394660133 1394657437 ProxyTime: D=1004 ProxyServer: proxy06.fedoraproject.org Location: https://fedoraproject.org/wiki/Test_Day:2013-06-04_Network_Manager?rd=Test_Day:Current [following] AppTime: D=203923 AppServer: app03.phx2.fedoraproject.org X-Varnish: 1394660135 1394659945 ProxyTime: D=2148 ProxyServer: proxy06.fedoraproject.org $ wget -S 'https://fedoraproject.org/wiki/Test_Day:Current' 2>&1 | grep -E '(Location|Proxy|App|Varnish)' Location: https://fedoraproject.org/wiki/Test_Day:2013-06-04_Network_Manager?rd=Test_Day:Current AppTime: D=222456 AppServer: app02.phx2.fedoraproject.org X-Varnish: 1394660182 1394657437 ProxyTime: D=997 ProxyServer: proxy06.fedoraproject.org Location: https://fedoraproject.org/wiki/Test_Day:2013-06-04_Network_Manager?rd=Test_Day:Current [following] AppTime: D=203923 AppServer: app03.phx2.fedoraproject.org X-Varnish: 1394660184 1394659945 ProxyTime: D=984 ProxyServer: proxy06.fedoraproject.org $ wget -S 'https://fedoraproject.org/wiki/Test_Day:Current' 2>&1 | grep -E '(Location|Proxy|App|Varnish)' Location: https://fedoraproject.org/wiki/Test_Day:2013-06-06_FreeIPA_Two_Factor_Authentication?rd=Test_Day:Current AppTime: D=235654 AppServer: app01.phx2.fedoraproject.org X-Varnish: 701077009 ProxyTime: D=289580 ProxyServer: proxy08.fedoraproject.org Location: https://fedoraproject.org/wiki/Test_Day:2013-06-06_FreeIPA_Two_Factor_Authentication?rd=Test_Day:Current [following] AppTime: D=202226 AppServer: app02.phx2.fedoraproject.org X-Varnish: 701077010 ProxyTime: D=255810 ProxyServer: proxy08.fedoraproject.org $ wget -S 'https://fedoraproject.org/wiki/Test_Day:Current' 2>&1 | grep -E '(Location|Proxy|App|Varnish)' Location: https://fedoraproject.org/wiki/Test_Day:2013-06-06_FreeIPA_Two_Factor_Authentication?rd=Test_Day:Current AppTime: D=232571 AppServer: app01.phx2.fedoraproject.org X-Varnish: 1479552245 1479552013 ProxyTime: D=933 ProxyServer: proxy09.fedoraproject.org Location: https://fedoraproject.org/wiki/Test_Day:2013-06-06_FreeIPA_Two_Factor_Authentication?rd=Test_Day:Current [following] AppTime: D=221575 AppServer: app02.phx2.fedoraproject.org X-Varnish: 1479552246 1479552014 ProxyTime: D=1003 ProxyServer: proxy09.fedoraproject.org $ wget -S 'https://fedoraproject.org/wiki/Test_Day:Current' 2>&1 | grep -E '(Location|Proxy|App|Varnish)' Location: https://fedoraproject.org/wiki/Test_Day:2013-06-06_FreeIPA_Two_Factor_Authentication?rd=Test_Day:Current AppTime: D=251166 AppServer: app02.phx2.fedoraproject.org X-Varnish: 1982866665 1982866323 ProxyTime: D=984 ProxyServer: proxy03.fedoraproject.org Location: https://fedoraproject.org/wiki/Test_Day:2013-06-06_FreeIPA_Two_Factor_Authentication?rd=Test_Day:Current [following] AppTime: D=178417 AppServer: app04.phx2.fedoraproject.org X-Varnish: 1982866666 1982866326 ProxyTime: D=1078 ProxyServer: proxy03.fedoraproject.org }}}
It seems that app02 sometimes redirects correctly and sometimes does not.
Opening https://fedoraproject.org/wiki/Test_Day:Current?action=purge (even with &redirect=no) did not help, the redirect still returns an old content from time to time.
The problem is proxy06: both requests that hit proxy06 returned the old info, the others returned fresh info. Also, since with both those requests the varnish which-request-created-the-cache entry is the same, I think varnish is misbehaving.
Thanks for notifying us and providing info, I will take this into my next batch of varnish configurations.
Also, please note that these problems will not occur if the user is logged in, because varnish does not cache any pages that are requested by logged in users, nor will it return cached pages to logged in users.
This issue is also visible in the "Random page" link.
I will reconfigure varnish to not cache any page content after the freeze is lifted.
This should be fixed now.
It seems that one of the proxy servers is still broken, proxy02.fp.o:
This is OK: {{{ $ wget -S 'https://fedoraproject.org/wiki/Test_Day:Current' 2>&1 | grep -E '(Location|Proxy|App|Varnish)' Location: https://fedoraproject.org/wiki/Test_Day:2013-10-10_Gnome_3.10?rd=Test_Day:Current AppTime: D=248238 AppServer: app04.phx2.fedoraproject.org X-Varnish: 1314475389 1314471012 ProxyTime: D=1114 ProxyServer: proxy08.fedoraproject.org Location: https://fedoraproject.org/wiki/Test_Day:2013-10-10_Gnome_3.10?rd=Test_Day:Current [following] AppTime: D=241462 AppServer: app03.phx2.fedoraproject.org X-Varnish: 1314475392 1314472986 ProxyTime: D=931 ProxyServer: proxy08.fedoraproject.org }}}
This is not OK: {{{ $ wget -S 'https://fedoraproject.org/wiki/Test_Day:Current' 2>&1 | grep -E '(Location|Proxy|App|Varnish)' Location: https://fedoraproject.org/wiki/Test_Day:2013-06-06_FreeIPA_Two_Factor_Authentication?rd=Test_Day:Current AppTime: D=233914 AppServer: app04.phx2.fedoraproject.org X-Varnish: 1141624043 1141578733 ProxyTime: D=849 ProxyServer: proxy02.fedoraproject.org Location: https://fedoraproject.org/wiki/Test_Day:2013-06-06_FreeIPA_Two_Factor_Authentication?rd=Test_Day:Current [following] AppTime: D=235349 AppServer: app01.phx2.fedoraproject.org X-Varnish: 1141624044 1141623728 ProxyTime: D=478 ProxyServer: proxy02.fedoraproject.org }}}
Do you still see this?
My theory is that this happens when one of the proxies is off on time...
Can you reopen this if/when you see it again?
We have all the proxies set on ntp now and they all should be caching correctly.
The problem is here again, some proxy servers are not working properly:
{{{
$ wget -S 'https://fedoraproject.org/wiki/Test_Results:Current_Installation_Test' 2>&1 | grep -E '(Location|Proxy|App|Varnish)' Location: https://fedoraproject.org/wiki/Test_Results:Fedora_23_Beta_TC5_Installation?rd=Test_Results:Current_Installation_Test AppTime: D=678 AppServer: proxy02.fedoraproject.org X-Varnish: 8879249 7830368 Location: https://fedoraproject.org/wiki/Test_Results:Fedora_23_Beta_TC5_Installation?rd=Test_Results:Current_Installation_Test [following] AppTime: D=988 AppServer: proxy02.fedoraproject.org X-Varnish: 7028362 9222952
$ wget -S 'https://fedoraproject.org/wiki/Test_Results:Current_Installation_Test' 2>&1 | grep -E '(Location|Proxy|App|Varnish)' Location: https://fedoraproject.org/wiki/Test_Results:Fedora_23_Beta_TC4_Installation?rd=Test_Results:Current_Installation_Test AppTime: D=561 AppServer: proxy05.fedoraproject.org X-Varnish: 5442506 4328399 Location: https://fedoraproject.org/wiki/Test_Results:Fedora_23_Beta_TC4_Installation?rd=Test_Results:Current_Installation_Test [following] AppTime: D=5243 AppServer: proxy05.fedoraproject.org X-Varnish: 4977908 3534572
$ wget -S 'https://fedoraproject.org/wiki/Test_Results:Current_Installation_Test' 2>&1 | grep -E '(Location|Proxy|App|Varnish)' Location: https://fedoraproject.org/wiki/Test_Results:Fedora_23_Beta_RC1_Installation?rd=Test_Results:Current_Installation_Test AppTime: D=36024 AppServer: proxy11.fedoraproject.org X-Varnish: 11010061 10171613 Location: https://fedoraproject.org/wiki/Test_Results:Fedora_23_Beta_RC1_Installation?rd=Test_Results:Current_Installation_Test [following] AppTime: D=11308 AppServer: proxy11.fedoraproject.org X-Varnish: 9980254 9819284
$ wget -S 'https://fedoraproject.org/wiki/Test_Results:Current_Installation_Test' 2>&1 | grep -E '(Location|Proxy|App|Varnish)' Location: https://fedoraproject.org/wiki/Test_Results:Fedora_23_Beta_RC1_Installation?rd=Test_Results:Current_Installation_Test AppTime: D=758 AppServer: proxy07.fedoraproject.org X-Varnish: 1115626 524364 Location: https://fedoraproject.org/wiki/Test_Results:Fedora_23_Beta_RC1_Installation?rd=Test_Results:Current_Installation_Test [following] AppTime: D=767 AppServer: proxy07.fedoraproject.org X-Varnish: 361936 459027
$ wget -S 'https://fedoraproject.org/wiki/Test_Results:Current_Installation_Test' 2>&1 | grep -E '(Location|Proxy|App|Varnish)' Location: https://fedoraproject.org/wiki/Test_Results:Fedora_23_Beta_TC5_Installation?rd=Test_Results:Current_Installation_Test AppTime: D=666 AppServer: proxy10.phx2.fedoraproject.org X-Varnish: 5175513 4431183 Location: https://fedoraproject.org/wiki/Test_Results:Fedora_23_Beta_TC5_Installation?rd=Test_Results:Current_Installation_Test [following] AppTime: D=11048 AppServer: proxy10.phx2.fedoraproject.org X-Varnish: 4315255 4044461 }}}
It seems that servers proxy07.fp.o and proxy11.fp.o are working as expected, while proxy02.fp.o, proxy05.fp.o and proxy10.fp.o redirect to wrong page.
The caches have been cleared forcible, resolving this case.
I think this was meant to be closed... ;)
Login to comment on this ticket.