#8186 Fedora wiki gateway timeout
Closed: Will Not/Can Not fix 18 days ago by kevin. Opened a month ago by pnemade.

Really this happens always when I needed to create a meeting agenda page. Irrespective of I am using VPN or not, I always get timeout errors. Today I tried from Firefox as well as Chromium browser multiple times but always got
"Gateway Timeout
The gateway did not receive a timely response from the upstream server or application."

And now while reporting this ticket IRC notif came that page https://fedoraproject.org/wiki/I18N/Meetings/2019-09-10 has been created. I tried for 15 min to commit this initial version of page.

I am very much frustrated and not sure where to look for help hence as a first step reporting here.


This is almost surely due to database backups. We had this issue in the past and we moved the database backups to a different time, but it seems now that time is bad for you. ;(

I do see we are doing a database backup still of a large db we don't need anymore, so I disabled that from backups.

Can you try again around the same time tomorrow and see if it's working any better?

Metadata Update from @kevin:
- Issue priority set to: Waiting on Reporter (was: Needs Review)

a month ago

Thank you for your explanation. Yes today I tried to update same wiki page and it worked fine. You can close this ticket.

Metadata Update from @pingou:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

a month ago

So this started again today and I could not update the wiki page for 5 times. That update is still pending. I am re-opening this to understand what time I should avoid daily. I will never attempt to commit anything in Fedora wiki pages during that time. I faced wiki update issue at around 5.00 AM UTC time.

Metadata Update from @pnemade:
- Issue status updated to: Open (was: Closed)

a month ago

I found that either 5th attempt or 1st attempt only worked but did not return me committed wiki page but instead keep giving me timeout errors. Next time this error occurs I will cross check by reopening same page in another tab.
I come to know this only when I cross-checked my datagrepper profile which showed wiki page update did happen.

The db backup starts at 05:00 UTC. Looks like it finishes about 05:30 UTC.

Would moving it a hour earlier or later help ? or two hours?

No you please continue having your own timings set already. I don't do much work on Fedora wiki but when I need to I usually edits 2-3 times only. So for me please don't change your already regular timings. I can be adjustable here. I will simply avoid editing around that timing :)

Thank you Kevin.

At a $previous_employer, we set up a database replica so that we didn't have to lock the production database during backups (because backups took a long time). The replica would lock during the backup, and then when the backup was done it would replay the logs and catch up to production. It worked pretty well. We also sometimes used the replica to do analytics too to keep load off of production.

No you please continue having your own timings set already. I don't do much work on Fedora wiki but when I need to I usually edits 2-3 times only. So for me please don't change your already regular timings. I can be adjustable here. I will simply avoid editing around that timing :)
Thank you Kevin.

I've tried adding a ionice call to it to see if that lets it process while the backup is running. Can you try testing it at 05-05:30 and see if it's any more responsive...?

At a $previous_employer, we set up a database replica so that we didn't have to lock the production database during backups (because backups took a long time). The replica would lock during the backup, and then when the backup was done it would replay the logs and catch up to production. It worked pretty well. We also sometimes used the replica to do analytics too to keep load off of production.

Well, the wiki and paste are the only things left on mariadb, so I don't think we want to do a cluster just for that. Also, it's not clear to me that it's locking, I think it's just doing so much IO that the web part becomes unresponsive...

I start creating new meeting wiki page from 05:11 AM UTC to 05:17 AM UTC. Attempted 3 times to create page, all the time it failed with "gateway timeout".

I will attempt to create this wiki page later today.

I guess the backup window was 0500 to 0530 hrs in which I attempted 3 times. I checked by opening new tab whether page has been created already, but it was not. The moment its 05:30 hrs, IRC fedora-notif informed me page has been created. I checked that link and yes the wiki page appeared.

Then I did edit the meeting wiki page and it got edited quickly. so only time it does not respond back quickly was in between 0500 to 0530 hrs.

Alright then, I guess it's just not going to be very usable during that 30min. ;(

We can move it around if needed, or look more if it's causing more problems. In the mean time I will close this out, but feel free to reopen if it becomes more of a problem.

Metadata Update from @kevin:
- Issue close_status updated to: Will Not/Can Not fix
- Issue status updated to: Closed (was: Open)

18 days ago

Login to comment on this ticket.

Metadata