#9355 Planned Outage - pagure.io - 2020-10-01 08:00 UTC
Closed: Fixed 3 years ago by pingou. Opened 3 years ago by pingou.

Planned Outage - pagure.io - 2020-10-01 08:00 UTC

There will be an outage starting at 2020-10-01 08:00 UTC,
which will last approximately 4 hours.

To convert UTC to your local time, take a look at
http://fedoraproject.org/wiki/Infrastructure/UTCHowto
or run:

date -d '2020-10-01 08:00 UTC'

Reason for outage:

We are moving the service to a new server running RHEL8 and python3.

Affected Services:

pagure.io
pagure.org

Ticket Link:

https://pagure.io/fedora-infrastructure/issue/9355

Please join #fedora-admin or #fedora-noc on irc.freenode.net
or add comments to the ticket for this outage above.


Metadata Update from @pingou:
- Issue tagged with: outage

3 years ago

To do this we need to:

  • Stop apache and all pagure* services on pagure01
  • rsync all of /srv from pagure01 to pagure02
  • rsync all of /var/www from pagure01 to pagure02
  • Dump the postgresql database in pagure01
  • Drop the pagure DB in pagure02
  • Load the postgresql dump from pagure01 into the DB server on pagure02
  • Update the DNS (more precisely: pagure.io.template and master/pagure.org) to reflect the new IP

A potential way to prevent git push via ssh while we do the migration:

        diff --git a/inventory/group_vars/pagure b/inventory/group_vars/pagure
        index 5f6c22b29..53acfea67 100644
        --- a/inventory/group_vars/pagure
        +++ b/inventory/group_vars/pagure
        @@ -8,12 +8,14 @@ num_cpus: 16
         # for systems that do not match the above - specify the same parameter in
         # the host_vars/$hostname file

        -tcp_ports: [ 22, 25, 80, 443,
        -    8442, 8443, 8444, 8445,
        +#tcp_ports: [ 22, 25, 80, 443,
        +#    8442, 8443, 8444, 8445,
             # Used for the eventsource
        -    8088,
        +#    8088,
             # This is for the pagure public fedmsg relay
        -    9940]
        +#    9940]
        +
        +custom_rules: [ '-A INPUT -p tcp -m tcp -s 38.145.60.16 --dport 22 -j ACCEPT'

         stunnel_service: "eventsource"
         stunnel_source_port: 8088

Thanks @kevin for the diff ^ :)

Looks ok to me. I guess send to devel-announce a copy ? Or devel-announce and announce?

Or devel-announce and announce?

I've sent it to both and approved it on devel-announce. I let the moderators of announce decide if this is worthy of that list :)

Thanks for the review!

Metadata Update from @mobrien:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: high-gain, medium-trouble, ops

3 years ago

Download everything you need guys.

Is this warning I get from the dist-git server when pushing a commit related?

remote: --- Logging error ---
remote: Traceback (most recent call last):
remote:   File "/usr/lib64/python3.6/logging/handlers.py", line 72, in emit
remote:     self.doRollover()
remote:   File "/usr/lib64/python3.6/logging/handlers.py", line 396, in doRollover
remote:     self.rotate(self.baseFilename, dfn)
remote:   File "/usr/lib64/python3.6/logging/handlers.py", line 113, in rotate
remote:     os.rename(source, dest)
remote: PermissionError: [Errno 13] Permission denied: '/var/log/pagure/pagure_auth.log' -> '/var/log/pagure/pagure_auth.log.2020-09-30'
remote: Call stack:
remote:   File "hooks/pre-receive", line 48, in <module>
remote:     run_hook_file(hooktype)
remote:   File "/usr/lib/python3.6/site-packages/pagure/hooks/__init__.py", line 547, in run_hook_file
remote:     pull_request,
remote:   File "/usr/lib/python3.6/site-packages/pagure/hooks/__init__.py", line 323, in run_project_hooks
remote:     authbackend = get_git_auth_helper()
remote:   File "/usr/lib/python3.6/site-packages/pagure/lib/git_auth.py", line 66, in get_git_auth_helper
remote:     _log.info("Looking for backend: %s", backend)
remote: Message: 'Looking for backend: %s'
remote: Arguments: ('distgit',)

@ppisar not at all, this outage concerns only pagure.io not dist-git.

I've tried to fix the error (at least the first one), feel free to ping me on IRC if it persists, I'm going to stop pagure.io for the migration.

Test comment for switchover

This outage should now be complete with pagure.io/pagure.org now on a RHEL8 server with pythyon3 please raise any issues should they occur.

Metadata Update from @pingou:
- Issue assigned to pingou

3 years ago

Issue status updated to: Closed (was: Open)
Issue close_status updated to: Fixed

3 years ago

Login to comment on this ticket.

Metadata
Boards 1
ops Status: Done