#653 copr-backend-log service should survive `dnf update -y redis`
Closed: Fixed 3 years ago by praiskup. Opened 5 years ago by praiskup.

Our recent ansible task "auto package update" updated redis package on backend, and copr-backend-log died => which in turn brought down the whole copr backend (several hours of not-processing build queue).


Happened again today, queue blocked on backend for several hours...

Upstream documentation clearly says that logging to the same file from multiple
processes isn't supported [1], and right after that mentions several approaches
how to way around this (none of them suggest Redis).

I tend to say that we should migrate to some of those methods, and drop the
non-standard RedisLogger class. My preference is the [2], because we might move
to openshift soon - and we might need some network solution for logging there.

One important thing is that (at least with the [2] solution) we don't have to
take care of serializing/deserializing messages, everything is done by design
in the provided snippet with pickle for free (so we can drop the hacks like
RedisPublishHandler.emit(), and friends).

[1] https://docs.python.org/3/howto/logging-cookbook.html#logging-to-a-single-file-from-multiple-processes
[2] https://docs.python.org/3/howto/logging-cookbook.html#sending-and-receiving-logging-events-across-a-network

Metadata Update from @praiskup:
- Issue untagged with: ansible

4 years ago
● postgresql.service - PostgreSQL database server
   Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2019-08-20 11:06:11 UTC; 1 months 19 days ago
 Main PID: 26998 (postmaster)
    Tasks: 8 (limit: 4915)
   Memory: 37.6M
   CGroup: /system.slice/postgresql.service
           ├─26998 /usr/bin/postmaster -D /var/lib/pgsql/data
           ├─26999 postgres: logger   
           ├─27001 postgres: checkpointer   
           ├─27002 postgres: background writer   
           ├─27003 postgres: walwriter   
           ├─27004 postgres: autovacuum launcher   
           ├─27005 postgres: stats collector   
           └─27006 postgres: logical replication launcher   

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

I haven't looked at this, so I'm asking anyone (including myself in future): Is the message caused by logrotate? Can this cause some race/offline window in redis (systemctl redis status showed the same output)?

There was some issue with logging which caused that copr-backend-action went down..., and I suspect this is related, dunno atm.
Because it seems like there was some issue with logging

Metadata Update from @praiskup:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

3 years ago

Login to comment on this ticket.

Metadata
Related Pull Requests
  • #1058 Merged 4 years ago