Learn more about these different git repos.
Other Git URLs
Internal copr failed this week several times:
Oct 10 19:59:53 coprbe.devel.redhat.com systemd[1]: Started Copr Backend service, Build Dispatcher component. Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: Failed to publish log record to redis, Traceback (most recent call last): Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: File "/usr/lib/python3.7/site-packages/redis/connection.py", line 181, in _read_from_socket Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: data = recv(self._sock, socket_read_size) Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: File "/usr/lib/python3.7/site-packages/redis/_compat.py", line 71, in recv Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: return sock.recv(*args, **kwargs) Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: ConnectionResetError: [Errno 104] Connection reset by peer Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: During handling of the above exception, another exception occurred: Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: Traceback (most recent call last): Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: File "/usr/share/copr/backend/helpers.py", line 414, in emit Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: self.rc.publish(constants.LOG_PUB_SUB, json.dumps(record.__dict__)) Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: File "/usr/lib/python3.7/site-packages/redis/client.py", line 2716, in publish Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: return self.execute_command('PUBLISH', channel, message) Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: File "/usr/lib/python3.7/site-packages/redis/client.py", line 775, in execute_command Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: return self.parse_response(connection, command_name, **options) Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: File "/usr/lib/python3.7/site-packages/redis/client.py", line 789, in parse_response Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: response = connection.read_response() Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: File "/usr/lib/python3.7/site-packages/redis/connection.py", line 636, in read_response Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: raise e Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: File "/usr/lib/python3.7/site-packages/redis/connection.py", line 633, in read_response Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: response = self._parser.read_response() Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: File "/usr/lib/python3.7/site-packages/redis/connection.py", line 291, in read_response Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: response = self._buffer.readline() Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: File "/usr/lib/python3.7/site-packages/redis/connection.py", line 223, in readline Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: self._read_from_socket() Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: File "/usr/lib/python3.7/site-packages/redis/connection.py", line 198, in _read_from_socket Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: (e.args,)) Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: redis.exceptions.ConnectionError: Error while reading from socket: (104, 'Connection reset by peer') Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: Traceback (most recent call last): Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: File "/usr/lib/python3.7/site-packages/redis/connection.py", line 181, in _read_from_socket Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: data = recv(self._sock, socket_read_size) Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: File "/usr/lib/python3.7/site-packages/redis/_compat.py", line 71, in recv Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: return sock.recv(*args, **kwargs) Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: ConnectionResetError: [Errno 104] Connection reset by peer Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: During handling of the above exception, another exception occurred: Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: Traceback (most recent call last): Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: File "/usr/bin/copr_run_build_dispatcher.py", line 12, in <module> Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: main() Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: File "/usr/bin/copr_run_build_dispatcher.py", line 9, in main Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: build_dispatcher.run() Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: File "/usr/share/copr/backend/daemons/build_dispatcher.py", line 195, in run Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: job.chroot) Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: File "/usr/share/copr/backend/vm_manage/manager.py", line 228, in acquire_vm Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: ready_vmd_list = self.get_ready_vms(group) Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: File "/usr/share/copr/backend/vm_manage/manager.py", line 209, in get_ready_vms Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: vmd_list = self.get_all_vm_in_group(group) Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: File "/usr/share/copr/backend/vm_manage/manager.py", line 324, in get_all_vm_in_group Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: vm_name_list = self.rc.smembers(KEY_VM_POOL.format(group=group)) Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: File "/usr/lib/python3.7/site-packages/redis/client.py", line 1916, in smembers Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: return self.execute_command('SMEMBERS', name) Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: File "/usr/lib/python3.7/site-packages/redis/client.py", line 775, in execute_command Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: return self.parse_response(connection, command_name, **options) Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: File "/usr/lib/python3.7/site-packages/redis/client.py", line 789, in parse_response Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: response = connection.read_response() Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: File "/usr/lib/python3.7/site-packages/redis/connection.py", line 636, in read_response Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: raise e Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: File "/usr/lib/python3.7/site-packages/redis/connection.py", line 633, in read_response Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: response = self._parser.read_response() Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: File "/usr/lib/python3.7/site-packages/redis/connection.py", line 291, in read_response Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: response = self._buffer.readline() Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: File "/usr/lib/python3.7/site-packages/redis/connection.py", line 223, in readline Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: self._read_from_socket() Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: File "/usr/lib/python3.7/site-packages/redis/connection.py", line 198, in _read_from_socket Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: (e.args,)) Oct 10 23:12:28 coprbe.devel.redhat.com copr_run_build_dispatcher.py[20283]: redis.exceptions.ConnectionError: Error while reading from socket: (104, 'Connection reset by peer') Oct 10 23:12:28 coprbe.devel.redhat.com systemd[1]: copr-backend-build.service: Main process exited, code=exited, status=1/FAILURE Oct 10 23:12:28 coprbe.devel.redhat.com systemd[1]: copr-backend-build.service: Failed with result 'exit-code'. Oct 11 05:08:47 coprbe.devel.redhat.com systemd[1]: Started Copr Backend service, Build Dispatcher component.
Very similar issue on Frontend, I'm curious whether there's some redis configuration problem? We use the default config file..
Main process exited, code=killed, status=9/KILL
Ah, looking at the log - this looks like OOM: Main process exited, code=killed, status=9/KILL
But the messaging traceback is ugly anyways (there are two servers, one and fallback ... and this is the fallback scenario).
Any ideas?
Ok, it really seems to be memory issue.
Metadata Update from @praiskup: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.