Hey!
A couple of hours ago I noticed that I can't sync any larger artifacts to the artifacts box (artifacts.ci.centos.org), even though it worked fine this morning:
artifacts.ci.centos.org
# hostname n25.crusty.ci.centos.org # cat /etc/centos-release CentOS Stream release 8 # cut -b-13 /duffy.key >rsync.key # chmod o-rwx rsync.key # dd if=/dev/zero of=test.img bs=1M count=128 # rsync -vvv --password-file=rsync.key -av test.img systemd@artifacts.ci.centos.org::systemd/ opening tcp connection to artifacts.ci.centos.org port 873 Connected to artifacts.ci.centos.org (172.19.0.7) msg checking charset: UTF-8 sending daemon args: --server -vvvvlogDtpre.iLsfxC --sparse-block=1024 . systemd/ (5 args) (Client) Protocol versions: remote=31, negotiated=31 sending incremental file list [sender] make_file(test.img,*,0) [sender] flist start=1, used=1, low=0, high=0 [sender] i=1 <NULL> test.img mode=0100644 len=134,217,728 uid=0 gid=0 flags=1005 send_file_list done [sender] flist_eof=1 file list sent send_files starting rsync: read error: Connection reset by peer (104) [sender] _exit_cleanup(code=10, file=io.c, line=785): entered rsync error: error in socket IO (code 10) at io.c(785) [sender=3.1.3] [sender] _exit_cleanup(code=10, file=io.c, line=785): about to call exit(10)
In one of the jobs I noticed this error:
Error: rsync: on remote machine: --sparse-block=1024: unknown option rsync error: syntax or usage error (code 1) at main.c(1765) [server=3.2.4] rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(226) [sender=3.1.3]
which might point at the root cause.
This seems to happen only with the latest version of rsync (rsync-3.1.3-16.el8.x86_64), since after downgrading to rsync-3.1.3-14.el8.2.x86_64 everything works as it should.
rsync
rsync-3.1.3-16.el8.x86_64
rsync-3.1.3-14.el8.2.x86_64
Hmpf, I guess this is a bug in rsync then, since the --sparse-block= option was introduced by https://gitlab.com/redhat/centos-stream/rpms/rsync/-/commit/e7590313b78c1377ff18a975da6cc89a04e9c1d2#6706bd27bc5ff12e5e052349fc4a8311587720c0_76_77 as a response to https://bugzilla.redhat.com/show_bug.cgi?id=2043753, which makes it incompatible with rsync daemons that don't support that option?
--sparse-block=
/cc'ing @mruprich, if he could chime in here.
Hello Frantisku, can you try using it together with -S or --sparse? It could be that when I added the option, it was not added properly when running a daemon, I will look into that. Just a note - setting the --sparse-block to 1024 has no effect since that is the default value anyway.
Cheers, Michal
Hello Frantisku, can you try using it together with -S or --sparse? It could be that when I added the option, it was not added properly when running a daemon, I will look into that. Just a note - setting the --sparse-block to 1024 has no effect since that is the default value anyway. Cheers, Michal
-S/--sparse doesn't seem to affect the result:
-S/--sparse
# rsync --remote-option=--msgs2stderr --debug=all4 --sparse -a -vvv --password-file=rsync.key test.img systemd@artifacts.ci.centos.org::systemd/ opening tcp connection to artifacts.ci.centos.org port 873 Connected to artifacts.ci.centos.org (172.19.0.7) msg checking charset: UTF-8 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 sending daemon args: --server -vvvlogDtprSe.iLsfxC --sparse-block=1024 --debug=ALL4 --msgs2stderr . systemd/ (7 args) [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 [sender] safe_read(3)=1 (Client) Protocol versions: remote=31, negotiated=31 [sender] safe_read(3)=1 [sender] safe_read(3)=4 FILE_STRUCT_LEN=24, EXTRA_LEN=4 sending incremental file list [sender] make_file(test.img,*,0) [sender] flist start=1, used=1, low=0, high=0 [sender] i=1 <NULL> test.img mode=0100644 len=134,217,728 uid=0 gid=0 flags=1005 send_file_list done [sender] flist_eof=1 file list sent send_files starting rsync: read error: Connection reset by peer (104) [sender] _exit_cleanup(code=10, file=io.c, line=785): entered rsync error: error in socket IO (code 10) at io.c(785) [sender=3.1.3] [sender] _exit_cleanup(code=10, file=io.c, line=785): about to call exit(10)
Yup, I see the bug, the sparse-block is added to the arguments sent to the server every time. Thanks for pointing it out, I am going to try and fix this!
Thank you!
Closing this ticket, since the issue is not related to the CentOS Infra.
Metadata Update from @mrc0mmand: - Issue close_status updated to: Wrong tracker - Issue status updated to: Closed (was: Open)
@mruprich hey .. do you want a BZ about this ? I confirm that whole mirror.stream.centos.org was impacted by this , blocking all CentOS Stream 8 and 9 updates to go out to mirror network Downgrading (as a workaround) is the only solution to unblock updates to go out
Here is the BZ report (for people willing to track this issue) : https://bugzilla.redhat.com/show_bug.cgi?id=2043753
Testing a scratch build right now, I am planning to get this out today.
The build is out now, took a bit longer to get through our testing because the infrastructure is a bit unstable lately. The fix is in rsync-3.1.3-17.el8. I am not sure how long does it take to get to the C8S repo.
Login to comment on this ticket.