In many of the last several releases (11, 13, 14, and now 16), at least some of the Alpha or Beta torrents contain only unsigned checksum files. This would be easy to prevent by examining the .torrent files, which contain file sizes (signing a checksum file adds about 1K to the size). Unfortunately, at present these are not made available for testing prior to being posted on http://torrent.fedoraproject.org , and when the problem is pointed out, no matter how quickly, one is told that the torrent can't be replaced since people are already downloading it. This makes it important to catch the problem in advance.
Many (but not all) of the torrent files for the last several releases are still available at http://torrent.fedoraproject.org/torrents/ and http://torrent.fedoraproject.org/spins/ , and can be examined for example with gtorrentviewer. I have not checked any older than 11, and not all the ones after that are available, so the above list of affected releases is probably incomplete.
A less serious issue is when the checksum files get signed more than once. For example, the checksum files for F15 Final install discs were signed twice, first for the torrents and again for the mirrors - see http://robatino.fedorapeople.org/checksums/15-Final/Fedora/ . The checksums are identical, and both signatures are valid, but still, it shouldn't happen.
Looking at https://fedoraproject.org/wiki/Release_Engineering_Release_Tickets , it says that for Alpha and Beta, the torrents should be staged before the mirrors, but the reverse for Final. I've asked why on #fedora-releng but gotten no response yet. It says nothing about signing the checksum files, though the linked page https://fedoraproject.org/wiki/Stage_final_release_for_mirrors (under the section "Final") mentions it. This may explain why Alpha and Beta torrents are much less likely to have signed files. If possible, it would be nice for the order (torrents vs. mirrors) to be the same for all three, and in any case, the checksum files should be signed once and then used for both torrents and mirrors. None of this is currently documented.
sounds reasonable - to go with the other 'image sanity' tests, right?
would it also be good to file tickets with rel-eng to improve their process here?
thanks!
IIUIC this could go into AutoQA trac, provided that the torrents are accessible for some time on a well-defined location prior to publishing. Then we could write a watcher and a test for it. If they are not accessible prior to publishing, all we can do is to ask rel-eng to improve their processes.
I filed https://fedorahosted.org/rel-eng/ticket/4906 (some form of QA access to torrent and mirror content prior to public posting).
I found the page describing how torrents are created:
https://fedoraproject.org/wiki/Infrastructure/SOP/TorrentRelease
If someone in QA had an account on this machine, it would be possible to verify the content (.torrent file plus uploaded files, including the signed checksum file) prior to running step 9. I wasn't aware that the content had to exist on the server together with the .torrent file, so there doesn't seem to be any easy way to exclude access to the signed checksum file. (Although since the mirrors which get access to it days in advance usually leak anyway, maybe it should just be quietly made public prior to the official release date.)
According to https://fedoraproject.org/wiki/Go_No_Go_Meeting , the decision to declare a release GOLD is irrevocable. Would it make sense to just have rel-eng sign the checksum files after this happens, send them to QA, and have QA verify them and also post the torrents (after running tests to verify those as well)?
What's the status here, Andre? Are you stuck waiting on releng? Anything I can do to help move this along? Is https://fedorahosted.org/rel-eng/ticket/4986 relevant to this issue too?
Replying to [comment:6 adamwill]:
What's the status here, Andre?
No change as far as I know. [https://fedoraproject.org/wiki/Release_Engineering_Release_Tickets] is the same. We didn't have any problems with 16 Final, but that was just because of the recent attention due to these tickets, not any process change.
Are you stuck waiting on releng? Anything I can do to help move this along? Is https://fedorahosted.org/rel-eng/ticket/4986 relevant to this issue too?
That ticket would greatly help making these kinds of mistakes less likely, but I still think QA should be able to test everything before release, including checksum and torrent files, not just ISOs, as it's doing now.
There's another issue which I brought up in [https://bugzilla.redhat.com/show_bug.cgi?id=727387] which is that I think it would be a good idea for the checksum files to include file sizes in bytes, as comments. The original motivation is that most of our ISOs are hybrid now, so if you burn it to media you can't read it off and verify the checksum without knowing independently what the file size is (since you can't trust the ISO header anymore). Also, truncation is probably the most common form of corruption, and people downloading an ISO via browser or a buggy bittorrent client don't know what the exact size is supposed to be. If they did, they could just finish the last few KB instead of downloading the full ISO a half dozen times. I'm not sure if that issue belongs in one of these tickets. If the checksum files were to include this, they would have to be changed separately for install and live images, since I believe install checksums are made automatically by pungi, and live checksums manually.
.torrent files are available before GA the always have been. they can be used to check that the CHECKSUM is signed. Do note that we do make the torrents as one of the last things we do in a release.
Replying to [comment:8 ausil]:
This is correct as far as it goes, the problem is that every one of the times I pointed out that the .torrent file was unsigned, I was told that it was impossible to change it, since people were already downloading. So in other words, yes, it can be checked, but since it first appears in a public location, if there's a mistake, there's nothing that can be done about it, which makes the check useless.
Now personally, I don't understand why the .torrent can't be changed after people start downloading. The worst thing I can see happening is that peers would be split between two different torrents, and seeders could temporarily seed both, using a hard link for the ISO to avoid wasting space, until the people on the obsolete torrent were finished. (Or just post a message telling people to load the new .torrent file in their client.) If the .torrent file can in fact be changed, then there is no great need to make sure the .torrent file is perfect the first time it's posted publicly. But everyone so far has been unanimous in telling me no, we can't do that, even when i pointed out the problem within hours.
If its wrong it needs to be pointed out before users start downloading it. there is a small window where it can be verrified. usually its about 8 hours. but once its announced to the public we can not change it. we spot issues when its just the seeders and those who watch and jump on the torrents before announced im ok with fixing it. you have always come after we have announced and thats too late
Replying to [comment:10 ausil]:
The window of opportunity before announcement needs to be formalized, then. For the last few releases I've been frequently checking for the existence of the torrents before and after release, anticipating problems (especially for Alpha and Beta) and have still never been able to report them soon enough for anyone to be willing to fix them. There's no chance that someone just randomly noticing could do it soon enough. I'd be happy to check several hours before release assuming the torrents are guaranteed to exist at that time. The 16 FEL Spin wasn't available until after release, though (Dec. 1) so it still wouldn't be possible to fix in such cases. There's also the problem that if a mistake is noticed before release, the new .torrent might not be posted until after release, or with not enough time to test it, making it impossible to fix possible mistakes in that.
Personally I still think it's a bad idea for the .torrent files to be publicly available in the usual location before they've been verified, for the above reasons, but a formal window would help.
It should also be formalized that before release, it's permissible to change the .torrent files. Most people currently seem to think it's never okay to do this, so there should be something to point them to.
I'd be happy to check several hours before release assuming the torrents are guaranteed to exist at that time.
Just checked, and the 16 Alpha torrents were both unsigned and only available 36 minutes after official release (10:36 AM EDT on August 23) so there was no window for that.
In any case it was incorrect for Dennis to close this, as this is a QA group task to create a test case / SOP to verify the torrent files, not any kind of request to releng. The releng request ticket is/was https://fedorahosted.org/rel-eng/ticket/4906 ; if you want to close anything, Dennis, close that. Unfortunately, I don't seem to be able to re-open this ticket.
Please also see
https://lists.fedoraproject.org/pipermail/test/2011-December/104897.html
I think the main problem here is that torrents and checksum files are being written to their final locations without being put in a temporary location, tested and verified first as happens with the ISOs. This means it's a race to fix problems before a broken version is released, and also that they tend to get posted shortly before release, when there's little time to fix things. By going to a temporary location first, they can be created shortly after Go/No Go, and any problems fixed well before release.
Is this still an active concern, Andre? Sorry to keep pinging :(
AFAIK nothing has changed, except that recently the checksum files happen to have been signed regularly. But to be honest I haven't checked them in the last 1 or 2 releases.
Metadata Update from @adamwill: - Issue priority set to: wishlist (was: normal) - Issue set to the milestone: Undetermined Future
clearly nobody is doing anything about this.
Metadata Update from @adamwill: - Issue close_status updated to: Wontfix - Issue set to the milestone: None (was: Undetermined Future) - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.