Currently I'm using inotifywait on alt.fedoraproject.org to get instant notifications of filesystem changes - specifically, files moved to or created in /srv/pub/alt/stage/. This way I can start creating the delta ISOs immediately after new install ISOs are posted. The inotify-tools package is not installed, but I work around that by extracting binaries from CentOS 6 RPMs. Except for the workaround, my script looks like this:
ssh robatino@secondary01.fedoraproject.org inotifywait -e create -e moved_to /srv/pub/alt/stage xset dpms force on while true ; do echo -e "CHANGED\a" sleep 1 done
This kind of notification could save people time during TC/RC testing. Unfortunately, few people have an account on alt.fp.o, but many have a fedorapeople.org account. I could set something up that would immediately make filesystem changes in my fedorapeople account corresponding to those on alt.fp.o (since I have accounts on both) and then anyone with a fedorapeople account could monitor that. This would allow people to do things like trigger a script that would immediately start downloading when new files become available.
I did some googling to see if something like inotify-tools that would work on a remote server without having to have an account on it already existed, but couldn't find anything. However, it's pretty easy to get a fedorapeople account, and most contributors probably have one.
I don't have any great objection... but several things:
alt (aka secondary01) is under freeze right now until Beta is out.
Would this result in people pulling images before they are fully uploaded, resulting in some confusion?
I don't know that inotify load on people02 would be great... but you could update a web reachable file and they could poll that? Or for that matter, we could setup something like the kernel.org finger server that lists the current releases/pre-releases.
Perhaps we should have some more discussion of how to solve the high level problem before heading to inotify?
I personally have a problem with you going and installing random software to do something on one of our servers. there is always the possibility of something going wrong with that. there is a reason why we have logins auto logout after idle time. the security implications of this are not great. i think there are better ways that we could solve the problem. though im not sure it is entirely a problem. please bring up the issue on the infrastructure list for discussion
Replying to [comment:1 kevin]:
I don't have any great objection... but several things: alt (aka secondary01) is under freeze right now until Beta is out.
No rush.
It depends on how the images are uploaded. I create each delta ISO directory in a hidden subdirectory and then move it to a public one when finished, so it's finished at the instant it's visible. The TC/RC ISOs are usually rsync'd into place, so during uploading there's a temporary dotfile for each file, until it's fully uploaded (which takes at most a few minutes). Hopefully people would learn from experience what to expect, and they should be verifying the checksums in any case to make sure the files are good.
With polling you have to make a tradeoff between frequent polling bogging down the server vs. infrequently and having a notification delay. With inotify, you'd have the slight memory overhead of an inotify process running constantly, but it wouldn't use any CPU. The point would be to reduce the load on the server associated with getting prompt notifications. Of course, people can use polling right now (with something like netstiff or urlwatch) or grab the inotify binaries from CentOs like I did to work around it not being installed.
How about we open a discussion of this on the list?
I think this might be a nice way for us to finally look at deploying a message bus of some kind. It would be a very simple data to export out and an easy goal instead of getting mired in all the possible other uses. ;)
(Changing summary a bit).
I've posted to the infrastructure list to ask about ideas on this. I think we could do something better than inotify... but perhaps not.
I think this could be solved by the fedmsg message bus.
rel-eng could indicate when they are done syncing up with a message.
I'll talk to ralph about this. Would probibly need to wait until after the current freeze.
My plan to slove this is with a composedb, we could tie notifications into fedmsg.
I have a patch to the rel-eng repo that may make some progress. I can't test it though; lmacken has volunteered to deploy/test after the freeze.
http://lists.fedoraproject.org/pipermail/infrastructure/2012-September/012148.html
composes are signaled via fedmsg, therefore I consider this to be fixed. Please re-open the ticket if this is not the case.
Login to comment on this ticket.