Odd. I don't see that sha in the repomd.xml there.

I see:

{{{

8b20a393f4caa93598f1148bc171d320b45c1814
1247256927
4624653
18318336
4397a8772eb39c4fecfd5c6443d018ad96a0dbcb
10

}}}

and that file is there and looks fine.

Also, I will note that when something moves to archive, we don't touch it anymore, so unless it's been broken since we archived it it's unlikely to be a change in the repodata on our end.

Whats in your /etc/yum.repos.d/ for updates-newkey?

Can you attach a 'URLGRABBER_DEBUG=1 yum -d9 update' from the failing machine?

Every single mirror has this same problem--If you happen to have a Fedora 9 system and do "yum install foo" you will see every mirror tried and failed due to the same missing file error.

{{{
[updates-newkey]
name=Fedora $releasever - $basearch - Updates Newkey
failovermethod=priority

baseurl=http://download.fedoraproject.org/pub/fedora/linux/updates/$releasever/$basearch.newkey/

mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=updates-released-f$releasever.newkey&arch=$basearch
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-8-and-9-$basearch
}}}

I think I know what happened. My rsync doesn't see that change in repomd.xml, but if I sync it to a different place and compare, they are different. The server's repomd.xml and my local repomd.xml have the same size and same timestamp, so rsync doesn't see the change, and the -c checksum option is refused by the server so I can't automatically sync it. There would need to be a timestamp or size change on the server for mirrors to see the changed file and sync it.

{{{
...updates/9/i386.newkey/repodata]#stat repomd.xml
File: `repomd.xml'
Size: 3449 Blocks: 8 IO Block: 4096 regular file
Device: fd08h/64776d Inode: 4718833 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2012-05-06 17:01:58.086003269 -0400
Modify: 2009-07-10 22:50:55.000000000 -0400
Change: 2012-05-05 17:18:38.374878035 -0400

stat /tmp/repodata/repomd.xml

File: `/tmp/repodata/repomd.xml'
Size: 3449 Blocks: 8 IO Block: 4096 regular file
Device: fd04h/64772d Inode: 507914 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 263/ UNKNOWN) Gid: ( 263/ UNKNOWN)
Access: 2012-05-07 08:47:12.000000000 -0400
Modify: 2009-07-10 22:50:55.000000000 -0400
Change: 2012-05-07 08:47:03.000000000 -0400
}}}

It is clear that the server's version is correct/newer from the data inside the repomd.xml:

{{{
server repomd.xml timestamps:

revision 1247252992 Fri, 10 Jul 2009 19:09:52 GMT
primary_db 8b20 1247256927 Fri, 10 Jul 2009 20:15:27 GMT

mirrors repomd.xml timestamps:

revision 1246641411 Fri, 03 Jul 2009 17:16:51 GMT
primary_db f6f2 1246643667 Fri, 03 Jul 2009 17:54:27 GMT
}}}

So this problem might have manifested around Fri, 10 Jul 2009 20:15:27 GMT and gone unreported until now. That was the date of EOL:

https://www.redhat.com/archives/fedora-announce-list/2009-July/msg00004.html

I recommend fixing this problem like this on the server, to increment the modified time by 1 second:

touch -m -t 200907102250.56 updates/9/i386.newkey/repodata/repomd.xml

That should allow rsync to notice the change and the mirrors to be corrected. An announcement might need to be made in case mirrors aren't syncing the content anymore, thinking it won't change.

The same problem exists in the x86_64 repo.

{{{

stat backup/updates/9/x86_64.newkey/repodata/repomd.xml updates/9/x86_64.newkey/repodata/repomd.xml
File: backup/updates/9/x86_64.newkey/repodata/repomd.xml' Size: 3451 Blocks: 8 IO Block: 4096 regular file Device: fd02h/64770d Inode: 532504 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2012-05-07 06:42:22.000000000 -0400 Modify: 2009-07-10 22:51:20.000000000 -0400 Change: 2012-05-07 09:55:33.000000000 -0400 File:updates/9/x86_64.newkey/repodata/repomd.xml'
Size: 3451 Blocks: 8 IO Block: 4096 regular file
Device: fd02h/64770d Inode: 540698 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 263/ UNKNOWN) Gid: ( 263/ UNKNOWN)
Access: 2012-05-07 09:56:32.000000000 -0400
Modify: 2009-07-10 22:51:20.000000000 -0400
Change: 2012-05-07 09:56:09.000000000 -0400

sha1sum backup/updates/9/x86_64.newkey/repodata/repomd.xml updates/9/x86_64.newkey/repodata/repomd.xml
a76f32854b4d2d117e089dd0769762058079838c backup/updates/9/x86_64.newkey/repodata/repomd.xml
8abc1befea61581832126a35e3b64959d4e1b959 updates/9/x86_64.newkey/repodata/repomd.xml
}}}

How could this have happened? Did someone manually twiddle the timestamps for some reason?

I'm running repoclosures on the i386.newkey and x86_64.newkey repos after updating to the correct repomd.xml to verify that everything is good.

Same problem with updates/testing/9/x86_64.newkey/repodata/repomd.xml:

{{{

stat backup/updates/testing/9/x86_64.newkey/repodata/repomd.xml updates/testing/9/x86_64.newkey/repodata/repomd.xml
File: backup/updates/testing/9/x86_64.newkey/repodata/repomd.xml' Size: 3434 Blocks: 8 IO Block: 4096 regular file Device: fd02h/64770d Inode: 540736 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2012-05-07 10:10:44.000000000 -0400 Modify: 2009-07-10 22:42:35.000000000 -0400 Change: 2012-05-07 10:09:39.000000000 -0400 File:updates/testing/9/x86_64.newkey/repodata/repomd.xml'
Size: 3434 Blocks: 8 IO Block: 4096 regular file
Device: fd02h/64770d Inode: 540713 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2012-05-07 10:15:47.000000000 -0400
Modify: 2009-07-10 22:42:35.000000000 -0400
Change: 2012-05-07 10:14:51.000000000 -0400

sha1sum backup/updates/testing/9/x86_64.newkey/repodata/repomd.xml updates/testing/9/x86_64.newkey/repodata/repomd.xml
de23472686c7d4e6e2a6450a67fa416823e36b7c backup/updates/testing/9/x86_64.newkey/repodata/repomd.xml
9740d812a95c756c57680ab7005f02d405909bd8 updates/testing/9/x86_64.newkey/repodata/repomd.xml
}}}

After fixing that too, here are the repoclosures:

x86_64 repoclosure:

{{{
Reading in repository metadata - please wait....
Checking Dependencies
Repos looked at: 7
updates-testing
fedora
updates-debuginfo
updates-newkey
updates-testing-newkey
updates-testing-debuginfo
updates
Num Packages in Repos: 22503
package: autotrust-0.2.1-0.2.rc1.fc9.x86_64 from updates-newkey
unresolved deps:
libunbound.so.0()(64bit)
package: openvas-libraries-1.0.2-2.fc9.i386 from updates-newkey
unresolved deps:
libresolv.so.2(GLIBC_PRIVATE)
package: openvas-libraries-1.0.2-2.fc9.x86_64 from updates-newkey
unresolved deps:
libresolv.so.2(GLIBC_PRIVATE)(64bit)
package: reinteract-0.5.0-2.noarch from updates-testing-newkey
unresolved deps:
python-matplotlib >= 0:0.98.0
}}}

i386 repoclosure:

{{{
Reading in repository metadata - please wait....
Checking Dependencies
Repos looked at: 5
updates-newkey
fedora
updates
updates-testing-newkey
updates-testing
Num Packages in Repos: 17844
package: autotrust-0.2.1-0.2.rc1.fc9.i386 from updates-newkey
unresolved deps:
libunbound.so.0
package: openvas-libraries-1.0.2-2.fc9.i386 from updates-newkey
unresolved deps:
libresolv.so.2(GLIBC_PRIVATE)
package: reinteract-0.5.0-2.noarch from updates-testing-newkey
unresolved deps:
python-matplotlib >= 0:0.98.0
}}}

I am '''very reluctant''' to change archived releases.

The repoml.xml with:

{{{
revision 1246641411 Fri, 03 Jul 2009 17:16:51 GMT
}}}

Where did you get that?
you had it from a long ago sync? or was it present anywhere on mirrors?

there is no way that we will be touching any file in a EOL release. we dont touch the releases tree once its done. not sure how you got messed up. but using F9 is no longer supported and has not been for a long long time. this is the first report like this and its a long time since the content was last changed. sorry but your on your own here.