Learn more about these different git repos.
Other Git URLs
Later this year our storage provider is moving to newer appliances. These appliances will only have 2 grades of storage: sata disks or flash. The idea is that anything that doesn't require access very often or speed can be on the sata disks, and anything that is accessed/used a lot can be on the flash storage.
Currently our fedora_koji volume is 45TB and growing.
Currently our fedora_ftp master mirror volume is 12TB.
In order to take advantage of the new flash storage we need to seperate out our data by usage.
For fedora_ftp we should easily be able to split out the archive part. That ends up being about 6.5TB. Even if that data is accessed a fair bit having archive and non archive would help us better be able to fit into flash with it.
For fedora_koji we can take advantage of koji's "volumes" support. Basically we make a new volume and mount it on /mnt/koji/koji_archive and then tell koji about that volume and then we script something that moves builds over to that volume. We could start with the oldest fc6 builds and move up from there. I am not sure how much space all the EOL releases will get us, we will just have to wait and see, but it should help reduce the massive size and make it more possible to get the active set into flash.
So, actions here:
We should be able to start on this after freeze...
splitting off archive will break the model we use to archive releases. we need to coordinate with the mirrors.
all for the koji split
Yeah, moving archive to it's own volume will mean we cannot hard link it to the content for a while and then remove it after mirrors have it. However... we seem to have very few archive mirrors anyhow. Checking a metalink right now I see 4 archive mirrors.
Perhaps @adrian can check the db and see how many have it enabled?
Using
$ curl -s "https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-20&arch=x86_64&country=global" # repo = fedora-20 arch = x86_64 country = global http://mirror.math.princeton.edu/pub/fedora-archive/fedora/linux/releases/20/Everything/x86_64/os/ http://mirrors.rit.edu/fedora/archive/fedora/linux/releases/20/Everything/x86_64/os/ https://pubmirror1.math.uh.edu/fedora-buffet/archive/fedora/linux/releases/20/Everything/x86_64/os/ https://fedora-archive.ip-connect.vn.ua/fedora/linux/releases/20/Everything/x86_64/os/ https://ftp-stud.hs-esslingen.de/pub/Mirrors/archive.fedoraproject.org/fedora/linux/releases/20/Everything/x86_64/os/ https://pubmirror2.math.uh.edu/fedora-buffet/archive/fedora/linux/releases/20/Everything/x86_64/os/ http://ftp.cuhk.edu.hk/pub/linux/fedora-archive/fedora/linux/releases/20/Everything/x86_64/os/ http://fedora-mirror01.rbc.ru/pub/fedora-archive/fedora/linux/releases/20/Everything/x86_64/os/ https://dl.fedoraproject.org/pub/archive/fedora/linux/releases/20/Everything/x86_64/os/
I see nine mirrors. In the database we have 46 hosts with the "Fedora Archive" category.
mirrormanager2=> select name, private from host, host_category where host.id = host_category.host_id and host_category.category_id = 12 order by private; name | private --------------------------------------------+--------- mirror.hmc.edu | f mirror.nbtelecom.com.br | f mirrors.netix.net | f fedora-archive.ip-connect.vn.ua | f dl.fedoraproject.org | f ftp-2.tsukuba.wide.ad.jp | f http://mirror.rackcentral.com.au/ | f mirror.beyondhosting.net | f kdeforge.unl.edu | f fedora.mirror.liquidtelecom.com | f www.nttsapporo.co.jp | f mirrors.netix.net | f pubmirror2-archive | f mirror1.hs-esslingen.de | f mirrorarchive.math.princeton.edu | f ftp.cuhk.edu.hk | f ftp.pbone.net | f ftp.riken.jp | f fedora-mirror01.rbc.ru | f mirrors.rit.edu | f pubmirror1-archive | f ftp.kddilabs.jp | f mirrors.sgu.ru | t download.wpi.edu | t titan2.idc.corp | t s3-mirror-us-east-1.fedoraproject.org | t s3-mirror-eu-west-1.fedoraproject.org | t s3-mirror-us-west-1.fedoraproject.org | t dev-fit.cpi.local | t cpe-97-99-207-171.tx.res.rr.com | t s3-mirror-ap-northeast-1.fedoraproject.org | t lgmt.teknofile.org | t mirror.magmasoft.com.ec | t p50937f9e.dip0.t-ipconnect.de | t 71.178.26.29 | t 172.30.4.2 | t mesozoictech.com/mirror | t lenovo-t420s.mende.spb.ru | t mail2.furryball.org | t Fedora Uptodate | t ansplus4.aplv.lan | t zinc.sa.bluecoat.com | t fedora-mirror.fc.hp.com | t mirror.entersect.co.za | t mirror.m4com.de | t s3-mirror-us-west-2.fedoraproject.org | t (46 rows)
So from 22 non-private mirrors nine are working.
@kevin is this still a thing we are looking to do?
@mohanboddu confirms that @kevin is working on this.
Metadata Update from @syeghiay: - Issue assigned to kevin
@kevin, any update?
I have had this on hold for a while because we have been discussing longer term storage plans with storage folks. That doesn't seem like it's going to happen anytime soon, so I will check with them and then start moving foward on this.
@kevin , any progress on this since your last update?
Nope. We are in freeze so I am not going to disrupt koji.
However, it looks like this should be something we move forward on, so I will try and get started on it after Beta is out the door.
Since we are now in F29 final freeze, we expect @kevin to continue working this ticket into F30.
Yep. We are gettiing some new storage hardware in to try and meet this need, but it will be after f29 release.
Metadata Update from @syeghiay: - Issue tagged with: meeting
@kevin , any update?
Yeah, we were all set to start on this, but then we needed to move the storage we had allocated to another more urgent need. ;(
We are however getting more storage hopefully in this quarter and can start on it then.
Sorry for the long delay here. ;(
This work has started happening. I am tracking it in https://pagure.io/fedora-infrastructure/issue/8065
We can keep this open to track it further here, or just close in favor of the other one, either is fine with me.
I'd say close so there is one less place for you to come back to.. People who want to follow can do so over there.
Sure, makes sense.
Metadata Update from @kevin: - Issue close_status updated to: Duplicate - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.