#922 Switch 'Fedora Archive' category to scan-primary-mirror
Merged 2 years ago by adrian. Opened 2 years ago by adrian.
fedora-infra/ adrian/ansible 2022-01-13-scan-archive  into  main

@@ -28,6 +28,7 @@ 

  	CATEGORY="Fedora Secondary Arches"

  elif [ "${1}" ==  "archive" ]; then

  	CATEGORY="Fedora Archive"

+ 	SCANNER="/usr/local/bin/scan-primary-mirror"

  elif [ "${1}" ==  "codecs" ]; then

  	CATEGORY="Fedora Codecs"

  	SCANNER="/usr/local/bin/scan-primary-mirror"

@@ -1,7 +1,7 @@ 

  max_propagation_days = 2

  max_stale_days = 3

  excludes=[".*\\.snapshot", ".*/\\.~tmp~"]

- skip_paths_for_version=["pub/alt"]

+ skip_paths_for_version=["pub/alt", "pub/archive"]

  test_paths=["/test/", "/stage/"]

  skip_repository_paths = ["Cloud", "Workstation", "Server", "drpms" ]

  do_not_display_paths = ["_Beta"]
@@ -94,3 +94,38 @@ 

  regex="codecs.fedoraproject.org/openh264/[\\.\\d]+/.*"

  prefix="fedora-cisco-openh264"

  

+ ### Fedora Archive ###

+ 

+ [[category]]

+ name="Fedora Archive"

+ type="directory"

+ url="/srv/pub/archive/"

+ 

+ [[repository_mapping]]

+ regex="^pub/archive/fedora(-secondary)?(/linux)?/(core|releases|development)/(rawhide|[\\.\\d]+)/.*"

+ prefix="fedora"

+ 

+ [[repository_mapping]]

+ regex="^pub/archive/fedora(-secondary)?(/linux)?/updates/[\\.\\d]+/Everything/.*"

+ prefix="updates-released"

+ 

+ [[repository_mapping]]

+ regex="^pub/archive/fedora(-secondary)?(/linux)?/updates/[\\.\\d]+/Modular/.*"

+ prefix="updates-released-modular"

+ 

+ [[repository_mapping]]

+ regex="^pub/archive/fedora(-secondary)?(/linux)?/updates/testing/[\\.\\d]+/Everything/.*"

+ prefix="updates-testing"

+ 

+ [[repository_mapping]]

+ regex="^pub/archive/fedora(-secondary)?(/linux)?/updates/testing/[\\.\\d]+/Modular/.*"

+ prefix="updates-testing-modular"

+ 

+ # pre modular structure

+ [[repository_mapping]]

+ regex="^pub/archive/fedora(-secondary)?(/linux)?/updates/testing/([\\.\\d]+)/[^ME].*"

+ prefix="updates-testing"

+ 

+ [[repository_mapping]]

+ regex="^pub/archive/fedora(-secondary)?(/linux)?/updates/([\\.\\d]+)/[^ME].*"

+ prefix="updates-released"

Scanning 'Fedora Archive' with the Python based scanner produces
following error for some time now:

/srv/pub/archive/fullfiletimelist-archive has changed since last run. Running umdl for Fedora Archive at Thu Jan 13 12:15:01 UTC 2022
Traceback (most recent call last):
File "/usr/bin/mm2_update-master-directory-list", line 1122, in <module>
sys.exit(main())
File "/usr/bin/mm2_update-master-directory-list", line 1105, in main
session, config, i['path'], category)
File "/usr/bin/mm2_update-master-directory-list", line 970, in sync_directories_from_fullfiletimelist
sync_category_directory(session, config, category, dirname, ctimes[dirname], file_dict, is_repo)
File "/usr/bin/mm2_update-master-directory-list", line 617, in sync_category_directory
session, config, relativeDName, D, category, 'repomd.xml')
File "/usr/lib/python2.7/site-packages/mirrormanager2/lib/umdl.py", line 294, in make_repo_file_details
yumrepo = rpmmd.repoMDObject.RepoMD('repoid', absolutepath)
File "/usr/lib/python2.7/site-packages/rpmmd/repoMDObject.py", line 145, in init
self.parse(srcfile)
File "/usr/lib/python2.7/site-packages/rpmmd/repoMDObject.py", line 165, in parse
for event, elem in parser:
File "<string>", line 100, in next
File "/usr/lib/python2.7/site-packages/rpmmd/misc.py", line 150, in read
return self.checksums.read(self._fo, size)
File "/usr/lib/python2.7/site-packages/rpmmd/misc.py", line 104, in read
self.update(data)
File "/usr/lib/python2.7/site-packages/rpmmd/misc.py", line 100, in update
sumalgo.update(data.encode('utf-8'))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 273: ordinal not in range(128)
umdl for Fedora Archive returned non-zero. Something failed. Please check umdl.log. Thu Jan 13 12:17:45 UTC 2022

The new tool, scan-primary-mirror, does not have this problem and
I wanted to switch to it for some time already.

With this the error is gone and the new tool is used.

rebased onto 10073a6edf5fde595a6fd2dee4efbbf1cf21a5fa

2 years ago

rebased onto df910b3

2 years ago

Pull-Request has been merged by adrian

2 years ago