Learn more about these different git repos.
Other Git URLs
The way external repos are currently handled is insufficient for modularity because:
See also: https://docs.google.com/document/d/1NxBbTPVJ17Ax4k-NbtorzxYspJ_CsjQ9m3In2wqWqVU/edit?usp=sharing
Draft change (doesn't address everything): https://github.com/mikem23/koji-playground/commits/dumb-merge-mode
Further discussion this morning https://docs.google.com/document/d/1cxr5TWw2WEyeEtAd-5lXW4nYG4zLhU9Aow0bCGJTKk8/edit
Short version: add a "bare" merge mode to Koji that just calls mergerepos_c with --pkgorigins (a new option to be added). The merge behavior will be normal mergerepo, which now handles module metadata, but also generate pkgorigins. No filtration, no block list.
Metadata Update from @mikem: - Issue priority set to: High (was: Normal) - Issue set to the milestone: 1.18
@mikem I can't read the document...
Commit e7a38d7 fixes this issue
Commit be6ae53 fixes this issue
@ngompa can you read it now?
@ktdreyer No, I've requested permission again, though.
Sorry, @ngompa, it appears that docs for rh accounts have gotten much stricter access controls now. Seems to be a recent change. :disappointed:
The doc on the description should be readable, but I can't do anything with the doc in the following comment. Luckily, there isn't really anything there that isn't covered between this issue, the pr, and the createrepo_c pr.
I think there's some things missing here... the kojid side and cli seems there, but the hub side doesn't?
Trying to set this on an existing external repo gives:
2019-06-17 11:40:08,785 [ERROR] koji: ParameterError: edit_external_repo() got an unexpected keyword argument 'merge_mode'
I don't seem to see a way to tell what the merge mode is on an external repo? The hub web page on the repo doesn't say...
Adding a new external repo with it set works, but then when you try and regen that repo you get:
Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: 2019-06-17 19:07:04,903 [WARNING] {3371} koji.TaskManager:1314 TRACEBACK: Traceback (most recent call last): Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: File "/usr/lib/python3.7/site-packages/koji/__init__.py", line 2527, in _callMethod Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: return self._sendCall(handler, headers, request) Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: File "/usr/lib/python3.7/site-packages/k oji/__init__.py", line 2449, in _sendCall Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: return self._sendOneCall(handler, head ers, request) Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: File "/usr/lib/python3.7/site-packages/k oji/__init__.py", line 2491, in _sendOneCall Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: ret = self._read_xmlrpc_response(r) Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: File "/usr/lib/python3.7/site-packages/k oji/__init__.py", line 2503, in _read_xmlrpc_response Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: result = u.close() Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: File "/usr/lib64/python3.7/xmlrpc/client .py", line 656, in close Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: raise Fault(**self._stack[0]) Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: xmlrpc.client.Fault: <Fault 1000: 'failed to merge repos: /usr/bin/mergerepo_c --pkgorigins -a aarch64 -b /mnt/koji/repos/epel8-build/9000357/aarch64/b locklist -o /var/tmp/koji/tasks/3058/90003058/repo -g /mnt/koji/repos/epel8-build/9000357/groups/comps.xml -r file:///var/tmp/koji/tasks/3058/90003058/repo_9000357_premerge/ -r https://infrastructure.fedoraproject.org/ repo/rhel/rhel8/koji/latest//RHEL-8-001/ exited with status 1'> Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: During handling of the above exception, an other exception occurred: Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: Traceback (most recent call last): Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: File "/usr/lib/python3.7/site-packages/k oji/daemon.py", line 1295, in runTask Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: response = (handler.run(),) Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: File "/usr/lib/python3.7/site-packages/k oji/tasks.py", line 311, in run Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: return koji.util.call_with_argcheck(se lf.handler, self.params, self.opts) Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: File "/usr/lib/python3.7/site-packages/k oji/util.py", line 263, in call_with_argcheck Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: return func(*args, **kwargs) Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: File "/usr/sbin/kojid", line 5051, in ha ndler Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: results = self.wait(to_list(subtasks.v alues()), all=True, failany=True) Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: File "/usr/lib/python3.7/site-packages/k oji/tasks.py", line 395, in wait Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: self.session.getTaskResult(task) Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: File "/usr/lib/python3.7/site-packages/k oji/__init__.py", line 2055, in __call__ Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: return self.__func(self.__name, args, opts) Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: File "/usr/lib/python3.7/site-packages/k oji/__init__.py", line 2546, in _callMethod Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: raise err Jun 17 19:07:04 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: koji.GenericError: failed to merge repos: /usr/bin/mergerepo_c --pkgorigins -a aarch64 -b /mnt/koji/repos/epel8-build/9000357/aarch64/blocklist -o /var /tmp/koji/tasks/3058/90003058/repo -g /mnt/koji/repos/epel8-build/9000357/groups/comps.xml -r file:///var/tmp /koji/tasks/3058/90003058/repo_9000357_premerge/ -r https://infrastructure.fedoraproject.org/repo/rhel/rhel8/ koji/latest//RHEL-8-001/ exited with status 1 Jun 17 19:07:19 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: 2019-06-17 19:07:19,928 [INFO] {2398} koji .TaskManager:779 pids: {90003057: 3371} Jun 17 19:07:20 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: 2019-06-17 19:07:20,008 [INFO] {2398} koji .TaskManager:980 Task 90003057 (pid 3371) exited with status 0 Jun 17 19:07:20 buildvm-03.stg.phx2.fedoraproject.org kojid[2398]: 2019-06-17 19:07:20,021 [INFO] {2398} koji .TaskManager:1125 Expiring subsession 53836145 (task 90003057)
So it's still passing -b to the 'bare' mode? (blocklist)?
Metadata Update from @kevin: - Custom field Size adjusted to None - Issue status updated to: Open (was: Closed)
koji remove-external-repo <repo> <tag> && koji add-external-repo <repo> --tag <tag> --mode bare
latest//RHEL-8-001/
$arch
option should have never been there: https://pagure.io/koji/pull-request/1499 Only way how to change it now is to run koji remove-external-repo <repo> <tag> && koji add-external-repo <repo> --tag <tag> --mode bare option is really hidden - as we're going to see it more now, it is reasonable to list it everywhere - filed #1500 call in example seems to be correct (passing --pkgorigins which is 'bare' mode). Problem there is missing arch in latest//RHEL-8-001/ part of repo. Unescaped $arch in add-external-repo call maybe?
When I tried to fix $arch, I hit the following issue:
$ stg-koji add-external-repo rhel8-base 'https://infrastructure.fedoraproject.org/repo/rhel/rhel8/koji/latest/$arch/RHEL-8-001/' --tag epel8-base --mode bare Usage: koji add-external-repo [options] name [url] (Specify the --help global option for a list of other help options) koji: error: Invalid mode: bare
As far as I can tell, #1499 is not required for anything to work. It simply removes a nonworking option that should not have been there.
The affected command (edit-external-repo) is not the correct way to adjust the merge mode for a tag_external_repo entry. The only way is to remove it and re-add it with the desired merge_mode as @tkopecek suggests above:
edit-external-repo
Only way how to change it now is to run koji remove-external-repo <repo> <tag> && koji add-external-repo <repo> --tag <tag> --mode bare
koji: error: Invalid mode: bare
EDIT: yes the client does validate the mode The client does not validate the mode value, so this error is likely coming from the hub.This is odd, because the stage hub seems to know about this merge mode. It appears that the setup @mohanboddu created earlier had the correct merge mode (but not the correct url, as tomas surmised).
[mike@localhost koji]$ koji -p stg list-history --after 2019-06-17 -v --external-repo=rhel8-base -v Mon Jun 17 15:03:41 2019 (eid 41024228) external repo entry for rhel8-base removed from tag epel8-base by mohanboddu Mon Jun 17 15:05:10 2019 (eid 41024229) external repo deleted: rhel8-base by mohanboddu Mon Jun 17 15:05:17 2019 (eid 41024230) new external repo: rhel8-base by mohanboddu url: https://infrastructure.fedoraproject.org/repo/rhel/rhel8/koji/latest//RHEL-8-001/ Mon Jun 17 15:05:17 2019 (eid 41024231) external repo entry for rhel8-base added to tag epel8-base by mohanboddu merge_mode: bare priority: 5 Tue Jun 18 10:16:00 2019 (eid 41024233) external repo deleted: rhel8-base by mohanboddu Tue Jun 18 10:16:00 2019 (eid 41024233) external repo entry for rhel8-base removed from tag epel8-base by mohanboddu
Merge modes work. We've been using them in Brew for a while. This feature was just to add a new one. The changes in #1411 are very small.
The client does not validate the mode value, so this error is likely coming from the hub. This is odd, because the stage hub seems to know about this merge mode. It appears that the setup @mohanboddu created earlier had the correct merge mode (but not the correct url, as tomas surmised).
@mikem I tried the right URL, but still it didn't work. I tried it with both quotes and escape character:
$ stg-koji add-external-repo rhel8-base https://infrastructure.fedoraproject.org/repo/rhel/rhel8/koji/latest/\$arch/RHEL-8-001/ --tag epel8-base --mode bare Usage: koji add-external-repo [options] name [url] (Specify the --help global option for a list of other help options) koji: error: Invalid mode: bare $ stg-koji add-external-repo rhel8-base 'https://infrastructure.fedoraproject.org/repo/rhel/rhel8/koji/latest/$arch/RHEL-8-001/' --tag epel8-base --mode bare Usage: koji add-external-repo [options] name [url] (Specify the --help global option for a list of other help options) koji: error: Invalid mode: bare
@mohanboddu how does that command differ from the one you used before that had the wrong url?
I was wrong above, the client does check the for valid merge modes and Mohan's second run was on a host that lacked the lib update.
So, the hub was fine the whole time.
call in example seems to be correct (passing --pkgorigins which is 'bare' mode). Problem there is missing arch in latest//RHEL-8-001/ part of repo. Unescaped $arch in add-external-repo call maybe?
Fixed this, but still facing the issue: https://koji.stg.fedoraproject.org/koji/taskinfo?taskID=90003153
PR #1502 should fix this
Commit e73bd36 relates to this ticket
Commit 0899f74 relates to this ticket
Commit 3b0028a relates to this ticket
Commit 79f1c8b relates to this ticket
@mohanboddu Are things looking better now with #1502 ?
@mikem Yeah, it seems to be working now. Even though we faced an issue with how the repo is getting created, @smooge found a solution for it.
For now, I think we are good.
Metadata Update from @tkopecek: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.