Learn more about these different git repos.
Other Git URLs
[This came up during the discussion about reproducible builds during Flock.] To make a build that reproduces an earlier build, we will need to create a build environment that matches an earlier build environment, in particular have the same package set including versions, and the same set of macros. This information is currently available as logs (once https://pagure.io/koji/issue/3875 is resolved ;)), which is OK for debugging, but not very nice for programatic access. We will be setting up a service that does continuous rebuilds of packages to check reproducibility (https://pagure.io/fedora-reproducible-builds/project/issue/1). Ideally, this service would be able to query koji (^) for a file.
I think the most natural format would be json. Maybe something like this:
{ "manifest_version": 1, "build": { "name": "foobar", "distribution": "fedora", "architecture": "x86_64", "release": "38" }, "packages": [ { "type": "rpm", "name": "alternatives", "version": "1.22-1.fc38", "architecture": "x86_64" }, ... ], "macros":[ { "name": "_with_bootstrap", "value": "1" }, ... ] }
(The format is loosely inspired by https://systemd.io/ELF_PACKAGE_METADATA/#well-known-keys.)
Other prior art: https://wiki.debian.org/ReproducibleBuilds/BuildinfoFiles. But that format is very specific to Debian and not very nice in general.
(^) I think koji should make this file available and link it from a build page next to the build log, root log, and the other metadata files that are already provided. Once rpms have been signed and become part of a repo, we might consider pushing this additional metadata to mirrors somehow. But even if that happens, we need to have it in koji first.
It is also available via api. You can also retrigger the same build in exactly the same environment which is one of the design ideas. So, not sure if another service for testing reproducibility is needed if you can just trigger the new (scratch) task for this. For list of packages installed in buildroot you can see e.g. this page (https://koji.fedoraproject.org/koji/rpmlist?buildrootID=44725691&type=component) (getBuildrootListing API call) For macros used you can similarly call getBuildConfig(tag, event=xyz), etc. So, everything is available already. Converting it to manifest doesn't make much sense to me, but it is a possibility to create a plugin which will add it to the build logs. Is the API sufficient for you (and does it make sense to rebuild rpms outside of koji duplicating resources, etc.?)
getBuildrootListing
getBuildConfig(tag, event=xyz)
Metadata Update from @tkopecek: - Custom field Size adjusted to None
Thanks!
You can also retrigger the same build in exactly the same environment which is one of the design ideas. So, not sure if another service for testing reproducibility is needed if you can just trigger the new (scratch) task for this.
In general we want reproducible builds to happen in a completely separate system. The current idea would be to have a machine which listens for notifications about new builds in koji on fedmsg and tries to recreate the builds, in order to be able to detect non-reproducible builds and set up a reporting dashboard similar to https://tests.reproducible-builds.org/debian/reproducible.html. OTOH, repeating builds in koji is useful while this is being set up, just to test if builds are reproducible and to debug things.
For now, the API should be enough.
Converting it to manifest doesn't make much sense to me, but it is a possibility to create a plugin which will add it to the build logs.
I think there'd still be value to have a manifest file, just to make it easier to do this in a generic way, independent of the koji API. For example, once reproducible builds are the norm, people would probably like to rebuilds rpms from our koji, or from CentoOS/RHEL/whatever, OBS and OpenSuse, etc. For that, having one file that can be saved somewhere would be much more convenient. But we're still pretty far from this, so let's close this, at least for now.
Metadata Update from @zbyszek: - Issue close_status updated to: Invalid - Issue status updated to: Closed (was: Open)
For list of packages installed in buildroot you can see e.g. this page (https://koji.fedoraproject.org/koji/rpmlist?buildrootID=44725691&type=component) (getBuildrootListing API call). For macros used you can similarly call getBuildConfig(tag, event=xyz), etc.
Hmm, how do I got from a build id to the build config? .getBuild() returns a bunch of stuff, but this doesn't contain any tags or event ids?
.getBuild()
Hmm, I tried .getBuildrootListing(), but what is the id that it expects:
.getBuildrootListing()
>>> SESSION.getBuild(2273180) {'build_id': 2273180, 'cg_id': None, 'completion_time': '2023-08-13 05:15:53.232475+00:00', 'completion_ts': 1691903753.232475, 'creation_event_id': 115984031, 'creation_time': '2023-08-13 05:08:18.437162+00:00', 'creation_ts': 1691903298.437162, 'epoch': None, 'extra': {'source': {'original_url': 'git+https://src.fedoraproject.org/rpms/python-pikepdf.git#c0930c826d143fda7ea7fbac19aeb888691d6599'}}, 'id': 2273180, 'name': 'python-pikepdf', 'nvr': 'python-pikepdf-8.3.2-1.fc40', 'owner_id': 3972, 'owner_name': 'qulogic', 'package_id': 27416, 'package_name': 'python-pikepdf', 'release': '1.fc40', 'source': 'git+https://src.fedoraproject.org/rpms/python-pikepdf.git#c0930c826d143fda7ea7fbac19aeb888691d6599', 'start_time': '2023-08-13 05:08:18.430490+00:00', 'start_ts': 1691903298.43049, 'state': 1, 'task_id': 104765444, 'version': '8.3.2', 'volume_id': 0, 'volume_name': 'DEFAULT', 'cg_name': None} >>> SESSION.getBuildroot(2273180) {'arch': 'armhfp', 'br_type': 0, 'cg_id': None, 'cg_name': None, 'cg_version': None, 'container_arch': 'armhfp', 'container_type': None, 'create_event_id': 8039888, 'create_event_time': '2014-08-16 23:50:02.307609+00:00', 'create_ts': 1408233002.307609, 'extra': None, 'host_arch': None, 'host_id': 149, 'host_name': 'arm04-builder20.arm.fedoraproject.org', 'host_os': None, 'id': 2273180, 'repo_create_event_id': 8038704, 'repo_create_event_time': '2014-08-16 23:10:48.516665+00:00', 'repo_id': 411463, 'repo_state': 3, 'retire_event_id': 8040341, 'retire_event_time': '2014-08-17 00:01:13.565192+00:00', 'retire_ts': 1408233673.565192, 'state': 3, 'tag_id': 278, 'tag_name': 'f22-build', 'task_id': 7348938}
Link is the task which created the build. So, steps: 1) get build 2) get its task 3) handle each subtask if needed. Every architecture, etc. could be a bit different (dependencies, etc.)
So, rough skeleton:
import koji s = koji.ClientSession("https://koji.fedoraproject.org/kojihub") binfo = s.getBuild("python-fastapi-0.101.1-1.fc40") #build_task = s.getTaskInfo(binfo['task_id']) for subtask in s.getTaskDescendents(binfo['task_id']): tinfo = s.getTaskInfo(subtask) print(f"Build task {tinfo['id']}") if tinfo['method'] in ('buildSRPMFromSCM', 'buildArch'): print(f"Task {tinfo['id']}") for br in s.listBuildroots(taskID=tinfo['id']): print(f"Buildroot {br['id']} contents:") for item in s.getBuildrootListing(br['id']): print('%(name)s-%(version)s-%(release)s' % item)
Thanks, this works.
I still cannot figure out how to call getBuildConfig() though:
>>> SESSION.getBuild('python-referencing-0.30.2-1.fc40') {'build_id': 2272961, 'cg_id': None, 'completion_time': '2023-08-12 08:25:18.428532+00:00', 'completion_ts': 1691828718.428532, 'creation_event_id': 115933981, 'creation_time': '2023-08-12 08:22:40.777705+00:00', 'creation_ts': 1691828560.777705, 'epoch': None, 'extra': {'source': {'original_url': 'git+https://src.fedoraproject.org/rpms/python-referencing.git#6e3caffaaed9717d18a33e048d8f6939c963ae1c'}}, 'id': 2272961, 'name': 'python-referencing', 'nvr': 'python-referencing-0.30.2-1.fc40', 'owner_id': 1943, 'owner_name': 'mattia', 'package_id': 37443, 'package_name': 'python-referencing', 'release': '1.fc40', 'source': 'git+https://src.fedoraproject.org/rpms/python-referencing.git#6e3caffaaed9717d18a33e048d8f6939c963ae1c', 'start_time': '2023-08-12 08:22:40.766990+00:00', 'start_ts': 1691828560.76699, 'state': 1, 'task_id': 104722957, 'version': '0.30.2', 'volume_id': 0, 'volume_name': 'DEFAULT', 'cg_name': None} >>> SESSION.getTaskDescendents(104722957).keys() dict_keys(['104722957', '104722959', '104723037', '104723106']) >> SESSION.getTaskInfo(104723037) {'arch': 'noarch', 'awaited': False, 'channel_id': 1, 'completion_time': '2023-08-12 08:24:46.445509+00:00', 'completion_ts': 1691828686.445509, 'create_time': '2023-08-12 08:22:40.876171+00:00', 'create_ts': 1691828560.876171, 'host_id': 402, 'id': 104723037, 'label': 'noarch', 'method': 'buildArch', 'owner': 1943, 'parent': 104722957, 'priority': 19, 'start_time': '2023-08-12 08:22:43.789668+00:00', 'start_ts': 1691828563.789668, 'state': 2, 'waiting': None, 'weight': 1.5365856122916666} >>> SESSION.listTags(2272961) [{'arches': 'None', 'id': 71268, 'locked': False, 'maven_include_all': False, 'maven_support': False, 'name': 'f40', 'perm': 'autosign', 'perm_id': 17}]
So the only tag name that I can see is f40.
f40
>>> SESSION.getBuildConfig('f40') {'arches': 'None', 'id': 71268, 'locked': False, 'maven_include_all': False, 'maven_support': False, 'name': 'f40', 'perm': 'autosign', 'perm_id': 17, 'extra': {'mock.new_chroot': 1, 'mock.package_manager': 'dnf'}, 'config_inheritance': {'extra': {}, 'arches': None}}
??
The relevant info is in buildroot itself. So for this build it is e.g. in x86_64 buildroot (from listBuildroots):
listBuildroots
br = {'arch': 'x86_64', 'br_type': 0, 'cg_id': None, 'cg_name': None, 'cg_version': None, 'container_arch': 'x86_64', 'container_type': 'chroot', 'create_event_id': 115933987, 'create_event_time': '2023-08-12 08:22:44.231163+00:00', 'create_ts': 1691828564.231163, 'extra': None, 'host_arch': None, 'host_id': 402, 'host_name': 'buildvm-x86-31.iad2.fedoraproject.org', 'host_os': None, 'id': 44840933, 'repo_create_event_id': 115928853, 'repo_create_event_time': '2023-08-12 05:46:04.026327+00:00', 'repo_id': 5330107, 'repo_state': 3, 'retire_event_id': 115934093, 'retire_event_time': '2023-08-12 08:24:46.430724+00:00', 'retire_ts': 1691828686.430724, 'state': 3, 'tag_id': 71278, 'tag_name': 'f40-build', 'task_id': 104723037} s.getBuildConfig(br['tag_id']) {'arches': 'i686 x86_64 aarch64 ppc64le s390x', 'id': 71278, 'locked': False, 'maven_include_all': False, 'maven_support': False, 'name': 'f40-build', 'perm': 'admin', 'perm_id': 1, 'extra': {'mock.new_chroot': 1, 'mock.package_manager': 'dnf'}, 'config_inheritance': {'extra': {'mock.new_chroot': {'id': 71268, 'name': 'f40'}, 'mock.package_manager': {'id': 71268, 'name': 'f40'}}, 'arches': None}}
You can use this, or generate whole mock config (better via CLI koji mock-config --task <task_id)). This is a bit clumsy and #3875 also proposes to store the mock config with the build, so it could be just downloaded and not reconstructed via CLI on user side. Anyway, in the mock config you should just need to replace repo urls with your new repo (as a) koji repo will probably not physically exist in that point of time and would need a special newRepo call to be recreated b) you want to populate the buildroot in some other way or from you repo outside of koji).
koji mock-config --task <task_id)
newRepo
Log in to comment on this ticket.