#9229 Improving the state of rust packaging experience
Opened 9 months ago by dustymabe. Modified 2 months ago

Currently if you are a rust packager you need to:

  • make changes to dist-git master branch
  • git pull && git checkout master && fedpkg build
  • ping @ignatenkobrain to do a build for you using this SOP and tag into a candidate tag
  • then create bodhi update

The ideal rust packager workflow would be the same as other fedora packagers:

  • merge PR to master branch
  • git pull && git checkout master && fedpkg build
  • backport PR to release branch (i.e. f31) and push to distgit
  • git pull && git checkout f31 && fedpkg build
  • then create bodhi update

Currently the extra bits are needed because sidetag admins can't unblock/block packages. There is some ongoing work that needs to be done in order to allow users to self serve (i.e. not include @ignatenkobrain in the process). There is also some other work to be done to make the fedpkg build experience the same. I'll enumerate the work below:

Can we organize some parties together to try to make progress on making the rust packaging experience better in Fedora? I think we probably need to include:

  • Fedora releng
  • Fedora rust sig
  • koji developers
  • fedpkg developers

I don't think that ideal workflow would be to checkout f31 branch. The ideal workflow would be to run fedpkg --release f31 build from master branch and it should do the thing. Ofc, if you specifically need to diverge from master, you should have separate branch.

I don't think that ideal workflow would be to checkout f31 branch. The ideal workflow would be to run fedpkg --release f31 build from master branch and it should do the thing. Ofc, if you specifically need to diverge from master, you should have separate branch.

I mostly put that because it's how it's done for every other package. I think having it not be a special case would help, but I do see why not having a separate branch could be attractive.

Metadata Update from @mohanboddu:
- Issue tagged with: meeting

9 months ago

I'm interested in this process to build Golang binaries the same way for stable branches and EPEL8 if possible. We don't use DBR in Golang so this might work with EPEL8 although I'm not sure because we would need to import base package like redhat-rpm-config to use our macros over RHEL's one. This would be great if this could work because as of now we can't build any Go packages in EPEL8 due to them using too old redhat-rpm-config.

Hopefully this is coming in koji 1.22 (@tkopecek), so in a few months we should be good.

I believe that all necessary pieces have landed into the 1.21, so I guess this is waiting for koji to be updated in infra and then I can submit a patch with necessary configuration for kojihub policies.

ok I'm following the process from https://gist.github.com/ignatenkobrain/f2529a3f9e34848fa63587db94089a0f and I'm hitting a policy violation:

$ pwd 
/var/b/shared/code/src.fedoraproject.org/rpms/rust-coreos-installer
$ export release=31
$ export build=$(fedpkg verrel)
$ export pkg=${build%-*-*}
$ export tag=$(koji add-sidetag "f${release}-build")
$ export buildroot=$(./rust-buildroot.py list-buildroot $build)
$ export buildroot_pkgs=$(for b in $buildroot; do echo ${b%-*-*}; done | grep '^rust-' | sort)
$ koji add-pkg --owner releng $tag $buildroot_pkgs $pkg --force
Adding 130 packages to tag f31-build-side-25822
2020-07-24 15:41:11,933 [ERROR] koji: ActionNotAllowed: policy violation (package_list)

I tried with using my name as the owner too with just a single package:

$ koji add-pkg --owner dustymabe f31-build-side-25822 rust-coreos-installer --force
Adding 1 packages to tag f31-build-side-25822
2020-07-24 15:38:32,043 [ERROR] koji: ActionNotAllowed: policy violation (package_list)

Still get a policy violation. What updates do we need to make for this?

Okay, we have to discuss this in our next meeting and come up with a plan.

I think we just need to allow hub policy to allow this.

This didn't work for me either when @ignatenkobrain had me try to do this a while back...

I think this may need the 'update' action there... you are 'updating' the pkglist. (and 'remove' to remove pkgs)

Yes. I've applied that, can you try again now?

still doesn't work for me:

$ koji add-pkg --owner dustymabe f31-build-side-25908 rust-coreos-installer --force
Adding 1 packages to tag f31-build-side-25908
2020-07-26 16:00:30,472 [ERROR] koji: ActionNotAllowed: policy violation (package_list)

$ koji add-pkg --owner releng f31-build-side-25908 rust-coreos-installer --force
Adding 1 packages to tag f31-build-side-25908
2020-07-26 16:00:39,938 [ERROR] koji: ActionNotAllowed: policy violation (package_list)

Huh, that tag doesn't seem to exist? are you sure thats right?

Huh, that tag doesn't seem to exist? are you sure thats right?

Yeah. I cleaned it up with fedpkg remove-side-tag. When we're ready for me to try again I'll re-execute the steps in https://pagure.io/releng/issue/9229#comment-667247

I know testing different configurations can be hard when you have superroot powers. I'm perfectly willing to test drive any changes you make.

@mikem We have a config in place to allow side tag owners to manipulate the side tag, buts its not working.

From https://pagure.io/releng/issue/9229#comment-667358, @dustymabe is the owner of the side tag but still he is not able to add packages to the tag.

Hmm, policy looks correct. I've tested similar policy (with this exact rule) and it is working for me. You can see how each rule in policy was evaluated in hub's log with DEBUG level. It is always covered between "policy start" and "policy done" messages followed by potential traceback if policy led to refusing action.

Should look similar to this (stripped timestamps)

[DEBUG] m=packageListAdd u=test2 p=2379 r=192.168.122.71:60422 koji.policy: policy start
[DEBUG] m=packageListAdd u=test2 p=2379 r=192.168.122.71:60422 koji.db: b'\\nSELECT id, name, status, usertype\\n  FROM users\\n  LEFT JOIN user_krb_principals ON users.id = user_krb_principals.user_id\\n WHERE (users.id = 29)\\n \\n \\n\\n \\n'
[DEBUG] m=packageListAdd u=test2 p=2379 r=192.168.122.71:60422 koji.db: Execute operation completed in 0.0004 seconds
[DEBUG] m=packageListAdd u=test2 p=2379 r=192.168.122.71:60422 koji.db: fetchall operation completed in 0.0000 seconds
[DEBUG] m=packageListAdd u=test2 p=2379 r=192.168.122.71:60422 koji.db: b'\\nSELECT krb_principal\\n  FROM user_krb_principals\\n\\n WHERE (user_id = 29)\\n \\n \\n\\n \\n'
[DEBUG] m=packageListAdd u=test2 p=2379 r=192.168.122.71:60422 koji.db: Execute operation completed in 0.0002 seconds
[DEBUG] m=packageListAdd u=test2 p=2379 r=192.168.122.71:60422 koji.db: fetchall operation completed in 0.0000 seconds
[DEBUG] m=packageListAdd u=test2 p=2379 r=192.168.122.71:60422 koji.db: b'\\nSELECT tag_config.arches, tag.id, tag_config.locked, tag_config.maven_include_all, tag_config.maven_support, tag.name, permissions.name, tag_config.perm_id\\n  FROM tag_config\\n  JOIN tag O
ag_config.active = TRUE))\\n   AND (tag.id = 211)\\n \\n \\n\\n \\n'
[DEBUG] m=packageListAdd u=test2 p=2379 r=192.168.122.71:60422 koji.db: Execute operation completed in 0.0005 seconds
[DEBUG] m=packageListAdd u=test2 p=2379 r=192.168.122.71:60422 koji.db: fetchall operation completed in 0.0000 seconds
[DEBUG] m=packageListAdd u=test2 p=2379 r=192.168.122.71:60422 koji.db: b'\\nSELECT key, value\\n  FROM tag_extra\\n\\n WHERE ((tag_extra.active = TRUE))\\n   AND (tag_id = 211)\\n \\n \\n\\n \\n'
[DEBUG] m=packageListAdd u=test2 p=2379 r=192.168.122.71:60422 koji.db: Execute operation completed in 0.0004 seconds
[DEBUG] m=packageListAdd u=test2 p=2379 r=192.168.122.71:60422 koji.db: fetchall operation completed in 0.0000 seconds
[DEBUG] m=packageListAdd u=test2 p=2379 r=192.168.122.71:60422 koji.policy: is_sidetag_owner  -> True
[DEBUG] m=packageListAdd u=test2 p=2379 r=192.168.122.71:60422 koji.policy:  match action add block unblock  -> True
[DEBUG] m=packageListAdd u=test2 p=2379 r=192.168.122.71:60422 koji.policy: matched: action=allow
[DEBUG] m=packageListAdd u=test2 p=2379 r=192.168.122.71:60422 koji.policy: policy done
[ERROR] m=packageListAdd u=test2 p=2379 r=192.168.122.71:60422 koji.hub: policy package_list gave allow, reason: , last rule: is_sidetag_owner && match action add block unblock :: allow

Does your instance have the fix from https://pagure.io/koji/pull-request/2322 (included in koji 1.22)? There's an important fix to the policy rule handlers in there.

We do not, and thats likely it. :) Thanks @mikem!

Thanks @mikem @tkopecek !

@kevin, @mohanboddu, is there a time we can schedule and update to koji to see if this fixes the problem?

https://pagure.io/fedora-infrastructure/issue/9194 is now closed. I'll try to test this workflow again tomorrow.

Looks like with the new koji update the koji add-pkg commands are working now. So the user can self service build packages.

It seems like the only remaining piece is automating the steps (see them laid out in https://pagure.io/releng/issue/9661) with fedpkg build. Is that correct?

@ignatenkobrain @decathorpe - how does https://fedoraproject.org/wiki/Changes/Rust_Crate_Packages_For_Release_Branches overlap with the goal of this ticket?

The "Instead, rust packages can be built like any other package in fedora." makes me think they are the same thing.

@dustymabe I was not aware of this ticket before you pinged me just now :(

It's not entirely the same as your proposal, since mine completely eliminates the need for special koji actions (creating side tags, tagging buildroot contents, building package in side tag, untagging buildroot contents). Instead, packagers should be able to build Rust packages just like any other package in fedora (also making changes to fedpkg and other scripts unnecessary).

@decathorpe even though it's not the exact same it has the same end goal, which is to make rust packaging at least "feel" like the normal package workflow. Either way works for me.

The F34 Change Proposal to make Rust packaging work like everything else was accepted, and the releng changes are submitted here:

https://pagure.io/releng/pull-request/9798

I think this should address this ticket too?

Login to comment on this ticket.

Metadata