#9308 F33 system wide change: Sqlite Rpmdb
Closed: Fixed 5 months ago by kevin. Opened a year ago by pmatilai.

Change rpmdb to the new sqlite format:
https://fedoraproject.org/w/index.php?title=Changes/Sqlite_Rpmdb

This depends on rpm rebase to 4.16 (#9286) but otherwise should be considered independently. Mass rebuild is not needed, the change does not affect built packages.

Container/chroot usage scenarios are affected as older rpm versions do not support the sqlite format and thus older distro versions cannot be used to query/manipulate the rpmdb in sqlite format. This likely includes distro compose tooling.


@pmatilai From our meeting today:

[14:07:12] <mboddu> .releng 9308
[14:07:14] <zodbot> mboddu: Issue #9308: F33 system wide change: Sqlite Rpmdb - releng - Pagure.io - https://pagure.io/releng/issue/9308
[14:07:50] <mboddu> I dont think it will affect any of the container building
[14:08:10] <mboddu> But just wanted to check if it has any other impacts before I close the ticket
[14:09:26] <cverna[m]> the only downside is that it will take longer to build the dnf metadata cache
[14:09:27] <nirik> so we need to carry a newer rpm for builders. again. ;(
[14:09:46] <cverna[m]> but other than that I don't see any problem
[14:10:02] <mboddu> cverna[m]: Okay
[14:10:26] <cverna[m]> for the sqlite I guess we just need to make a new release when the change lands
[14:10:29] <nirik> or override things to use the old db format until builders are upgraded to the new one
[14:10:32] <cverna[m]> with the new version of rpm
[14:11:23] <mboddu> nirik: Right that is a problem
[14:12:57] <nirik> we could/should ask how to override the sqlite choice... if we can do that in mock/koji that would be much easier than carrying a rpm in infra tags

Can you answer the question?

Thanks.

Metadata Update from @mohanboddu:
- Issue tagged with: changes, f33

a year ago

@mohanboddu For specific build tags, you can set %_db_backend macro to bdb to override the rpmdb backend it uses by default. This is probably what you want to do for < el9 unless sqlite rpmdb is backported to el8.

Yes, as @ngompa said you can override the default with the %_db_backend macro where needed.

That should avoid the need for a special new rpm on the builders for package building, but if you do that for Fedora >= 33 images, those images will end up having the old database format as default which we don't want. That should be avoidable by adding a 'rpmdb --rebuilddb' within the image chroot as the last step or such, but it's an extra step nevertheless, and if that route is taken then we'll miss a good deal of testing of the new backend in the form of koji churn. Note that I'm not fully up to speed with recent mock developments, feel free to correct me if I'm wrong, I'm assuming it still does (at least the initial) install with the host rpm and not some special bootstrap thing.

Which is to say that having rawhide rpm on the builders would be beneficial to the cause. I do get that it's far from ideal to you otherwise, especially as that rpm isn't even a stable release yet.

My initial plan was to enable sqlite in Fedora 33 but only switch the default in 34 to avoid this kind of situation, but I was asked to try and accelerate things because of - you guessed it - RHEL-9. I wont take offense if people say "NO", I'm almost expecting it. The point here is to see if this can be worked out somehow without placing unreasonable burden to others.

Note that I'm not fully up to speed with recent mock developments, feel free to correct me if I'm wrong, I'm assuming it still does (at least the initial) install with the host rpm and not some special bootstrap thing.

@pmatilai Mock itself does a special bootstrap thing by default, but Koji disables that functionality in Mock and reverts to the old behavior that you know and "love". There's an RFE to fix that, though it was filed mainly to deal with lack of YUM on Fedora, but having that implemented would also help in this scenario.

Provided that things work the way I think they do for Koji tags with macro settings and inheritance, if we upgrade rpm on the builders to F33 rpm on F32 (at least until F33 is out and everything can be reprovisioned), then we can just set for this backported rpm to use bdb by default, and then set f33 tag to use sqlite using the %_db_backend macro with the Koji feature to set macros in tags. I claim ignorance on whether tag inheritance also includes macro properties (I hope they do).

@dgregor, @mikem, or @tkopecek could probably answer with some clarity there on whether my idea would work.

Yeah I knew mock has some kind of bootstrap feature but last I heard, it still depended on the host rpm to some extent and thus was limited in what it could support. But seems there has been some very recent development to bootstrap via a pre-built image which finally achieves my long-standing dream: https://github.com/rpm-software-management/mock/wiki/Feature-container-for-bootstrap

@pmatilai and @ngompa Thanks for your input.

Also, from our releng meeting today:

#info mboddu will comment on the ticket asking them to let us know when the work is done so that we can do some testing in staging.

Please let us know when the work is done, we will run some tests in staging.

Thanks.

Metadata Update from @mohanboddu:
- Issue tagged with: change-ack

a year ago

@mohanboddu @kevin Side note: What would it take to enable bootstrap mock in Koji?

I am not sure. Last time we tried it just didn't work at all... but we could try in staging now and see?

Where do you want to track this? This ticker, another releng ticker, an infra ticket, or somewhere else?

I guess another releng ticket? blocking on the koji upstream one? Or we could bypass that one if we just were able to use bootstrap mode for everything I guess.

Please let us know when the work is done, we will run some tests in staging.

Eek, I nearly forgot this. I assume "work is done" here means 4.16 landing on rawhide? In that case, it's been there for about a week now, no major issues found so far. So I believe we're good to proceed with testing when you have time to do so. Thanks.

so, I enabled bootstrap mode in staging, but I am not sure it's working as I am not sure of the current status of staging. :(

@mohanboddu and @pingou Whats the current status of staging? we have all the f33 tags and targets, but all the old f32 builds in them?

Builds are definitely working, but it's unclear to me if they are using the right bootstrap...

As far as I can tell, if you build from master it goes to f33 and if you build from f32 it goes to f32, I am not sure if that answers your question though.

https://src.stg.fedoraproject.org/rpms/rpm commit are from 3 months ago. Let's sync that with real master + enable the sqlite DB there to test it in staging?

@kevin Once more, you don't need to enable anything like bootstrap mode. The only case which might break is calling rpm during the build and only in case RPM in chroot does not support host RPMDB format. This can become only a problem when you upgrade builders to F33, but again, only less than 10 packages in Fedora will break due to this. And they should not call rpm from inside chroot.

I think we can safely conclude by now that there's no need to temporarily switch back to 4.15.x due to alpha instability, so I'd really, really like to move on with the dabatase change now.

Rel-eng, when can I hope to proceed?

@kevin We probably want to backport and configure https://pagure.io/koji/pull-request/2166 in Fedora Koji if we really are worried about this. (I'm not...)

Otherwise, we should be ready to go based on what @ignatenkobrain has said.

First: we are in f32 final freeze and do not yet have a signed off rc. So, no, we are not going to enable this in prod now. :smile:

Second: I would really like to see this work in staging. If we don't need bootstrap mode, great. I can disable that. What do we need then? A updated rpm on the builders with sqlite support and a tag config to enable that? An updated rpm with sqlite default on builders and tag config to disable that for non rawhide? Or something else? I want to see actual builds work with whatever we are proposing to do to prod in staging first. :interrobang:

You need to do absolutely nothing from my POV :)

@pmatilai can you give me a patch which you'd like to apply to RPM.spec (better push it to master, but don't build) and I can build it in staging and try it out there?

Or just push it to staging git?

The systemd unit patches are incomplete. I'm providing feedback to @pmatilai in https://bugzilla.redhat.com/show_bug.cgi?id=1826658

Okay so F32 needs to go out first, ack, I can live with that.

My understanding is that nothing at all would break if we just flick the switch in rawhide rpm package, and do nothing else. Things might be a little inconsistent with BDB still used in places (package and image building the latter would then be converted on first boot) but it should be harmless.

Sorting the inconsistency would require either custom rpm version and distro specific overrides for database backend on builders, or mock boostrap enabled in koji. The latter seems far, far preferable from my POV. But if we can live with the slight inconsistency temporarily I think we could do this in two steps, with rawhide rpm leading the way and koji following whenever it suits rel-eng.

As for staging, I only know it exists someplace. I haven't got a slightest clue where it is or what is there, or how I may (or not) use it. Which makes this a bit frustrating hand-waving experience for me :smile:

@ignatenkobrain , no pushing patches to dist-git until we're ready to go, we need to be able to do new builds for other fixes in the meanwhile. Something like this at end of %prep or top of %build will do the trick: sed -i -e "s: bdb: sqlite:g" macros.in

And some builds with it, eg https://koji.stg.fedoraproject.org/koji/buildinfo?buildID=1432318 and https://koji.stg.fedoraproject.org/koji/buildinfo?buildID=1432321, latter of which does 'rpm -q foo' on spec evaluation. No nuclear meltdown observed, and the version retrieved by said query is sane. FWIW.

I guess @sgallagh ran into the same issue with redhat-rpm-config depending on unavailable things, and untagged it while I was looking for the magic koji incantation to do so, leading to certain amount of confusion on this end when the problematic package didn't seem to be in any tag at all :joy:

@pmatilai I'm working with @mohanboddu to get the missing fonts-rpm-macros package into the staging environment. I'll update here when that's available.

Oh, I don't actually need the new redhat-rpm-config for anything particular, it's just the first thing that occurred me to test, and didn't realize it would break it all.

Just to note: staging Is using bootstrap chroots. I don't know if it's working, but its set to use them.

SOme things that appear not working right:

DEBUG util.py:600: Unable to detect release version (use '--releasever' to specify release version)

DEBUG util.py:698: Executing command: /bin/rpm --root /var/lib/mock/f33-build-19138962-9005183/root -qa with env {'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOME': '/builddir', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'PROMPT_COMMAND': 'printf "\033]0;<mock-chroot>\007"', 'PS1': '<mock-chroot> \s-\v\$ ', 'LANG': 'C.UTF-8'} and shell True
DEBUG util.py:600: error: cannot open Packages index using db5 - Permission denied (13)
DEBUG util.py:600: error: cannot open Packages database in /var/lib/mock/f33-build-19138962-9005183/root/var/lib/rpm
DEBUG util.py:600: error: cannot open Packages index using db5 - Permission denied (13)
DEBUG util.py:600: error: cannot open Packages database in /var/lib/mock/f33-build-19138962-9005183/root/var/lib/rpm
DEBUG util.py:753: Child return code was: 1

It looks like perhaps the bootstrap chroot hasn't updated with the new rpm...

Okay so F32 needs to go out first, ack, I can live with that.
My understanding is that nothing at all would break if we just flick the switch in rawhide rpm package, and do nothing else. Things might be a little inconsistent with BDB still used in places (package and image building the latter would then be converted on first boot) but it should be harmless.
Sorting the inconsistency would require either custom rpm version and distro specific overrides for database backend on builders, or mock boostrap enabled in koji. The latter seems far, far preferable from my POV. But if we can live with the slight inconsistency temporarily I think we could do this in two steps, with rawhide rpm leading the way and koji following whenever it suits rel-eng.

ok. I guess we would need a test image build in staging.

Oh, I don't actually need the new redhat-rpm-config for anything particular, it's just the first thing that occurred me to test, and didn't realize it would break it all.

Regardless, this is fixed and the redhat-rpm-config package is tagged back in now.

Just to note: staging Is using bootstrap chroots. I don't know if it's working, but its set to use them.

Ah. That explains a few things. For one, I was expecting a warning messages from the rpm query in spec but didn't see one, but this explains (so no need to dig deeper then).

Always useful to know what exactly you're testing...

SOme things that appear not working right:
DEBUG util.py:698: Executing command: /bin/rpm --root /var/lib/mock/f33-build-19138962-9005183/root -qa with env {'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOME': '/builddir', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'PROMPT_COMMAND': 'printf "\033]0;<mock-chroot>\007"', 'PS1': '<mock-chroot> \s-\v\$ ', 'LANG': 'C.UTF-8'} and shell True
DEBUG util.py:600: error: cannot open Packages index using db5 - Permission denied (13)
DEBUG util.py:600: error: cannot open Packages database in /var/lib/mock/f33-build-19138962-9005183/root/var/lib/rpm

I missed this yesterday due to looking for messages in the other direction, thanks for pointing this out. So this is mock querying for installed packages in the chroot (from another log):

INFO backend.py:218: Installed packages:
DEBUG util.py:772: child environment: None
DEBUG util.py:698: Executing command: /bin/rpm --root /var/lib/mock/f33-build-19138983-9005212/root -qa with env {'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOME': '/builddir', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'PROMPT_COMMAND': 'printf "\033]0;<mock-chroot>\007"', 'PS1': '<mock-chroot> \s-\v\$ ', 'LANG': 'C.UTF-8'} and shell True

That amounts to a bug in mock I would say, in a bootstrap mode it should gather the info using the rpm in the chroot, not the one from the outside. I'll file a bug.

The good news is that this is (AFAICS) informational only, and the same information can be gathered from the root.log.

https://github.com/rpm-software-management/mock/issues/565 - afaics there are two cases, the above harmless informational issue and a second not so harmless but only affects dynamic buildrequires. I might've missed something of course.

ok. I guess we would need a test image build in staging.

Would be happy to test, but don't have a permission to do so it seems:

[pmatilai🎩︎lumikko fedora-kickstarts]$ stg-koji spin-livemedia rawhide-minimal 33 f33 x86_64 fedora-disk-minimal.ks
[====================================] 100% 00:00:00 201.00 B 1007.78 B/sec
2020-04-24 13:17:38,049 [ERROR] koji: ActionNotAllowed: livemedia permission required

(not that I know whether the above command would actually accomplish something meaningful, this is obviously my first foray into this territory)

afaics there are two cases, the above harmless informational issue and a second not so harmless but only affects dynamic buildrequires

So the dynamic buildrequires case that actually affects builds is fixed in mock 2.2, but staging has 2.1 and I don't have permissions to update it. Testing with entirely different mix of versions than what we'll be using is of somewhat limited use, but of course better than nothing.

I've updated all the staging builders to latest f31 updates (including mock 2.2).

I did a scratch image build: https://koji.stg.fedoraproject.org/koji/taskinfo?taskID=90044946 and it seems to have worked fine, and from the build tree I did see it using sqlite rpm db.

I'll try and test the iso later, or if someone else wants to, please feel free.

If anyone could test any corner cases in stg now that would be great. Then after f32 goes out next week we could upgrade koji to 1.21.0 and builders to all f31 updates and enable bootstrap chroot and switch rpm in rawhide to sqlite by default.

Then after f32 goes out next week we could upgrade koji to 1.21.0 and builders to all f31 updates and enable bootstrap chroot and switch rpm in rawhide to sqlite by default.

Any update on when I could expect this to happen? Just asking for a schedule so I don't unnecessarily spin wheels in "maybe tomorrow?" mode, this is now literally the final piece blocking the switch to sqlite.

So, we hit a bug in mock:

https://github.com/rpm-software-management/mock/issues/571

so, waiting on that.

So, what would you think of just landing sqlite rpm in rawide whenever now and we live with the images having the old format for a bit until we can schedule the builder/koji/mock updates and enable bootstrap?

Fine with me if it's fine with rel-eng. This is a scenario we haven't actually tested in koji though, because staging was configured differently.

But okay, I'll flick the switch tomorrow morning to have the full day to babysit it just in case. Unless there's a "wait, stop!" message in this space before then, that is.

In case you want to tripple-check, you could:

  • request a side tag with fedpkg request-side-tag
  • build the flicked rpm in that side tag
  • throw a bunch of scratch builds to that side tag and observe
  • create a rawhide bodhi update from that side tag when happy

Thanks for the tip, although I did run out of patience in the end (been waiting for this moment since early December really).

So we're live with the change now in rawhide, some packages have been built by it. There are a couple of "warning: Found bdb Packages database while attempting sqlite backend: using bdb backend." messages in build logs due to mock running rpm inside the chroot but this is as per expectations, if those messages were not there it would be fishy.

Any news here? Mock 2.3 (fixing the issues discovered so far) is out and in Fedora since May 22nd.

Yes, but our datacenter move has started.

That said, the new dc will have f32 builders and new mock... I don't think we want to enable bootstrap until after the move however.

Yup, I think I made the previous comment before seeing the datacenter reminder message on ml.
The timing is unfortunate as I'm going on a long vacation at end of June, ie just about when this change is about to finally land, but it is what it is. In the meanwhile, good luck with the move :smile:

So, this dropped off my radar. ;)

We are in the new datacenter. We are on f32 builders with new mock.

We don't have bootstrap enabled, but... apparently we don't really need to?

Is there anything further to do here? Or do we need bootstrap still for some reason?

We technically shouldn't need bootstrap until we move builders to f33 (but especially f34, when bdb support will be ripped out).

@pmatilai Also the beta build of rpm 4.16 is okay for the mass rebuild or should we need to wait?
If its need to waited, it needs a FESCo approval as the mass rebuild is scheduled for tomorrow.

Metadata Update from @mohanboddu:
- Issue tagged with: mass rebuild

11 months ago

Back from vacation.

We might not technically need bootstrap to keep things limping along for the time being, but we want to enable it the sooner the better. I thought there was an agreement to do so once the datacenter move is out of the way, with no further ado.

There are any number of reasons to enable it, but to remind you of a few:
- images will get built with correct rpmdb from the go
- koji churn is further testing of sqlite backend
- open a whole new level of testing of new rawhide stuff
- shake out any possible remaining bootstrap issues while it is still technically optional!

There was some pushback that bootstrap builds would be slower... but our stg env isn't back yet to do any testing on. I'll do so when I can.

Whether it's slower or not is kinda moot when it doesn't work without.

While it appears to sorta work now (except for image builds getting wrong rpmdb and similar), it will stop doing so, and we don't want to get caught with our pants down at that point.

For memory, https://bugzilla.redhat.com/1865223 (php-pecl-rpminfo is FTBFS because of this)

Ping?

F33 is out, and for F34 the plan is to remove the libdb dependency entirely (https://bugzilla.redhat.com/show_bug.cgi?id=1787311, https://bugzilla.redhat.com/show_bug.cgi?id=1778802), but we're still stuck on this item.

Yes, this is still on my radar. We are waiting for one last firewall fix on our staging s390x builder, then staging should be back to fully working and I can do some testing of this. :)

We actually had it enabled for a day or so and only 2 items broke:

  • kojid wasn't cleaning up boostrap mock repos, so builders filled up disk space. This has since been fixed.

  • epel7 broke horribly because we have devtoolset enabled/available and for some reason a library gets pulled from that for the base repo and blows up because it's the wrong version and yum won't run in the repo. This is what I need to debug.

I also hope to get some benchmarks... at least ballpark.

epel7 broke horribly because we have devtoolset enabled/available and for some reason a library gets pulled from that

That is caused by historical bugs in SCL packages :-( we have

config_opts['yum_install_command'] += " --disablerepo=centos-sclo*"

in the mock-core-configs configuration (with some x86_64 jinja sugar).

Thanks @praiskup ! will give that a try...

ok. I got that working... :)

Our stg koji/builders are now using bootstrap mode... feel free to test it out (but note that the s390x builder isn't working currently, it's waiting on a firewall rule change).

I've just broken python3-six in a side tag https://koji.stg.fedoraproject.org/koji/builds?inherited=0&tagID=31516&order=-build_id&latest=1 -- https://src.stg.fedoraproject.org/rpms/python-six/c/8bee2f3aeaf24580755ace77812e5fb3b0ae51e6?branch=breakage

The bootstrap mock root uses python3-six (it is brought by dnf) and when the bootstrap mock root is created, as expected, it fails.

https://koji.stg.fedoraproject.org/koji/taskinfo?taskID=90050931

Start(bootstrap): dnf install
ERROR: Command failed: 
 # /usr/bin/dnf --installroot /var/lib/mock/f34-build-side-31516-23218148-9002671-bootstrap/root/ --setopt=install_weak_deps=0 --disableplugin=local --disableplugin=spacewalk install dnf dnf-plugins-core --setopt=tsflags=nocontexts
Unable to detect release version (use '--releasever' to specify release version)
No matches found for the following disable plugin patterns: local, spacewalk
Last metadata expiration check: 0:00:19 ago on Wed Nov 25 16:37:53 2020.
Error: 
 Problem: package python3-dnf-plugins-core-4.0.16-4.fc33.noarch requires python3-dateutil, but none of the providers can be installed
  - package dnf-plugins-core-4.0.16-4.fc33.noarch requires python3-dnf-plugins-core = 4.0.16-4.fc33, but none of the providers can be installed
  - package python3-dateutil-1:2.8.1-2.fc33.noarch requires python3.9dist(six) >= 1.5, but none of the providers can be installed
  - package python3-dateutil-1:2.8.1-2.fc33.noarch requires python3-six, but none of the providers can be installed
  - conflicting requests
  - nothing provides xxxx-nope-nope-xxxx needed by python3-six-1.15.0-2.fc34.noarch
(try to add '--skip-broken' to skip uninstallable packages)

This is a bummer, because as soon as we strat building new Python version in a side tag, this is what will happen.

  • Is it possible to turn the bootstrap option off in a specific side tag to prevent this from happening?

Perhaps? :)

try:

koji edit-tag f34-build-side-31516 -x 'mock.use_bootstrap=0'

  • Is it possible to use regular rawhide as a bootstrap mock root?

Nope, not that I know of. Might be possible though with koji patching/work

yeah I don't think thats a good approach.

$ stg-koji edit-tag f34-build-side-31516 -x 'mock.use_bootstrap=0'
2020-12-01 12:08:47,417 [ERROR] koji: ActionNotAllowed: tag permission required (logged in as churchyard)

Could you please do that for us, so I can test a build?

done. But that I guess is a drawback here... we don't have allow sidetag owners to adjust extra data currently because then they could adjust arbitrary mock config. ;(

Well, for the Python side tag, we request that by a releng ticket anyway.

OK, that fixed the problem, as expected. I think it's fine to note when we do the change that in situations like this, it is possible to opt-out from mock's bootstrap option via releng. I expect this will impact very little group of maintainers, possibly just us.

ok, bootstrap is active in prod now. I'm still working on upgrading the arm (aarch64/armv7) builders to f33, then I will announce all that...

Arm builders are done now too... so closing this now.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

5 months ago

It was a long time coming, but finally there! :fireworks:
Thanks!

Login to comment on this ticket.

Metadata