#7125 Please rebuild the f26 base container image
Closed: Fixed 6 years ago Opened 6 years ago by bowlofeggs.

Greetings!

I've been experiencing an extreme rate of CI failures in Bodhi's test suite due to dnf failing to install Bodhi's dependencies with this error:

error: db5 error(-30969) from dbenv->open: BDB0091 DB_VERSION_MISMATCH: Database environment version mismatch

@ignatenkobrain advised me that this was due to a mismatch between dnf and glibc and that I need a refreshed container image to resolve the problem. Hence, my request ☺


Metadata Update from @mohanboddu:
- Issue assigned to maxamillion

6 years ago

Done. Sorry for the lag time, I meant to do this but got busy and it fell off the map.

Wait, do we also need a refresh on the docker hub or just in our registry?

Refreshing docker hub would help some other users who have been experiencing this problem too.

Metadata Update from @maxamillion:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

6 years ago

The current f26 container doesn't seem to have the newest available glibc or libsolv, which means the problem still occurs. This is on a machine that has no containers pulled at all:

sudo docker run registry.fedoraproject.org/fedora:26 dnf upgrade -y
Unable to find image 'registry.fedoraproject.org/fedora:26' locally
Trying to pull repository registry.fedoraproject.org/fedora ... 
sha256:e2e195a50de87661b4a9a54123ea820cb6e03d7189a96a89a619482ed48915a1: Pulling from registry.fedoraproject.org/fedora
c9744bec7352: Pull complete 
Digest: sha256:e2e195a50de87661b4a9a54123ea820cb6e03d7189a96a89a619482ed48915a1
Status: Downloaded newer image for registry.fedoraproject.org/fedora:26
Fedora 26 - x86_64 - Updates                    5.3 MB/s |  16 MB     00:03    
Fedora 26 - x86_64                              7.9 MB/s |  53 MB     00:06    
Last metadata expiration check: 0:00:05 ago on Tue Nov  7 19:18:31 2017.
Dependencies resolved.
================================================================================
 Package                 Arch         Version               Repository     Size
================================================================================
Upgrading:
 audit-libs              x86_64       2.8.1-1.fc26          updates       110 k
 curl                    x86_64       7.53.1-12.fc26        updates       317 k
 glibc                   x86_64       2.25-12.fc26          updates       3.4 M
 glibc-common            x86_64       2.25-12.fc26          updates       890 k
 glibc-langpack-en       x86_64       2.25-12.fc26          updates       291 k
 gnutls                  x86_64       3.5.16-1.fc26         updates       756 k
 libcrypt-nss            x86_64       2.25-12.fc26          updates        54 k
 libcurl                 x86_64       7.53.1-12.fc26        updates       272 k
 libdnf                  x86_64       0.11.1-1.fc26         updates       128 k
 libsolv                 x86_64       0.6.30-1.fc26         updates       365 k
 popt                    x86_64       1.16-12.fc26          updates        58 k
 python3-hawkey          x86_64       0.11.1-1.fc26         updates        54 k
 systemd                 x86_64       233-7.fc26            updates       3.0 M
 systemd-libs            x86_64       233-7.fc26            updates       454 k
 systemd-pam             x86_64       233-7.fc26            updates       176 k

Transaction Summary
================================================================================
Upgrade  15 Packages

Total download size: 10 M
Downloading Packages:
(1/15): audit-libs-2.8.1-1.fc26.x86_64.rpm      328 kB/s | 110 kB     00:00    
(2/15): libcurl-7.53.1-12.fc26.x86_64.rpm       554 kB/s | 272 kB     00:00    
(3/15): curl-7.53.1-12.fc26.x86_64.rpm          549 kB/s | 317 kB     00:00    
(4/15): glibc-langpack-en-2.25-12.fc26.x86_64.r 742 kB/s | 291 kB     00:00    
(5/15): libcrypt-nss-2.25-12.fc26.x86_64.rpm    564 kB/s |  54 kB     00:00    
(6/15): glibc-common-2.25-12.fc26.x86_64.rpm    1.2 MB/s | 890 kB     00:00    
(7/15): libdnf-0.11.1-1.fc26.x86_64.rpm         1.4 MB/s | 128 kB     00:00    
(8/15): python3-hawkey-0.11.1-1.fc26.x86_64.rpm 926 kB/s |  54 kB     00:00    
(9/15): gnutls-3.5.16-1.fc26.x86_64.rpm         1.8 MB/s | 756 kB     00:00    
(10/15): libsolv-0.6.30-1.fc26.x86_64.rpm       1.8 MB/s | 365 kB     00:00    
(11/15): popt-1.16-12.fc26.x86_64.rpm           816 kB/s |  58 kB     00:00    
(12/15): systemd-pam-233-7.fc26.x86_64.rpm      1.7 MB/s | 176 kB     00:00    
(13/15): systemd-libs-233-7.fc26.x86_64.rpm     1.9 MB/s | 454 kB     00:00    
(14/15): glibc-2.25-12.fc26.x86_64.rpm          2.2 MB/s | 3.4 MB     00:01    
(15/15): systemd-233-7.fc26.x86_64.rpm          3.8 MB/s | 3.0 MB     00:00    
--------------------------------------------------------------------------------
Total                                           3.7 MB/s |  10 MB     00:02     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Running scriptlet: systemd                                                1/1 
  Preparing        :                                                        1/1 
  Upgrading        : glibc-common-2.25-12.fc26.x86_64                      1/30 
  Upgrading        : glibc-langpack-en-2.25-12.fc26.x86_64                 2/30 
  Running scriptlet: glibc-2.25-12.fc26.x86_64                             3/30 
  Upgrading        : glibc-2.25-12.fc26.x86_64                             3/30 
  Running scriptlet: glibc-2.25-12.fc26.x86_64                             3/30 
  Upgrading        : libsolv-0.6.30-1.fc26.x86_64                          4/30 
  Running scriptlet: libsolv-0.6.30-1.fc26.x86_64                          4/30 
  Upgrading        : libdnf-0.11.1-1.fc26.x86_64                           5/30 
  Running scriptlet: libdnf-0.11.1-1.fc26.x86_64                           5/30 
  Upgrading        : audit-libs-2.8.1-1.fc26.x86_64                        6/30 
  Running scriptlet: audit-libs-2.8.1-1.fc26.x86_64                        6/30 
  Upgrading        : libcurl-7.53.1-12.fc26.x86_64                         7/30 
  Running scriptlet: libcurl-7.53.1-12.fc26.x86_64                         7/30 
  Upgrading        : libcrypt-nss-2.25-12.fc26.x86_64                      8/30 
  Running scriptlet: libcrypt-nss-2.25-12.fc26.x86_64                      8/30 
  Upgrading        : systemd-libs-233-7.fc26.x86_64                        9/30 
  Running scriptlet: systemd-libs-233-7.fc26.x86_64                        9/30 
  Upgrading        : systemd-pam-233-7.fc26.x86_64                        10/30 
  Running scriptlet: systemd-233-7.fc26.x86_64                            11/30 
  Upgrading        : systemd-233-7.fc26.x86_64                            11/30 
  Running scriptlet: systemd-233-7.fc26.x86_64                            11/30 
  Upgrading        : curl-7.53.1-12.fc26.x86_64                           12/30 
  Upgrading        : python3-hawkey-0.11.1-1.fc26.x86_64                  13/30 
  Upgrading        : gnutls-3.5.16-1.fc26.x86_64                          14/30 
  Running scriptlet: gnutls-3.5.16-1.fc26.x86_64                          14/30 
  Upgrading        : popt-1.16-12.fc26.x86_64                             15/30 
  Running scriptlet: popt-1.16-12.fc26.x86_64                             15/30 
  Cleanup          : systemd-pam-233-6.fc26.x86_64                        16/30 
  Running scriptlet: systemd-233-6.fc26.x86_64                            17/30 
  Cleanup          : systemd-233-6.fc26.x86_64                            17/30 
  Running scriptlet: systemd-233-6.fc26.x86_64                            17/30Failed to connect to bus: No such file or directory

  Cleanup          : python3-hawkey-0.11.0-1.fc26.x86_64                  18/30 
  Cleanup          : libdnf-0.11.0-1.fc26.x86_64                          19/30 
  Running scriptlet: libdnf-0.11.0-1.fc26.x86_64                          19/30 
  Cleanup          : curl-7.53.1-11.fc26.x86_64                           20/30 
  Cleanup          : libcurl-7.53.1-11.fc26.x86_64                        21/30 
  Running scriptlet: libcurl-7.53.1-11.fc26.x86_64                        21/30 
  Cleanup          : libsolv-0.6.29-2.fc26.x86_64                         22/30 
  Running scriptlet: libsolv-0.6.29-2.fc26.x86_64                         22/30 
  Cleanup          : audit-libs-2.7.8-1.fc26.x86_64                       23/30 
  Running scriptlet: audit-libs-2.7.8-1.fc26.x86_64                       23/30 
  Cleanup          : libcrypt-nss-2.25-10.fc26.x86_64                     24/30 
  Running scriptlet: libcrypt-nss-2.25-10.fc26.x86_64                     24/30 
  Cleanup          : systemd-libs-233-6.fc26.x86_64                       25/30 
  Running scriptlet: systemd-libs-233-6.fc26.x86_64                       25/30 
  Cleanup          : popt-1.16-8.fc26.x86_64                              26/30 
  Running scriptlet: popt-1.16-8.fc26.x86_64                              26/30 
  Cleanup          : gnutls-3.5.15-1.fc26.x86_64                          27/30 
  Running scriptlet: gnutls-3.5.15-1.fc26.x86_64                          27/30 
  Cleanup          : glibc-common-2.25-10.fc26.x86_64                     28/30 
  Cleanup          : glibc-langpack-en-2.25-10.fc26.x86_64                29/30 
  Cleanup          : glibc-2.25-10.fc26.x86_64                            30/30 
  Running scriptlet: glibc-2.25-10.fc26.x86_64                            30/30 
  Running scriptlet: systemd-233-7.fc26.x86_64                            30/30Failed to connect to bus: No such file or directory
BDB1539 Build signature doesn't match environment

  Verifying        : audit-libs-2.8.1-1.fc26.x86_64                        1/30 
  Verifying        : curl-7.53.1-12.fc26.x86_64                            2/30 
  Verifying        : libcurl-7.53.1-12.fc26.x86_64                         3/30 
  Verifying        : glibc-2.25-12.fc26.x86_64                             4/30 
  Verifying        : glibc-common-2.25-12.fc26.x86_64                      5/30 
  Verifying        : glibc-langpack-en-2.25-12.fc26.x86_64                 6/30 
  Verifying        : libcrypt-nss-2.25-12.fc26.x86_64                      7/30 
  Verifying        : gnutls-3.5.16-1.fc26.x86_64                           8/30 
  Verifying        : libdnf-0.11.1-1.fc26.x86_64                           9/30 
  Verifying        : python3-hawkey-0.11.1-1.fc26.x86_64                  10/30 
  Verifying        : libsolv-0.6.30-1.fc26.x86_64                         11/30 
  Verifying        : popt-1.16-12.fc26.x86_64                             12/30 
  Verifying        : systemd-233-7.fc26.x86_64                            13/30 
  Verifying        : systemd-pam-233-7.fc26.x86_64                        14/30 
  Verifying        : systemd-libs-233-7.fc26.x86_64                       15/30 
  Verifying        : libcrypt-nss-2.25-10.fc26.x86_64                     16/30 
  Verifying        : libcurl-7.53.1-11.fc26.x86_64                        17/30 
  Verifying        : popt-1.16-8.fc26.x86_64                              18/30 
  Verifying        : audit-libs-2.7.8-1.fc26.x86_64                       19/30 
  Verifying        : libdnf-0.11.0-1.fc26.x86_64                          20/30 
  Verifying        : systemd-pam-233-6.fc26.x86_64                        21/30 
  Verifying        : python3-hawkey-0.11.0-1.fc26.x86_64                  22/30 
  Verifying        : glibc-2.25-10.fc26.x86_64                            23/30 
  Verifying        : glibc-common-2.25-10.fc26.x86_64                     24/30 
  Verifying        : glibc-langpack-en-2.25-10.fc26.x86_64                25/30 
  Verifying        : systemd-libs-233-6.fc26.x86_64                       26/30 
  Verifying        : gnutls-3.5.15-1.fc26.x86_64                          27/30 
  Verifying        : curl-7.53.1-11.fc26.x86_64                           28/30 
  Verifying        : libsolv-0.6.29-2.fc26.x86_64                         29/30 
  Verifying        : systemd-233-6.fc26.x86_64                            30/30 

Upgraded:
  audit-libs.x86_64 2.8.1-1.fc26          curl.x86_64 7.53.1-12.fc26           
  glibc.x86_64 2.25-12.fc26               glibc-common.x86_64 2.25-12.fc26     
  glibc-langpack-en.x86_64 2.25-12.fc26   gnutls.x86_64 3.5.16-1.fc26          
  libcrypt-nss.x86_64 2.25-12.fc26        libcurl.x86_64 7.53.1-12.fc26        
  libdnf.x86_64 0.11.1-1.fc26             libsolv.x86_64 0.6.30-1.fc26         
  popt.x86_64 1.16-12.fc26                python3-hawkey.x86_64 0.11.1-1.fc26  
  systemd.x86_64 233-7.fc26               systemd-libs.x86_64 233-7.fc26       
  systemd-pam.x86_64 233-7.fc26          

Complete!

The issue is also still occurring on CentOS Ci, which gives a freshly provisioned machine for each test run (i.e., it's pulling the freshest available container on each run).

Metadata Update from @bowlofeggs:
- Issue status updated to: Open (was: Closed)

6 years ago

I believe that fixing this will alleviate the issues people are reporting in https://bugzilla.redhat.com/show_bug.cgi?id=1483553.

I am attempting to work around this by updating libsolv first, as suggested in the above BZ:

https://github.com/fedora-infra/bodhi/pull/1980

released a new image yesterday, please let me know if it worked

Metadata Update from @maxamillion:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

6 years ago

Login to comment on this ticket.

Metadata