fedora-infrastructure

#6158 Fix Bodhi's staging-sync script
Closed: Fixed Opened by bowlofeggs.

Closed: Fixed
Reopen Issue

I noticed today that Bodhi's staging sync script is incomplete. It wasn't dropping everything in the database, which was resulting in some errors when importing the production data. As it currently stands, it hits the following error when run:

[root@bodhi-backend01 ~][STG]# psql -h pgbdr.stg.phx2.fedoraproject.org bodhi2 bodhi2 -e --set ON_ERROR_STOP=on --set VERBOSITY=verbose < drop_bodhi_db.sql 
Password for user bodhi2: 
BEGIN;
BEGIN
SET LOCAL bdr.permit_ddl_locking = on;
SET
DELETE FROM alembic_version;
DELETE 1
DROP TABLE IF EXISTS comments CASCADE;
NOTICE:  00000: table "comments" does not exist, skipping
LOCATION:  DropErrorMsgNonExistent, tablecmds.c:736
DROP TABLE
DROP TABLE IF EXISTS groups CASCADE;
NOTICE:  00000: table "groups" does not exist, skipping
LOCATION:  DropErrorMsgNonExistent, tablecmds.c:736
DROP TABLE
DROP TABLE IF EXISTS cves CASCADE;
NOTICE:  00000: table "cves" does not exist, skipping
LOCATION:  DropErrorMsgNonExistent, tablecmds.c:736
DROP TABLE
DROP TABLE IF EXISTS packages CASCADE;
NOTICE:  00000: table "packages" does not exist, skipping
LOCATION:  DropErrorMsgNonExistent, tablecmds.c:736
DROP TABLE
DROP TABLE IF EXISTS stacks CASCADE;
NOTICE:  00000: table "stacks" does not exist, skipping
LOCATION:  DropErrorMsgNonExistent, tablecmds.c:736
DROP TABLE
DROP TABLE IF EXISTS stack_user_table CASCADE;
NOTICE:  00000: table "stack_user_table" does not exist, skipping
LOCATION:  DropErrorMsgNonExistent, tablecmds.c:736
DROP TABLE
DROP TABLE IF EXISTS update_cve_table CASCADE;
NOTICE:  00000: table "update_cve_table" does not exist, skipping
LOCATION:  DropErrorMsgNonExistent, tablecmds.c:736
DROP TABLE
DROP TABLE IF EXISTS releases CASCADE;
NOTICE:  00000: table "releases" does not exist, skipping
LOCATION:  DropErrorMsgNonExistent, tablecmds.c:736
DROP TABLE
DROP TABLE IF EXISTS comment_bug_assoc CASCADE;
NOTICE:  00000: table "comment_bug_assoc" does not exist, skipping
LOCATION:  DropErrorMsgNonExistent, tablecmds.c:736
DROP TABLE
DROP TABLE IF EXISTS comment_testcase_assoc CASCADE;
NOTICE:  00000: table "comment_testcase_assoc" does not exist, skipping
LOCATION:  DropErrorMsgNonExistent, tablecmds.c:736
DROP TABLE
DROP TABLE IF EXISTS stack_group_table CASCADE;
NOTICE:  00000: table "stack_group_table" does not exist, skipping
LOCATION:  DropErrorMsgNonExistent, tablecmds.c:736
DROP TABLE
DROP TABLE IF EXISTS testcases CASCADE;
NOTICE:  00000: table "testcases" does not exist, skipping
LOCATION:  DropErrorMsgNonExistent, tablecmds.c:736
DROP TABLE
DROP TABLE IF EXISTS buildroot_overrides CASCADE;
ERROR:  XX000: cache lookup failed for relation 7418164
LOCATION:  getRelationDescription, objectaddress.c:2539

I am unsure how to identify which relation that is. Very likely we need to adjust the order of the things that are dropped, or include more things that need to be dropped.

I think the easiest (and most reliable) fix here is to just update the drop script to first drop the database on both pgbdr01 and pgbdr02, and then recreate the dbs and permissions and set BDR back up.
Those commands are pretty trivial.

So basically:

  1. Log into both db servers and remove BDR from the DB (select bdr.remove_bdr_from_local_node(true, true);).
  2. Log into both and use dropdb.
  3. Log into both and createdb.
  4. On 01 import the db dump (no sed statements needed anymore).
  5. On 01, SELECT bdr.bdr_group_create(local_node_name := 'pgbdr01', node_external_dsn := 'host=pgbdr01 dbname=bodhi2 password={{ password }}');
  6. On both db servers, create extension btree_gist; create extension bdr;
  7. On 02 run SELECT bdr.bdr_group_join( local_node_name := 'pgbdr02', node_external_dsn := 'host=pgbdr02 dbname=bodhi2 password={{ password }}', join_using_dsn := 'host=pgbdr01 dbname=bodhi2 password={{ password }}');

Mostly. A few nits:

3.5 add bodhi user to 'own' the bodhi db
3.75 create bodhi db before importing

Otherwise looks pretty right.

So, I know you have spent a ton of time on bdr issues... and in the hackfest we decided we would only move things when application writers were ready, so would you prefer to switch bodhi back to non bdr in staging for now to avoid this hassle? Or do you think it's worthwhile to get it to the point where it would be happy in a BDR world?

@kevin excellent, thanks for the notes!

Patrick recent sent me a patch to give Bodhi the pks it needs:

https://github.com/fedora-infra/bodhi/pull/1689

It still needs a little work (I'll get to it soon), but I think it'll probably land in time for 2.10. Fixing that and this would get us all the "known" issues solved, but the unknown issues remain: does Bodhi use queries safely? Does that distributed transaction lock do what we think it does? However, those unknowns are questions that I think need to be answered before we put BDR into production (not staging) for Bodhi so I think we can get Bodhi back on BDR in staging by fixing this issue and getting that PR merged. Does that sound good to you too?
Edited by bowlofeggs

Sounds fine to me... Just wanted to give the option to turn off BDR if it was holding things up too much right now.

I guess we can close this now. Reopen if there's anything left to do on our side.

:volcano:

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
Metadata
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.