Great! Thanks for filing this.

• The copr-fe database backup is in place on copr-be in /var/tmp/restore-backups/

On copr keygen there's an issue. The database backup is always called "copr_keygen_keyring.tar.gz.gpg" so our backup server only has the most recent version of it. New ones just overwrite the old one. ;(
Can you adjust the copr-keygen backup to use the YYYY-MM-DD in the name?

Thanks, I checked that copr-fe database is backed up correctly.

Can you adjust the copr-keygen backup to use the YYYY-MM-DD in the name?

Sure, fixed. I'm glad we found this problem. I'll give it a few days and
then re-request the restore then.

Btw., we encrypt the keygen backup by pub key for which we don't have
access to the corresponding private key. So I suppose we have to ask you
to decrypt it as well (which is ok). The question is,
should we do the same for the copr-frontend backups..? Or is it OK to keep
the backup unencrypted?

Yeah, I will need to locate the key/passphrase, but I think I do have it somewhere.

I don't think encryption adds much value on copr-frontend backup. Is there much sensitive data in there?

I'll go ahead and close this and you can file a new one for the next test? Or just re-open this one, whichever you prefer.

I don't think encryption adds much value on copr-frontend backup. Is there much sensitive data in there?

There are integration/webhook credentials, and users' API tokens. But I suppose only
the subset of administrator who can actually access copr-fe can read the backups,
right?

Yeah, sysadmin-copr and sysadmin-main... and both those groups could just login to the prod instance and get that information. They should be trustworthy. :)

I'm reopening to test the copr-keygen backup. Can we test 2020-05-20 backup, decrypt
it and upload to the same location as before?

Is this a required thing to fix something or just testing? We are severely time constrained to get systems installed in IAD2 and Kevin and I need to focus on that first and foremost.

If you need this to fix something we can do it and just put in a 20 hour day versus 14 hour day. If you need this to test things.. we would like you to do that after the move is complete.

Yes, this can wait, sure. This is just testing.

ok, next issue. :) I have the backups and they look fine...

but when trying to decrypt the keygen ones, I can't seem to find the private key. I have the passphrase and an older key apparently, but not the current one.

Personally, I don't think encrypting them is worth it here, but if you want to still thats fine. Would you like me to generate a new keypair? Or would you like to do so and keep it on your end?
Or should we just stop encrypting them?

Thanks for looking at this! I'm glad we debug those issues without any pressure :-)

Would you like me to generate a new keypair? Or would you like to do so and keep it on your end? Or should we just stop encrypting them?

We don't have other place to store any private key than {private}. So definitely
not on our side.

I don't want to overthink this and invent something "special".

How are the other signing GPG keys baked up? I mean e.g. for Fedora RPM signatures?
If they are just backed-up plain text - on the same backup host, and you are confident
the process is set-up securely, I think we can do the very same thing.

We don't have other place to store any private key than {private}. So definitely
not on our side.
I don't want to overthink this and invent something "special".

Yeah.

How are the other signing GPG keys baked up? I mean e.g. for Fedora RPM signatures?
If they are just backed-up plain text - on the same backup host, and you are confident
the process is set-up securely, I think we can do the very same thing.

Well, it's complicated with them because the way sigul works. In order to sign anything, the vault needs it's passphrase (entered at startup) and an admin with access to the key needs to enter their passphrase. So, really the keys are always encrypted... with 2 secrets.

Anyhow, for copr right now I think plain should be fine. It only goes over ssh to the backup host and few people have access there. But I am willing to make a new keypair and have you encrypt with that (and make sure the private key, etc is in our private repo so it's not lost again).

We may at some point revisit our backup setup and encrypt all backups, but right now the current process is acceptable. IMHO.

But I am willing to make a new keypair and have you encrypt with that

If I may be a bit lazy, I'd prefer this variant. It wouldn't force us to
change our backup scripts and deployment scripts :-) at least (if it
doesn't bring too much or any additional security).

I thought we use some well-known public key, but it seems to be a copr
specific one, meh:

src="{{ private }}/files/copr/keygen/backup_key.asc"

So it should be entirely OK to change it. OK, could you please replace this
one for us?

(and make sure the private key, etc is in our private repo so it's not
lost again).

It may be very well our fault, not yours. I wasn't working on copr those
days (so I can not tell), but it is entirely possible that our team gave
you only the public key...

Would you like me to generate a new keypair?

Gently ping on this :-)

Sorry for the delay. I had a tab with this open and almost got to it serveral times... before being pulled into something else. ;)

ok. I generated a new keypair.

The public key is the same ansible name:

"{{ private }}/files/copr/keygen/backup_key.asc"

I made sure to securely store the private key and passphrase so we can restore.

Shall we leave this open and check back next week that we can do a restore?

I've run the playbook to get the key on its place:

TASK [copr/keygen : copy pubkey for backup encryption] **************************************************************************************************************************************************************************************
Thursday 30 July 2020  06:30:34 +0000 (0:00:00.271)       0:03:10.424 ********* 
Thursday 30 July 2020  06:30:34 +0000 (0:00:00.271)       0:03:10.423 ********* 
changed: [copr-keygen.aws.fedoraproject.org]

TASK [copr/keygen : import pubkey for backup encryption] ************************************************************************************************************************************************************************************
Thursday 30 July 2020  06:30:35 +0000 (0:00:00.875)       0:03:11.299 ********* 
Thursday 30 July 2020  06:30:35 +0000 (0:00:00.875)       0:03:11.299 ********* 
changed: [copr-keygen.aws.fedoraproject.org]

Shall we leave this open and check back next week that we can do a restore?

Makes sense. Thank you.

@kevin, can you upload the decrypted keygen tarball from say 2020-08-05 onto Copr BE disk (as described in the initial issue description)?

Sorry for the delay here...

But... it looks like it's still using the old key?

gpg: encrypted with 4096-bit RSA key, ID D092D73D5617EE2F, created 2015-01-21
      "copr-sign-backup (This is used to encrypt copr sign backups.) <admin@fedoraproject.org>"

Did it properly import it? perhaps you need to delete the old one from the keyring?

Meh, obviously re-runing the playbook wasn't enough (some related log output about importing key was printed, so I thought it is OK)....

Ah, the key you generated did not have 'admin@fedoraproject.org' email. I
didn't ask you for that ... and that's why it didn't work .... so I
updated the backup script:

https://pagure.io/fedora-infra/ansible/c/5e38e064d0fc0d6ad4e2253fc1d4f27b6c99375c?branch=master

Now, any key (potentially) generated in future needs to be usable with
--recipient copr-keygen-backup-key instead.

Ok, I'm running a keygen playbook, let's wait two days more for the
updated tarball, please.

ok. Unencrypted restored version on copr-be. Please test and hopefully we can close this out. ;)

Thank you a lot for the patience! It looks good to me.