#32 Re-deploy ci-cc-rdu01
Opened 2 years ago by pingou. Modified 2 years ago

This host was created ~2 years ago at a time we needed things to be set up and running quickly.
So I did that, set it up quickly and badly.
Now it's time to pay back this technical debt and re-deploy it properly with ansible and all.

In addition, this host is still running F25, so we need to upgrade it anyway


I've started on this yesterday, since then:
- the host is now running F28.
- resultsdb is running on it
- the fedmsg consumer is uploading messages to the database
- rdbsync has been rebuilt for f28 and installed

Blocked on:
- the ssl cert for https seems to have been "lost", should we use certbot? (@bstinson do you know/remember how we did it back then? Would you still have the ssl cert for this host?)
- While I saved: /var/run/fedmsg/status/fedmsg-hub/PipelineConsumer on restart fedmsg-hub doesn't seem to check this file and ask datagrepper for the messages it missed :(

Ok, I unblocked the processing of the missed messages, since on this host I'm not using the base fedmsg role, I had to define for myself the datagrepper_url and the status_directory.
Adding this to ansible so things are in sync

Alright, it's now pulling properly from datagrepper but I ran into: Received invalid message RuntimeWarning('Failed to authn message.',)

Has anyone seen this before?

More clearly it says the signature is invalid when pulling from datagrepper:

[fedmsg.crypto.x509_ng   ERROR] message [{u'username': None, u'i': 1, u'timestamp': 1540913039, u'msg_id': u'2018-c248cadc-14b8-46ae-8384-70c178879fa6', u'crypto': u'x509', u'topic': u'org.centos.prod.ci.pipeline.package.ignore', u'msg': {u'CI_TYPE': u'custom', u'build_id': u'107138', u'repo': u'python-astropy-healpix', u'namespace': u'rpms', u'message-content': u'', u'build_url': u'https://jenkins-continuous-infra.apps.ci.centos.org/job/ci-pipeline-trigger/107138/', u'rev': u'0cf7a6fe164661bf52e9b50039217174489a6f43', u'CI_NAME': u'ci-pipeline-trigger', u'username': u'fedora-atomic', u'topic': u'org.centos.prod.ci.pipeline.package.ignore', u'status': u'SUCCESS', u'branch': u'f29', u'test_guidance': u"''", u'ref': u'fedora/f29/x86_64/atomic-host'}}] has an invalid signature:

Looking at rdbsync, it looks like I need an API token to push things to Fedora's resultsdb
--> Looking for the person who can grant me one

Looking at rdbsync, it looks like I need an API token to push things to Fedora's resultsdb

Turns out that service runs on fedora's resultsdb so outside of needing a restart, it wasn't affected by the reinstall.

I restarted the service and it seems to be running fine now :)

With the patch from @bstinson which is in 0.4.0 rdbsync seems to be syncing fine the results again.

Thanks @bstinson.

I believe the last remaining issue is the failure to download/retrieve the missed messages from datagrepper.

Storing this here so it doesn't get lost, the content of /var/run/fedmsg/status/fedmsg-hub/PipelineConsumer was when I rebuilt the box:

{"message":{"body":{"certificate":"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVPakNDQTZPZ0F3SUJBZ0lDQW5Fd0RRWUpL\nb1pJaHZjTkFRRUZCUUF3Z2FBeEN6QUpCZ05WQkFZVEFsVlQKTVFzd0NRWURWUVFJRXdKT1F6RVFN\nQTRHQTFVRUJ4TUhVbUZzWldsbmFERVhNQlVHQTFVRUNoTU9SbVZrYjNKaApJRkJ5YjJwbFkzUXhE\nekFOQmdOVkJBc1RCbVpsWkcxelp6RVBNQTBHQTFVRUF4TUdabVZrYlhObk1ROHdEUVlEClZRUXBF\nd1ptWldSdGMyY3hKakFrQmdrcWhraUc5dzBCQ1FFV0YyRmtiV2x1UUdabFpHOXlZWEJ5YjJwbFkz\nUXUKYjNKbk1CNFhEVEUzTURVeE1ERTBNamMwT0ZvWERUSTNNRFV3T0RFME1qYzBPRm93Z2NneEN6\nQUpCZ05WQkFZVApBbFZUTVFzd0NRWURWUVFJRXdKT1F6RVFNQTRHQTFVRUJ4TUhVbUZzWldsbmFE\nRVhNQlVHQTFVRUNoTU9SbVZrCmIzSmhJRkJ5YjJwbFkzUXhEekFOQmdOVkJBc1RCbVpsWkcxelp6\nRWpNQ0VHQTFVRUF4TWFabVZrYlhObkxYSmwKYkdGNUxtTnBMbU5sYm5SdmN5NXZjbWN4SXpBaEJn\nTlZCQ2tUR21abFpHMXpaeTF5Wld4aGVTNWphUzVqWlc1MApiM011YjNKbk1TWXdKQVlKS29aSWh2\nY05BUWtCRmhkaFpHMXBia0JtWldSdmNtRndjbTlxWldOMExtOXlaekNCCm56QU5CZ2txaGtpRzl3\nMEJBUUVGQUFPQmpRQXdnWWtDZ1lFQTJzYUJuSjNyTlhYQXV3Skt2UkJyQnJTYUdMWHgKYXg4VGhu\nZ0wxV2hCYS8wSFZVdVAxWEhWUEVweUh6YXZZK0dsRzFVclVUMkFMQzFuRk5nVUNpSjhWWWVoZElw\nWApzQzNiOHFnUmltekt0aHUxM2hqQ01kSTYzV3h1S3FBQk5UQTRkZWtBK1c2cE9EdVdIMEI1b0tq\nVjFmWkZRN2xFCjUzZlQybElBZWg4ZndZY0NBd0VBQWFPQ0FWY3dnZ0ZUTUFrR0ExVWRFd1FDTUFB\nd0xRWUpZSVpJQVliNFFnRU4KQkNBV0hrVmhjM2t0VWxOQklFZGxibVZ5WVhSbFpDQkRaWEowYVda\ncFkyRjBaVEFkQmdOVkhRNEVGZ1FVUytnVApwNmg2ZXpJZW5RK0lLUERnWmZWZHQ5a3dnZFVHQTFV\nZEl3U0J6VENCeW9BVWEwQmErUklJaVZubldlVUY5UUlkCkNrNS9GQUNoZ2Fha2dhTXdnYUF4Q3pB\nSkJnTlZCQVlUQWxWVE1Rc3dDUVlEVlFRSUV3Sk9RekVRTUE0R0ExVUUKQnhNSFVtRnNaV2xuYURF\nWE1CVUdBMVVFQ2hNT1JtVmtiM0poSUZCeWIycGxZM1F4RHpBTkJnTlZCQXNUQm1abApaRzF6WnpF\nUE1BMEdBMVVFQXhNR1ptVmtiWE5uTVE4d0RRWURWUVFwRXdabVpXUnRjMmN4SmpBa0Jna3Foa2lH\nCjl3MEJDUUVXRjJGa2JXbHVRR1psWkc5eVlYQnliMnBsWTNRdWIzSm5nZ2tBNDFBZVIwOFhIa1V3\nRXdZRFZSMGwKQkF3d0NnWUlLd1lCQlFVSEF3SXdDd1lEVlIwUEJBUURBZ2VBTUEwR0NTcUdTSWIz\nRFFFQkJRVUFBNEdCQUF5cApCUk43VXFaUU1vcUw3UkFnS09hMzFSVTh3R3lWaEJhd1NvZm1Qd1dT\nMUdEbVA1OU9FbElaRldrVisrTi92VXBSCmFjalFyTStoUEVEYXRaUVU5cEtiV3FmVy92WVVyaGpE\nYTNYV3dxeW1kT2hjWTFhWUR3aVE5NGlWekNGUkdFM2kKMXNkN2tuc2VjL2x4Z2NldmhYS2ZleTNK\nN241cXludFBYVGpVMjdGMQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==\n","crypto":"x509","i":1,"msg":{"CI_NAME":"ci-pipeline-trigger","CI_TYPE":"custom","branch":"rawhide","build_id":"107098","build_url":"https://jenkins-continuous-infra.apps.ci.centos.org/job/ci-pipeline-trigger/107098/","message-content":"","namespace":"rpms","ref":"fedora/rawhide/x86_64/atomic-host","repo":"elementary-greeter","rev":"0d32016fff3cdc618d787a4a6a28eed936fa8d39","status":"SUCCESS","test_guidance":"''","topic":"org.centos.prod.ci.pipeline.package.ignore","username":"fedora-atomic"},"msg_id":"2018-1b3fd906-4df6-458e-ae47-f6e4aafa395b","signature":"rPiAPtP5qhcwLPGNgHxT4vA/YahSRPVEeqOGXyuIXkOgw8096GM/CN8o8r+hIaUHcl7EsCfxWsDl\nSFQZ5daj3fSKPNHLt6r05DEPl5+k4/1JECAh2etqGd9CMiw1buNh+oF6HqYihsZcatPlUH51AFFV\nVYx/AKoDLpACO35xZAg=\n","timestamp":1540908164,"topic":"org.centos.prod.ci.pipeline.package.ignore"},"topic":"org.centos.prod.ci.pipeline.package.ignore"},"status":"post"}

This is complete, yes?

Metadata Update from @smooge:
- Issue status updated to: Closed (was: Open)

2 years ago

Nope, the https piece still needs fixing: https://resultsdb.ci.centos.org/resultsdb/results It's more cosmetic than required but I still would like to fix it.

Metadata Update from @pingou:
- Issue status updated to: Open (was: Closed)

2 years ago

Sorry.. I should not have closed last night. What can I do to fix that?

What can I do to fix that?

Let's schedule some time for both of us and go through the remaining :)

Metadata Update from @kevin:
- Issue set to the milestone: FY19 Q4 (was: FY19 Q3)

2 years ago

Login to comment on this ticket.

Metadata