This host was created ~2 years ago at a time we needed things to be set up and running quickly. So I did that, set it up quickly and badly. Now it's time to pay back this technical debt and re-deploy it properly with ansible and all.
In addition, this host is still running F25, so we need to upgrade it anyway
I've started on this yesterday, since then: - the host is now running F28. - resultsdb is running on it - the fedmsg consumer is uploading messages to the database - rdbsync has been rebuilt for f28 and installed
Blocked on: - the ssl cert for https seems to have been "lost", should we use certbot? (@bstinson do you know/remember how we did it back then? Would you still have the ssl cert for this host?) - While I saved: /var/run/fedmsg/status/fedmsg-hub/PipelineConsumer on restart fedmsg-hub doesn't seem to check this file and ask datagrepper for the messages it missed :(
/var/run/fedmsg/status/fedmsg-hub/PipelineConsumer
Ok, I unblocked the processing of the missed messages, since on this host I'm not using the base fedmsg role, I had to define for myself the datagrepper_url and the status_directory. Adding this to ansible so things are in sync
datagrepper_url
status_directory
Alright, it's now pulling properly from datagrepper but I ran into: Received invalid message RuntimeWarning('Failed to authn message.',)
Received invalid message RuntimeWarning('Failed to authn message.',)
Has anyone seen this before?
More clearly it says the signature is invalid when pulling from datagrepper:
[fedmsg.crypto.x509_ng ERROR] message [{u'username': None, u'i': 1, u'timestamp': 1540913039, u'msg_id': u'2018-c248cadc-14b8-46ae-8384-70c178879fa6', u'crypto': u'x509', u'topic': u'org.centos.prod.ci.pipeline.package.ignore', u'msg': {u'CI_TYPE': u'custom', u'build_id': u'107138', u'repo': u'python-astropy-healpix', u'namespace': u'rpms', u'message-content': u'', u'build_url': u'https://jenkins-continuous-infra.apps.ci.centos.org/job/ci-pipeline-trigger/107138/', u'rev': u'0cf7a6fe164661bf52e9b50039217174489a6f43', u'CI_NAME': u'ci-pipeline-trigger', u'username': u'fedora-atomic', u'topic': u'org.centos.prod.ci.pipeline.package.ignore', u'status': u'SUCCESS', u'branch': u'f29', u'test_guidance': u"''", u'ref': u'fedora/f29/x86_64/atomic-host'}}] has an invalid signature:
Looking at rdbsync, it looks like I need an API token to push things to Fedora's resultsdb --> Looking for the person who can grant me one
Looking at rdbsync, it looks like I need an API token to push things to Fedora's resultsdb
Turns out that service runs on fedora's resultsdb so outside of needing a restart, it wasn't affected by the reinstall.
I restarted the service and it seems to be running fine now :)
With the patch from @bstinson which is in 0.4.0 rdbsync seems to be syncing fine the results again.
Thanks @bstinson.
I believe the last remaining issue is the failure to download/retrieve the missed messages from datagrepper.
Storing this here so it doesn't get lost, the content of /var/run/fedmsg/status/fedmsg-hub/PipelineConsumer was when I rebuilt the box:
{"message":{"body":{"certificate":"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVPakNDQTZPZ0F3SUJBZ0lDQW5Fd0RRWUpL\nb1pJaHZjTkFRRUZCUUF3Z2FBeEN6QUpCZ05WQkFZVEFsVlQKTVFzd0NRWURWUVFJRXdKT1F6RVFN\nQTRHQTFVRUJ4TUhVbUZzWldsbmFERVhNQlVHQTFVRUNoTU9SbVZrYjNKaApJRkJ5YjJwbFkzUXhE\nekFOQmdOVkJBc1RCbVpsWkcxelp6RVBNQTBHQTFVRUF4TUdabVZrYlhObk1ROHdEUVlEClZRUXBF\nd1ptWldSdGMyY3hKakFrQmdrcWhraUc5dzBCQ1FFV0YyRmtiV2x1UUdabFpHOXlZWEJ5YjJwbFkz\nUXUKYjNKbk1CNFhEVEUzTURVeE1ERTBNamMwT0ZvWERUSTNNRFV3T0RFME1qYzBPRm93Z2NneEN6\nQUpCZ05WQkFZVApBbFZUTVFzd0NRWURWUVFJRXdKT1F6RVFNQTRHQTFVRUJ4TUhVbUZzWldsbmFE\nRVhNQlVHQTFVRUNoTU9SbVZrCmIzSmhJRkJ5YjJwbFkzUXhEekFOQmdOVkJBc1RCbVpsWkcxelp6\nRWpNQ0VHQTFVRUF4TWFabVZrYlhObkxYSmwKYkdGNUxtTnBMbU5sYm5SdmN5NXZjbWN4SXpBaEJn\nTlZCQ2tUR21abFpHMXpaeTF5Wld4aGVTNWphUzVqWlc1MApiM011YjNKbk1TWXdKQVlKS29aSWh2\nY05BUWtCRmhkaFpHMXBia0JtWldSdmNtRndjbTlxWldOMExtOXlaekNCCm56QU5CZ2txaGtpRzl3\nMEJBUUVGQUFPQmpRQXdnWWtDZ1lFQTJzYUJuSjNyTlhYQXV3Skt2UkJyQnJTYUdMWHgKYXg4VGhu\nZ0wxV2hCYS8wSFZVdVAxWEhWUEVweUh6YXZZK0dsRzFVclVUMkFMQzFuRk5nVUNpSjhWWWVoZElw\nWApzQzNiOHFnUmltekt0aHUxM2hqQ01kSTYzV3h1S3FBQk5UQTRkZWtBK1c2cE9EdVdIMEI1b0tq\nVjFmWkZRN2xFCjUzZlQybElBZWg4ZndZY0NBd0VBQWFPQ0FWY3dnZ0ZUTUFrR0ExVWRFd1FDTUFB\nd0xRWUpZSVpJQVliNFFnRU4KQkNBV0hrVmhjM2t0VWxOQklFZGxibVZ5WVhSbFpDQkRaWEowYVda\ncFkyRjBaVEFkQmdOVkhRNEVGZ1FVUytnVApwNmg2ZXpJZW5RK0lLUERnWmZWZHQ5a3dnZFVHQTFV\nZEl3U0J6VENCeW9BVWEwQmErUklJaVZubldlVUY5UUlkCkNrNS9GQUNoZ2Fha2dhTXdnYUF4Q3pB\nSkJnTlZCQVlUQWxWVE1Rc3dDUVlEVlFRSUV3Sk9RekVRTUE0R0ExVUUKQnhNSFVtRnNaV2xuYURF\nWE1CVUdBMVVFQ2hNT1JtVmtiM0poSUZCeWIycGxZM1F4RHpBTkJnTlZCQXNUQm1abApaRzF6WnpF\nUE1BMEdBMVVFQXhNR1ptVmtiWE5uTVE4d0RRWURWUVFwRXdabVpXUnRjMmN4SmpBa0Jna3Foa2lH\nCjl3MEJDUUVXRjJGa2JXbHVRR1psWkc5eVlYQnliMnBsWTNRdWIzSm5nZ2tBNDFBZVIwOFhIa1V3\nRXdZRFZSMGwKQkF3d0NnWUlLd1lCQlFVSEF3SXdDd1lEVlIwUEJBUURBZ2VBTUEwR0NTcUdTSWIz\nRFFFQkJRVUFBNEdCQUF5cApCUk43VXFaUU1vcUw3UkFnS09hMzFSVTh3R3lWaEJhd1NvZm1Qd1dT\nMUdEbVA1OU9FbElaRldrVisrTi92VXBSCmFjalFyTStoUEVEYXRaUVU5cEtiV3FmVy92WVVyaGpE\nYTNYV3dxeW1kT2hjWTFhWUR3aVE5NGlWekNGUkdFM2kKMXNkN2tuc2VjL2x4Z2NldmhYS2ZleTNK\nN241cXludFBYVGpVMjdGMQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==\n","crypto":"x509","i":1,"msg":{"CI_NAME":"ci-pipeline-trigger","CI_TYPE":"custom","branch":"rawhide","build_id":"107098","build_url":"https://jenkins-continuous-infra.apps.ci.centos.org/job/ci-pipeline-trigger/107098/","message-content":"","namespace":"rpms","ref":"fedora/rawhide/x86_64/atomic-host","repo":"elementary-greeter","rev":"0d32016fff3cdc618d787a4a6a28eed936fa8d39","status":"SUCCESS","test_guidance":"''","topic":"org.centos.prod.ci.pipeline.package.ignore","username":"fedora-atomic"},"msg_id":"2018-1b3fd906-4df6-458e-ae47-f6e4aafa395b","signature":"rPiAPtP5qhcwLPGNgHxT4vA/YahSRPVEeqOGXyuIXkOgw8096GM/CN8o8r+hIaUHcl7EsCfxWsDl\nSFQZ5daj3fSKPNHLt6r05DEPl5+k4/1JECAh2etqGd9CMiw1buNh+oF6HqYihsZcatPlUH51AFFV\nVYx/AKoDLpACO35xZAg=\n","timestamp":1540908164,"topic":"org.centos.prod.ci.pipeline.package.ignore"},"topic":"org.centos.prod.ci.pipeline.package.ignore"},"status":"post"}
This is complete, yes?
Metadata Update from @smooge: - Issue status updated to: Closed (was: Open)
Nope, the https piece still needs fixing: https://resultsdb.ci.centos.org/resultsdb/results It's more cosmetic than required but I still would like to fix it.
Metadata Update from @pingou: - Issue status updated to: Open (was: Closed)
Sorry.. I should not have closed last night. What can I do to fix that?
What can I do to fix that?
Let's schedule some time for both of us and go through the remaining :)
Metadata Update from @kevin: - Issue set to the milestone: FY19 Q4 (was: FY19 Q3)
Login to comment on this ticket.