I am kind of curious on how hard/easy is it to deploy solr in a container/openshift, do you know?

I do not think that it will be hard given that there is an official container and a distributed cloud mode that is well documented.

The only complicated part is the admin UI which looks like it supports authentication with kerberos and openid. It is strongly recommended to put that (and the api) behind a non-public network.

Is it possible to have one pod secure a write mount to a volume and have multiple read-only mounts for load-balancing? I've looked at the openshift docs and it seems impossible.

I not sure if thats possible, RWX (ReadWriteMany) access mode is likely needed, which we can provide with NFS backed storage.

• [0] https://docs.openshift.com/container-platform/3.6/architecture/additional_concepts/storage.html#pv-access-modes

Looking at examples roles and playbook at:

• https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/openshift-apps

and

• https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps

Do you think you could open a PR against the ansible repo with what is needed for this project?

I not sure if thats possible, RWX (ReadWriteMany) access mode is likely needed, which we can provide with NFS backed storage.

• [0] https://docs.openshift.com/container-platform/3.6/architecture/additional_concepts/storage.html#pv-access-modes

Thanks, I mostly just want this to have one pod control writing and have the rest just serve from it instead of having different copies per-instance. If this is a bad idea/too complicated, I can just keep it how it is now.

Do you think you could open a PR against the ansible repo with what is needed for this project?

Yes, I'll make a PR as soon as I have the time.

Commit a4c1b1448f421c489db42035479626fa58b3b661 (Add fedora-packages-static and solr to openshift) is in fedora-infra/ansible since Mon Apr 5 18:25:10 2021 +0000. What is the state of the deployment? I couldn't find it on packages{{ env_suffix }}.fedoraproject.org with .stg or .test as suffix, but there are many moving parts here and I might have missed something (or it might not be online yet?).

@fmux it was merged by me, but I didn't run the playbook on it... we have been sorting out permissions so @mymindstorm can run the playbook and do both the initial deployment and any iterations after.

So, it should be up soon I hope!

It is not online yet. As Kevin has mentioned there were some access issues and I made a mistake in the solr playbook (which packages-static playbook won't run without). PR soon.

@mymindstorm is there anything still blocking on getting this one done?

There is some sort of networking issue with my openshift builds: https://pagure.io/fedora-infra/ansible/pull-request/574#comment-149131

Other than that, I should have a testing version up soon. I will send out an email to the infrastructure list when it is ready.

https://packages.stg.fedoraproject.org/ is alive. Will put an email on the list tomorrow.

So I'm a bit confused on where exactly in the RFR process this is. What exactly are the next steps here? Is it appropriate to send an email out to devel for feedback at this point?

cc @puiterwijk for security audit. I have updated the ticket description with a self-audit.

So, our RFR process never really got updated for openshift applications, which I think should have a easier and simpler path than apps on vm's.

I think it would be fine to ask for feedback on devel now before rolling to production. :)

It would be nice/good to get at least a few more people familiar with the app and how it's setup in case you are unable to deal with it for whatever reason.

As a related, but seperate thing... what do we do if we want to try a solr for docs search? I guess that would be a seperate solr instance/app/project that indexes docs?

As a related, but seperate thing... what do we do if we want to try a solr for docs search? I guess that would be a seperate solr instance/app/project that indexes docs?

Yes, it's for the best to have a separate instance. Openshift does not play very nice with Solr due to it wanting to be configured after it is actually running. Additionally, we would probably want it running in cluster mode with a Zookeeper for docs. I have yet to read the source code of the prototype and figure out what sort of situation the networking will be in.

It would be nice/good to get at least a few more people familiar with the app and how it's setup in case you are unable to deal with it for whatever reason.

I can act as co-maintainer / backup for this - I'm interested and familiar with the project since I wrote the initial POC (although I still have to take a closer look to solr).

So, there was a little bit of feedback from devel... where do we stand? Should we roll this out to prod soon?

So, there was a little bit of feedback from devel... where do we stand? Should we roll this out to prod soon?

The only big change that still needs to be made are the solr changes. I am out of town currently and can't work on this at the moment. I will have an update next weekend.

Sounds great. Safe travels!

Sorry about the delays here. Currently, I am almost done with a new release that has feedback from the list incorporated. It is primarily blocked by some bigger search changes that are almost done and a few bugs that need to be investigated. After that is done and it has been tested for a few days in staging, it should be ready for a prod deployment.

I now have a version that would be a good candidate for production at https://packages.stg.fedoraproject.org/.

Fixes for the missing icons in fedmsg:
- https://github.com/fedora-infra/fedmsg_meta_fedora_infrastructure/pull/506
- https://pagure.io/fedora-infra/ansible/pull-request/682

When we do move a version to production, is there anything that openshift provides in regards to log monitoring? What's the preferred way to do alerting on container logs in openshift?

We do have logs, but there's no monitoring. We do have nagios for alerting, but it has no access in particular to openshift logs. ;(

We have a iniative right now to get our clusters moved to openshift 4 and setup metrics for all applications, after that we will look at alerting and logs...

Shall we deploy this in prod? what all do you need to do that?

So I looked over the logs and found a serious data issue thanks to the rebuild activity. I've fixed it, but i would like to let it run until the 6th and run my audit script again just to be safe and verify.

If openshift 4 is coming soon, would it be better to wait instead of migrating at a later date? If that is not an issue, then all I need are the volume claims to be created to get it running.

After further debugging, it turns out that the underlying issue (aside from a smaller issue that was easy to fix) is some sort of deleted pod that is still running in openshift. It's replacing repo database files while other scripts are using them and causing crashes during page generation. I deleted some pods that stuck in the "terminating" state at some point, that is likely the root cause of the issue.

I'm still new to openshift, what is the best way to prevent this from happening? I can't see the pod from the console or CLI. Should the volumes be changed to ReadWriteOnce?

Weird. I have no idea why that would happen... so, you can't see the pod any, but you think it's running?

Pretty much. My evidence:

• After deleting the database directory and shutting off the pod for a few hours, the database directory would be populated with current data when there should have been no files present.
• I created a build that adds a control to disable database updates and html generation, and activated the control. The timestamps for the files in repositories/ on fedora-packages-static-db-storage-stg show that something is modifying the files, even though there should be nothing happening to that directory.
• I can't reproduce any sort of "malformed database" or "table not found" errors when running my local computer from preliminary tests over a few days.

The issues with openshift seem to be resolved for the time being. A @kevin helped me get a version running in prod at https://packages.fedoraproject.org/ (data will be fully populated in a few hours).