We have 2 memcached instances, one prod and one stg:
memcached01.iad2.fedoraproject.org 8.10 memcached01.stg.iad2.fedoraproject.org 8.10
We need to reinstall these as rhel9 instances, make sure they work, etc.
So, The best way is probibly to make new memcached02 version of the stg one and make sure everything installs and configures right, then redo memcached01.stg first, confirm that apps that use it work, then do production.
Metadata Update from @phsmoura: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: low-gain, low-trouble, ops
Metadata Update from @zlopez: - Issue untagged with: low-gain, low-trouble - Issue tagged with: high-trouble, medium-gain
Could you share the ip config ? it would help to add new entry in ansible inventory.
Yeah, we will need to allocate new ip's for memcached02.stg... I can try and do that later, or you or @phsmoura could make a quick dns repo patch? Just needs an unused ip in the staging forward/reverse zones...
I just pushed a PR changing memcached01.stg urls to RHEL9, but still need that memcached02.stg thing.. want to work on it @seddik ?
Metadata Update from @phsmoura: - Issue untagged with: high-trouble, medium-gain, ops
Metadata Update from @phsmoura: - Issue tagged with: high-trouble, medium-gain
Yes I can carry out a PR for that, but have no idea about the process to allocate new ip for new node.. maybe juste try to ping unused ip in the network segment ?? another question, how we can build this ?? Please share any doc that could help
In the dns repo (which you can checkout but not push to directly), look at: master/166.3.10.in-addr.arpa and master/stg.iad2.fedoraproject.org
Find a ip marked 'unused' in the first one and update it with memcached02.stg.iad2.fedoraproject.org and update the other one with the ip picked. You can ping to make sure nothing is accidentally listening there. :)
For install, if you add it to inventory/inventory and add a inventory/host_vars/memcached02.stg.iad2.fedoraproject.org then running the playbooks/groups/memcached.yml playbook will see the vm doesn't exist, then install it and configure it. :)
So, first a patch for dns (I can push it), and then a PR for the ansible side, then I can get you both setup to run the playbook and that should deploy it. then, work through any changes needed for moving from rhel8 to 9, and once the playbook runs fine, the next step would be to move things using memcached01 to 02 and make sure they are still working. Then we can retire memcached01 and do it all again (more quickly) on prod. ;)
Make sense? Happy to expand on anything...
Yes right, I'll work on PR for this
PR here : https://pagure.io/fork/seddik/fedora-infra/ansible/diff/main..memcached02_stg And dns patch
<img alt="0001-add-dns-entries-for-memcached02.patch" src="/fedora-infrastructure/issue/raw/files/7b3ec49ad772f9c0613eeff633525c679a8944f11809b6a07c9d90fc4bfebe2b-0001-add-dns-entries-for-memcached02.patch" />
@kevin hope this help
ok. Merged the pr and pushed the dns change.
I also added you to sysadmin-noc and setup sysadmin-noc having access to run the playbook.
So, either you or @phsmoura or both of you can run the playbook now and see what needs to be adjusted for rhel9 in the memcached role.
Ok thanks, I'll try to look into and get back to you if I need any info
OK memcached02.stg is up and runnig. We should run some tests rto make sure memcached works perfectly ??
Yeah, if you want to test against it locally to make sure it looks ok that would be great.
Otherwise I think the next step is to move stg apps over to use it and see if they work as expected.
That would seem to be:
roles/mailman3/templates/settings.py.j2: 'LOCATION': 'memcached01:11211', roles/mediawiki/templates/LocalSettings.php.fp.j2: 0 => 'memcached01:11211', roles/openshift-apps/badges/templates/tahrir.cfg.py: "url": "memcached01{{env_suffix}}:11211", roles/openshift-apps/elections/templates/fedora-elections.cfg: 'url': ["memcached01:11211"], roles/openshift-apps/koschei/templates/config-backend.cfg.j2: "url": "memcached01", roles/openshift-apps/koschei/templates/config-frontend.cfg.j2: "url": "memcached01:11211", roles/packages3/web/templates/packages-app.ini.j2:beaker.cache.url = memcached01:11211
but we need to make sure to only change staging for now, until we know things work. Then we can git rid of memcached01.stg and move on to do the same in prod. ;)
I push temporary branch memcached_apps to test moving app to memcached02 .
memcached_apps
We can try to run deploy from this branch on batcave01 ? right ?
Ok PR pushed for iad2 , and dns patch in attachment <img alt="0001-dns-entries-for-memcached02.patch" src="/fedora-infrastructure/issue/raw/files/f35aff787526bafc108b63a5707727be344d2192685e81dba5bd64c6a9e6c3fa-0001-dns-entries-for-memcached02.patch" />
@kevin when you get a chance can you take look ??
So, looking at this again, I think the only thing left to do here is retire the memcached01 / memcached01.stg instances?
Can you make a PR that removes them? I can merge that, run noc playbook, then remove the vm's...
ok. Those instances are retired. :)
Many thanks @seddik !
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.