#10564 update firmware/passwords on the rest of our hardware
Closed: Fixed with Explanation 2 years ago by kevin. Opened 2 years ago by kevin.

Most/much of our hardware we upgraded eariler this week, but there are still iisolated things that we need to finish up.

  • buildhw-x86 01-16 (16 blades in the 2 dell fx bladecenters)
  • buildvmhost-s390x-01 and guests need reboting. I can do that.
  • bvmhost-p09 01-03 need rebooting, but I need to move them to Fedora 35 installs, so I can do that.
  • bvmhost-a64-osbs needs new password and rebooted
  • buildhw-a64 19 and 20 need password updated and rebooted.
  • backup01 needs password and reboot and backup ssh agent restarted after that
  • buildvm-s390x 01-14 (zvm) need rebooting

Metadata Update from @zlopez:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: high-gain, medium-trouble, ops

2 years ago

bvmhost-a64-osbs - done
buildhw-a64 19 and 20 rebooted but can't seem to log in to mgmt to change password

Those two (19 and 20) are caviums and had a different password on them due to their weird requirements. I'll try and reset them both to the new one.

Reboots will need to now wait until we are out of freeze. ;(

Is this an automated process or could this be a candidate for automation?

Currently it's not automated at all, but we could definitely look at automating. ;)

Most of our servers are dells and have ipmi and redfish (or whatever dell calls that api).

We have a number that aren't dells tho. (lenovo arm boxes, power9, etc). Almost everything has ipmi.

I am not sure how to get data from ipmi for this. Ansibles ipmi seems pretty limited.

Happy to discuss ways to automate it.

@kevin looks like Dell has a whole collection for Hardware management modules here. Based on some of them not only can we do firmware and the passwords but most management of the hardware. let me know what you think and lets see about getting some test hardware if possible.

I'm a bit worried about the complexity of the modules... and all our hardware isn't dell. ;(

It might be nice if we could just use ipmitool and call that from ansible to get what we need, but not sure how possible that is.

I have a test machine for you: vmhost-x86-01.stg isn't in use anymore. I can make you a login/password for testing. Can you ssh into batcave01.iad2.fedoraproject.org ( see https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/sshaccess/ ). Once you login once, your homedir should exist and I can just put the login/pass in a file in there.

Using that you should be able to login to it's web console, access the api and reach it via ipmitool via it's mgmt ip: vmhost-x86-01-stg.mgmt.iad2.fedoraproject.org (10.3.160.57)
Do note that this network you need to access via batcave01 or noc01 or the internal redhat vpn.

Let me know when you have logged in and I can setup the account for testing. Thanks for working on this!

@kevin I was able to login to batcave01 and reach the Idrac address

in your homedir: drac-access-info.txt should have login/pass you can use to test. :)

Metadata Update from @aheath1992:
- Issue assigned to aheath1992

2 years ago

@kevin The IPMI tool client needs an admin account to work. Can we please elevate the test account to admin?

ok I made test admin... give it a try now?

@aheath1992 Did you have time to try it?

ok, I think we will just close this now.

We confirmed all the passwords were changed.

Providing updates via api would be very nice, but I am not sure it's really possible, so lets just do them manually until it is.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed with Explanation
- Issue status updated to: Closed (was: Open)

2 years ago

Log in to comment on this ticket.

Metadata
Boards 1
ops Status: Backlog