#10486 Provide iscsi (block device) lun for CentOS Main/Ref mirror in IAD2
Closed: Fixed 2 years ago by arrfab. Opened 2 years ago by arrfab.

Describe what you would like us to do:


Based on email thread : the Dell r740 node in IAD2 dedicated for CentOS Main/Ref mirror is slowly reaching end-of-capacity status and we were told that instead of buying some other disks for local expansion (DAS), we should instead switch to the Fedora Netapp filer, and so ask a 20TiB iscsi lun, served over a specific storage vlan, and so asking also network switch to have a dedicated 10Gbit/SFP+ port configured (and fiber cable attached to our node)

When do you need this to be done by? (2022/05/31)


Just putting a date, as I know that it will take some time to have the lun configured, with proper ACL and also vlan in place, so this is not a hard deadline but clearly something to still do before storage capacity is full (as CentOS Stream 9 is pushing more and more data every day)


Metadata Update from @kevin:
- Issue assigned to kevin
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: medium-gain, medium-trouble, ops

2 years ago

The vlan for dedicated storage networks for systems on 703 vlan should be vlan 705 (10.3.165.0/24). The centos box should already have fibre to its secondary network port and should just need to get switched on and to the 705. Then the iscsi share will need to be set up forsharing to that.

@kevin @mobrien

As Stream 9 is getting bigger and bigger , we recently went under 5% threshold for disk usage (and can't extend it , except if we buy some disks, but don't think that it was possible reason why we were pointed to this solution with the Netapp) : do we have some kind of ETA about when we can get that access ? :)

Worth knowing that it's also the machine used by mirrormanager in IAD2 to retrieve content and so be able to produce up2date metalinks

@arrfab hopefully not too long.

Can you give me (here or elsewhere if you prefer) the mac address of that 10g interface we are going to use and the exact machine? I see two in inventory, both in rack 101 but one at U37 and one at U40.
May need you to coordinate with RHIT to generate traffic on the interface, etc.

I can file a ticket once I have the mac and which exact machine and I can tell them to coordinate with you on bringing the interface up?

After that it should just be creating the iscsi lun.

used internal channel to send you required info ;-)

Internal ticket filed.

ok, so we got the interface all on the right vlan and able to reach the netapp.

However, it seems there's a 16TB limit for iscsi luns. ;(

So, we could:

1) make 2 luns and you can divide things up accross them?
2) switch to just a nfs volume?
3) punt back to getting more local disks for the server.

yeah, as said on internal chat, going with two LUNs is fine, as I'll just use these with LVM so let's proceed with that plan.

ok. I think I have these created.

There's 4 ip's (for mpath) for portal: 10.3.177.11, 10.3.177.12, 10.3.177.13, 10.3.177.14

/vol/centos_mirror_iscsi_0/lun0
/vol/centos_mirror_iscsi_1/lun1

are the luns. It should advertise them to you on discovery.
Let me know if you need anymore info or something isnt working. :)

Thanks a lot @kevin !
I just had to configure multipath (it's been a while since I played with multipath as we never had any failure tolerant infra for centos.org :) ) but here we go :

3600a098038314334463f504b5a372f4d dm-5 NETAPP  ,LUN C-Mode      
size=10T features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=0 status=active
| `- 21:0:0:1 sdf     8:80  active undef running
`-+- policy='service-time 0' prio=0 status=enabled
  `- 19:0:0:1 sdd     8:48  active undef running
3600a098038314334463f504b5a372f4c dm-4 NETAPP  ,LUN C-Mode      
size=10T features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=0 status=active
| `- 21:0:0:0 sde     8:64  active undef running
`-+- policy='service-time 0' prio=0 status=enabled
  `- 19:0:0:0 sdc     8:32  active undef running

So I was then able (BAU) to create a new lvm VG/LV and I'll start moving data.
That should give use ~5TiB of margin for now (we were reaching local limit) and so we'll see just when/how we can just extend this, but that will be another ticket and hopefully not too soon :D

Metadata Update from @arrfab:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Metadata
Boards 1
ops Status: Backlog