Mentioned in SAL (#wikimedia-operations) [2025-07-17T22:01:29Z] <cmooney@cumin1003> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool site eqsin [reason: repool eqsin to test backhaul cct packet loss, T399221]

Mentioned in SAL (#wikimedia-operations) [2025-07-17T22:01:25Z] <cmooney@cumin1003> START - Cookbook sre.dns.admin DNS admin: pool site eqsin [reason: repool eqsin to test backhaul cct packet loss, T399221]

@cmooney @ssingh I just requested access through the online system. Thank you!

[Claiming this as the clinic duty person this week]

Hi @aranyap yeah you are not in that group.

cmooney@ldap-maint1001:~$ ldapsearch -x cn=wmf | grep aprum 
cmooney@ldap-maint1001:~$

Hi @cmooney ! I'm having some trouble trying to access JupyterHub and after some poking around with @dr0ptp4kt and @BTullis we think it's because I don't have wmf LDAP access but we aren't 100% sure.

Approved. Thank you

this ticket is a prerequisite for https://phabricator.wikimedia.org/T396672 and that @dr0ptp4kt is also readying a patch for additional access in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1165605 to be taken out of WIP once your initial SSH access is established.

These have been imaged

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1002 for host ganeti1054.eqiad.wmnet with OS bookworm completed:

• ganeti1054 (PASS)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • Host up (Debian installer)
  • Add puppet_version metadata (7) to Debian installer
  • Host up (new fresh bookworm OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507171848_vriley_1152194_ganeti1054.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB
  • Updated Netbox status planned -> active
  • The sre.puppet.sync-netbox-hiera cookbook was run successfully

In T215183#11014691, @CDanis wrote:

In T215183#11014363, @Eevans wrote:

[ ... ]

I also never spent much time looking at or thinking about RAID10 hosts, as you said. Honestly I don't remember what debian-installer does in the first place for RAID10 and bootloaders.

In T215183#11014691, @CDanis wrote:

In T215183#11014363, @Eevans wrote:

[ ... ]

For context: We replaced sda in aqs1012 recently (T396970) and were (I believe) bit by this issue. It would seem to have been reimaged since the partman recipe was fixed, and it does not appear in the April 2020 list posted in T215183#6086396, so I'm wondering if a prior replacement didn't get the bootloader installed.

@Eevans, can you check in the BIOS settings of aqs1012 to see if a setting like "Hard drive failover" exists, per T215183#6718961 ?

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1002 for host ganeti1054.eqiad.wmnet with OS bookworm

In T215183#11014363, @Eevans wrote:

Has there been any progress toward goal #2? I didn't see where anything had been added to the mentioned runbook.

As a follow-up, I did find a device with a missing bootloader: aqs1014, which went up after it's partman recipe was fixed (it has had SSDs replaced in the years since though)

ganeti1054 has moved into A4 U38

Regarding the jumbo-frame complication on the plan to move to one link we are arranging to connect a second 25G on each of these new hosts for the storage vlan. See below tasks:

Has there been any progress toward goal #2? I didn't see where anything had been added to the mentioned runbook.

checked the physical cables and everything lines up right. couldn't get into the BMC. re-ran the reqular provisioning script and can access the BMC now. But won't let me set the root password in the script. I can login to the BMC with the one printed on the luggage tag. I'll DM it to you. I don't wanna add the root user if you still need to test on that.

I've stopped it anyway, if you could start it up again after finishing, it would help me a lot, thank you.

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1002 for host ganeti1053.eqiad.wmnet with OS bookworm completed:

• ganeti1053 (PASS)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • Host up (Debian installer)
  • Add puppet_version metadata (7) to Debian installer
  • Host up (new fresh bookworm OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507162234_vriley_643342_ganeti1053.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB
  • Updated Netbox status planned -> active
  • The sre.puppet.sync-netbox-hiera cookbook was run successfully

Icinga downtime and Alertmanager silence (ID=ce8f4e27-d454-43c0-b1b5-892d46c710a6) set by jynus@cumin1003 for 1 day, 0:00:00 on 1 host(s) and their services with reason: failed disk

backup1007.eqiad.wmnet

Crickets in the main from Arelion, one update earlier.

2025-07-17 14:08

This time if fully Failed, so please change it. Do I stop the server first?

@jcrespo just fyi automated ticket was opened again for this host

This is now complete.

In T399221#11014023, @ssingh wrote:

I think leaving eqsin depooled given that it is off-peak there and observing this for a few hours is my vote.

Arelion want to close the ticket as they see no issue. I asked that they don't. Perhaps for now we just leave eqsin depooled and the circuit in the active traffic path? If the reported RTT remains steady we can re-pool after a reasonable period has elapsed? And if it jumps up we can re-do the iperf tests at the time to try to confirm the packet loss has returned?

Not sure how to progress this one. Still see zero packet loss over the link, even running for a longer period (5 mins this time):

cmooney@cp5017:~$ iperf -s -i10 -u -w512k
------------------------------------------------------------
Server listening on UDP port 5001
UDP buffer size: 1000 KByte (WARNING: requested  500 KByte)
------------------------------------------------------------
[  3] local 10.132.0.17 port 5001 connected with 10.192.48.35 port 39207
[ ID] Interval       Transfer     Bandwidth        Jitter   Lost/Total Datagrams
[  3] 0.0000-10.0000 sec   125 MBytes   105 Mbits/sec   0.023 ms    0/89169 (0%)
[  3] 10.0000-20.0000 sec   125 MBytes   105 Mbits/sec   0.011 ms    0/89165 (0%)
[  3] 20.0000-30.0000 sec   125 MBytes   105 Mbits/sec   0.009 ms    0/89164 (0%)
[  3] 30.0000-40.0000 sec   125 MBytes   105 Mbits/sec   0.025 ms    0/89165 (0%)
[  3] 40.0000-50.0000 sec   125 MBytes   105 Mbits/sec   0.010 ms    0/89164 (0%)
[  3] 50.0000-60.0000 sec   125 MBytes   105 Mbits/sec   0.014 ms    0/89165 (0%)
[  3] 60.0000-70.0000 sec   125 MBytes   105 Mbits/sec   0.022 ms    0/89165 (0%)
[  3] 70.0000-80.0000 sec   125 MBytes   105 Mbits/sec   0.022 ms    0/89164 (0%)
[  3] 80.0000-90.0000 sec   125 MBytes   105 Mbits/sec   0.015 ms    0/89164 (0%)
[  3] 90.0000-100.0000 sec   125 MBytes   105 Mbits/sec   0.038 ms    0/89161 (0%)
[  3] 100.0000-110.0000 sec   125 MBytes   105 Mbits/sec   0.023 ms    0/89169 (0%)
[  3] 110.0000-120.0000 sec   125 MBytes   105 Mbits/sec   0.031 ms    0/89164 (0%)
[  3] 120.0000-130.0000 sec   125 MBytes   105 Mbits/sec   0.040 ms    0/89166 (0%)
[  3] 130.0000-140.0000 sec   125 MBytes   105 Mbits/sec   0.016 ms    0/89164 (0%)
[  3] 140.0000-150.0000 sec   125 MBytes   105 Mbits/sec   0.018 ms    0/89164 (0%)
[  3] 150.0000-160.0000 sec   125 MBytes   105 Mbits/sec   0.054 ms    0/89166 (0%)
[  3] 160.0000-170.0000 sec   125 MBytes   105 Mbits/sec   0.017 ms    0/89163 (0%)
[  3] 170.0000-180.0000 sec   125 MBytes   105 Mbits/sec   0.032 ms    0/89165 (0%)
[  3] 180.0000-190.0000 sec   125 MBytes   105 Mbits/sec   0.033 ms    0/89165 (0%)
[  3] 190.0000-200.0000 sec   125 MBytes   105 Mbits/sec   0.023 ms    0/89164 (0%)
[  3] 200.0000-210.0000 sec   125 MBytes   105 Mbits/sec   0.016 ms    0/89165 (0%)
[  3] 210.0000-220.0000 sec   125 MBytes   105 Mbits/sec   0.040 ms    0/89165 (0%)
[  3] 220.0000-230.0000 sec   125 MBytes   105 Mbits/sec   0.014 ms    0/89165 (0%)
[  3] 230.0000-240.0000 sec   125 MBytes   105 Mbits/sec   0.021 ms    0/89164 (0%)
[  3] 240.0000-250.0000 sec   125 MBytes   105 Mbits/sec   0.017 ms    0/89165 (0%)
[  3] 250.0000-260.0000 sec   125 MBytes   105 Mbits/sec   0.014 ms    0/89165 (0%)
[  3] 260.0000-270.0000 sec   125 MBytes   105 Mbits/sec   0.020 ms    0/89165 (0%)
[  3] 270.0000-280.0000 sec   125 MBytes   105 Mbits/sec   0.017 ms    0/89163 (0%)
[  3] 280.0000-290.0000 sec   125 MBytes   105 Mbits/sec   0.022 ms    0/89165 (0%)
[  3] 290.0000-300.0000 sec   125 MBytes   105 Mbits/sec   0.021 ms    0/89162 (0%)
[  3] 300.0000-300.0001 sec  2.87 KBytes   367 Mbits/sec   0.035 ms    0/    2 (0%)
[  3] 0.0000-300.0001 sec  3.66 GBytes   105 Mbits/sec   0.035 ms    0/2674942 (0%)

@Jclark-ctr as discussed in our call on Tuesday we will be connecting the second SFP port on these hosts to the switches too, as we need to solve the MTU issue before proceeding with T399180: Cloudcephosd: migrate to single network uplink.

Mentioned in SAL (#wikimedia-operations) [2025-07-17T15:28:12Z] <topranks> un-drain Arelion transport circuit from codfw -> eqsin to test performance T399221

Mentioned in SAL (#wikimedia-operations) [2025-07-17T14:38:23Z] <cmooney@cumin1003> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool site eqsin [reason: depool eqsin to test backhaul cct packet loss, T399221]

Mentioned in SAL (#wikimedia-operations) [2025-07-17T14:38:19Z] <cmooney@cumin1003> START - Cookbook sre.dns.admin DNS admin: depool site eqsin [reason: depool eqsin to test backhaul cct packet loss, T399221]

Ok so I have a provision script change that seems to work, but it doesn't touch anything on the network PXE / FixedBootOrder config (except ensuring that UEFI Hdd is the first).

Change #1170301 merged by Stevemunene:

[operations/puppet@production] hdfs: Add an-worker 1176|1179|1186 to analytics cluster

https://gerrit.wikimedia.org/r/1170301

Updated Firmware on idrac while logged in thanks for assistance @jcrespo

Note my prediction is that we will need 3 new disks, not only 1 to be replaced (but this can be resolve for now).

I told @Jclark-ctr not to replace the 13th disk yet, as I was more worried about the jbod ones than the RAID:

root@backup1007:~$ megacli -PDList -aall | grep rro
Media Error Count: 0
Other Error Count: 0
Media Error Count: 0
Other Error Count: 0
Media Error Count: 0
Other Error Count: 0
Media Error Count: 0
Other Error Count: 0
Media Error Count: 0
Other Error Count: 0
Media Error Count: 0
Other Error Count: 0
Media Error Count: 0
Other Error Count: 0
Media Error Count: 0
Other Error Count: 0
Media Error Count: 0
Other Error Count: 0
Media Error Count: 0
Other Error Count: 0
Media Error Count: 0
Other Error Count: 0
Media Error Count: 0
Other Error Count: 0
Media Error Count: 0
Other Error Count: 0
Media Error Count: 6
Other Error Count: 1
Media Error Count: 0
Other Error Count: 0
Media Error Count: 0
Other Error Count: 0
Media Error Count: 0
Other Error Count: 0
Media Error Count: 0
Other Error Count: 0
Media Error Count: 0
Other Error Count: 0
Media Error Count: 0
Other Error Count: 0
Media Error Count: 0
Other Error Count: 0
Media Error Count: 0
Other Error Count: 0
Media Error Count: 0
Other Error Count: 0
Media Error Count: 0
Other Error Count: 0
Media Error Count: 0
Other Error Count: 1339
Media Error Count: 0
Other Error Count: 1339

Replaced Failed Drive Thanks for the assistance with this @BTullis