[Claiming this as the clinic duty person this week]

In T399419#11014274, @Vgutierrez wrote:

I like the idea of using two passes instead of rm -r, so something like:

• find /var/lib/acme-chief/certs -type f -mtime +365 -delete
• find /var/lib/acme-chief/certs -type d -empty -delete

thanks for the debugging @bd808

Arelion want to close the ticket as they see no issue. I asked that they don't. Perhaps for now we just leave eqsin depooled and the circuit in the active traffic path? If the reported RTT remains steady we can re-pool after a reasonable period has elapsed? And if it jumps up we can re-do the iperf tests at the time to try to confirm the packet loss has returned?

Things were stable for a few hours even after @cmooney made the fix above but starting ~20:00 UTC, we had a page for text-https in eqsin and also observed purged and ATS issues in talking to backends, indicating issues on the link.

Things look fine so marking as resolved; please re-open if there are any issues.

Done in https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1167222.

Expiry has been sent to end of FY (June 2026) and contact has been set to Suman to get this request going. We can always update that later.

@SCherukuwada: Hi, thanks for the approval above. Since the email ends in -ctr, can you let us know the contract period and also the point of contact (which I assume will be you?). Thanks!

Additionally please note that for logstash-access, you can simply request it via https://idm.wikimedia.org/. See https://wikitech.wikimedia.org/wiki/SRE/LDAP/Groups/Request_access.

Hi. For volunteers requesting access, https://wikitech.wikimedia.org/wiki/SRE/Production_access#Add_a_volunteer_to_an_access_group needs to be followed.

Please re-open if there any issues.

Resolving this as the above has been completed at least from SRE's side. @aranyap: please re-open if there are any issues.

One additional point: we also ruled out any Kafka-specific issues; thanks to @brouberol.

Summary of where we are now:

On the DNS hosts as of today, we have an alert in place if we detect a mismatch between the service state as defined by confd/confctl and the advertisements of the VIPs on the DNS hosts themselves. Here is how it works in case we are interested in expanding this to other places:

Happy to collaborate on this, FWIW.

I am going to tackle this for the DNS hosts at least and then we can revisit a generic solution.

I updated Markmonitor to further add the v6 glue records:

The ECH experiment has been reverted as of today.

I checked with Brandon and he confirmed that while not strictly required, we can add the AAAA records here so let me know if you want to do that and I can see how to effect that change in Markmonitor.

Ah, thanks for the context. Looking at the domains in question, all of them are delegated at Markmonitor, which explains why they have the glue records in place already. (As compared to wikimedia.cloud, which we delegate from ns[0-2] for obvious reasons to delegate specific subdomains). wikimediacloud.org has NS set to ns[0-2].wikimedia.org and further has glue records for ns[01].openstack.eqiad1.wikimediacloud.org set explicitly in Markmonitor.

You can add them if required even though my reading of the other tasks and RFC indicates it is optional. That being said, why only v6 glues and not v4? I don't see v4 glue records in the zone file for wikimedia.cloud but I guess I am missing some more context.

In T387145#10886841, @cmooney wrote:

Check with @cmooney for changes required to hieradata/common/lvs/interfaces.yaml (to add lvs1016 there) and also to hieradata/role/eqiad/lvs/balancer.yaml in case profile::lvs::tagged_subnets needs to be updated (don't think so but check).

To my knowledge this only now applies to low-traffic services in eqiad/codfw (behind K8s and some search ones). I believe everything else is using IPIP so L2 adjacency / extra interfaces is not required on the LVS nodes serving those.

Commenting on this with my own understanding and for review of others. After that, letting @BCornwall handle updating the task description.

@Muehlenhoff: Both of these are decommissioned. Let me know if any other action is required from my end, thanks!

Sounds good, thanks. Let me know if I can help with anything.

In T395796#10875858, @ayounsi wrote:

Had a quick chat with Moritz and Sukhbir.
We prefer not to wait for the Bird work to progress on setting up the Routed Ganeti cluster, so we're going to run doh/durum in a non-redundant way (only running on a single ganeti hosts), while creating a 3 nodes routed Ganeti cluster.

This will also permit me to look at setting up a temporary BGP setup on durum7003 (and extend it to doh7003 if it's solid enough).

So next step is for traffic to do all the steps in the task description except the doh7003 steps for now.

(Adding @Fabfur who will lead this from Traffic with Brett.)

In T394263#10839754, @MoritzMuehlenhoff wrote:

In T394263#10839700, @ssingh wrote:

We discussed this in the Traffic meeting and there are no concerns from our side in moving ahead. Let us know the date/time we want this roll out and we can coordinate!

We'd like to start next Monday. The firsts steps are mostly internal to Ganeti, we'll reach out when VMs need to be shuffled. Ok?

Hi @bd808: we discussed this in the Traffic meeting today and will need some more time to prioritize this. I will follow up again next week. (We are interested but with the end of the quarter, we are still triaging this among the other work.)

We discussed this in the Traffic meeting and there are no concerns from our side in moving ahead. Let us know the date/time we want this roll out and we can coordinate!

OK, should be fine. We will follow up with a confirmation after discussing this in the Traffic meeting on Tuesday.

@greg: The change has been made and Markmonitor has been updated. NS records have a fairly large TTL of 86400 seconds or one day, so it will be a while (1-2 days) before the various recursors pick up the change (not counting the time it takes for the update to propagate from the registrar to the gives TLD).

Hi @greg: (Using this task is perfectly fine, thanks). The delegation for wiki.gives will be done at Markmonitor's (registrar) side, so we will need to update the NS records there and point them to the above, moving them away from ns[0-2].wikimedia.org where they currently point to. There is no other DNS deployment required on our end in that case; we only need to do that if we were doing subdomain delegation, which we are not.

Thanks for the task and the detailed description! Mostly sounds good to me for the VMs that I "own" but will need to check with Traffic for others. Is there a tentative date for when you want to get to this?

In T358887#10817983, @bd808 wrote:

In T358887#10817876, @ssingh wrote:

That being said, if there is a requirement for constantly keeping blocked nets updated in Beta, then we should have the discussion of how to fix it and also to plan for it.

The new T393487: 2025 tracking task for Beta Cluster (deployment-prep) traffic overload protection (blocking unwanted crawlers) parent task and the https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/2RFFHXSI6INQDJ2AQ7U3IQ2HTHT5J4VL/ mailing list post give a bit of context for the current wave of excess traffic combat that we are doing in deployment-prep. In the big picture there are a couple of issues that I would like to resolve:

• Closer alignment with tools used in the production CDN edge would ideally provide a nicer experience when we need to block traffic manually. It also helps with the "drinking our own champaign" aspects of the Beta Cluster deployment by giving folks a place to practice skills and test features that are eventually bound for production use.
• More automated protections for Beta Cluster are desirable because there are few to no people dedicated to maintaining the environment vs a number of technical volunteers and Foundation and affiliate staff who use the platform quite regularly for integration, acceptance, and regression testing of various types. This work might lead to learning things that can help other bits of the Wikimedia infrastructure that are not protected by the current CDN for various reasons.

This task and T393481: Add allowlist to make poking holes in abuse_networks:blocked_nets:networks easier are probably more stop gaps than direct movement towards either of the larger goals.

In T358887#10817838, @bd808 wrote:

If I'm understanding @ssingh's comments on the now abandoned patch, it sounds like setting up the requestctl stack in Beta Cluster might be our better long term solution.

Thanks @RobH! @cmooney: yeah, I updated the task description to reflect that but we though we should get this checked out anyway, since it's the integrated NIC. Thanks for checking, both!

The host has been depooled so you can reboot or shut it down without checking with us. Thanks for the quick response Rob!

In T393616#10802011, @RobH wrote:

I'll open a case with Dell, which will inevitably require the firmware on the NIC, mainboard, and idrac be updated before they'll authorize a replacement.

Was the server under any particular load before the error we can recreate, or did it just randomly fire?

So we have trimmed it down even with a simple git maintenance run:

In T392848#10792865, @Volans wrote:

No, you're right, current_state in the icinga status reports the state independently of the downtime. An alternative option to Luca's implementation could be to add a skip_downtimed flag and when set ignore the status for the services that have scheduled_downtime_depth > 0. No particular strong opinion, just that in general is error prone to specify services names as they can easily change dynamically from Puppet's code and hence not match anymore. For this reason the current methods that work on services names use regexes to try to be a bit more flexible and future-proof.

In T392848#10791709, @Volans wrote:

Have you considered just downtiming the affected service during the operation with downtime_services or the related context manager version of it?

I have been facing this, intermittently, over the past week or so. Refreshing helps sometimes but not always. Some additional points that might help with debugging and as I have personally observed: the selectors that fail are random but during the interval within which they fail, it's pretty consistent.

In T393034#10781753, @LSobanski wrote:

The current hypothesis is that during the DNS change both hosts were considered to be primary and unexpected replication took place for approximately an hour and 20 minutes (at most). The result was gerrit2002 deleting changes made on gerrit1003.