In T388542#10810967, @Clement_Goubert wrote:

In T388542#10804697, @kamila wrote:

I attempted to migrate the two wikimediaevents jobs, and while the global one runs successfully, the per-wiki one logs

The "WikimediaEvents" extension must be installed for this script to run. Please enable it and then try again.

and exits 1.

The global and per-wiki job ran on the exact same image, so this does not seem like a mw-on-k8s problem. I wonder if this was already failing on bare metal. @Dreamy_Jazz could you please check on this?

(Note: As the per-wiki job runs on all.dblist, I'm wondering if this could be failing because the extension is not enabled on private wikis as per https://gerrit.wikimedia.org/g/operations/mediawiki-config/+/89964f271ad05e13a2a44916685ac0b35c062348/wmf-config/InitialiseSettings.php#8808 .)

You should be able to change the invocation from

/usr/local/bin/foreachwikiindblist /srv/mediawiki/dblists/all.dblist extensions/WikimediaEvents/maintenance/UpdatePeriodicMetrics.php --verbose

to

/usr/local/bin/foreachwikiindblist "all - private" extensions/WikimediaEvents/maintenance/UpdatePeriodicMetrics.php --verbose

in order to skip the private wikis. The former version of foreachwikiindblist didn't have set -e, so would happily continue the loop if one of the calls exited 1.

further progress soft-blocked by T388969 -- while the same workaround could be applied everywhere, it is not a great workaround.

I attempted to migrate the two wikimediaevents jobs, and while the global one runs successfully, the per-wiki one logs

All jobs are migrated and running successfully.

A manual re-run of the job just completed successfully. Thank you @hnowlan for the help!

Fix for my guess merged, should we re-run this job?

First look shows /etc/php/7.4/fpm/env in mounts, while the image runs on 8.1. Investigating further.

ServiceOps is currently working on several things that require modifying the CI code (k8s 1.31 compatibility, mw-script and mw-cron on k8s etc), so I wouldn't be surprised if everything was getting rerun just because the CI code keeps changing. We'd also rather not introduce more variability in what we expect as CI output while we're working on modifying CI. I believe it would make more sense to revisit this once we're done with these changes.

The hard-coded hack works, so kubectl no longer complains. I'm lowering the priority, but not closing the task, as I am not sure whether @JMeybohm considers it "resolved" with the hack.

Oh, thank you @jhathaway , excellent timing :D Much appreciated!

This happens on different networks (home wifi and mobile hotspot) and also in a private browser window.

This is what I see:

Maybe let's drop url label for 404 CM metrics for now, it seems like a good enough solution to me. I'm happy to assist/brainstorm on the prometheus relabeling part

The fix hasn't been merged. I pinged them, I'll see if I can get things moving.

@kamila do you think you could help with that?

Sure, I'll prepare a patch that adds those labels. For the comment, it looks like I can also export an annotation as a Prometheus annotation, I can include that too.

The easiest way to do this would be to add a header with the filename to the shellbox client. However, in the videoscalers case, the shellbox client currently doesn’t know the filename (TMH just passes a hardcoded “original.video”) and a non-hacky way to do this would require code changes across Shellbox, TMH and potentially other shellbox clients.

All done 🎉

All done 🎉

I did not work on this because I am not sure about running it in wikikube. Since people are making progress on k8s-aux in codfw, I think it makes more sense to wait for T381417 .

Core functionality is merged 🎉 I have filed tasks for remaining fixes/improvements.

+1 to reverting and leaving the pointer to T371273 in the comments, for the already mentioned reasons.

Done, let me know in case there are any problems.

Thanks for confirming @seanleong-WMDE, I will finalise this as soon as the group approvers confirm.

@odimitrijevic @Milimetric @WDoranWMF @Ahoelzl @Ottomata could one of you approve Sean's access request please? Thank you!

@Dzahn it appears that Sean is already in the LDAP groups, unless I'm misunderstanding something:

kamila@mwmaint2002:~$ ldapsearch -x cn=nda | grep sean
member: uid=seanleong-wmde,ou=people,dc=wikimedia,dc=org
kamila@mwmaint2002:~$ ldapsearch -x cn=wmde | grep sean
member: uid=seanleong-wmde,ou=people,dc=wikimedia,dc=org

This also requires approval of one of the group approvers.

@seanleong-WMDE Can you please confirm you've read https://wikitech.wikimedia.org/wiki/Data_Platform/Data_access#User_responsibilities ? Thank you!