☂ Migrate Wikitech to Kubernetes
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	jijiki
	Oct 7 2021, 11:00 AM

Description

This is an ☂ task for Wikitech's migration to Kubernetes.

Wikitech has been a long standing snowflake to the point where it is becoming unmanageable. The high level plan is to "weed out" everything that makes wikitech different than any other wiki, and then serve it from MW-on-K8s.

Accounts and Developer Services

Remove the dependency between Striker, Horizon, and Wikitech
Migrate Wikitech accounts off LDAP to the Single User Login (SUL)
Remove the OpenStack extension (SSH key management)

Infrastructure

Migrate Wikitech's jobqueue
mediawiki-config changes
apache changes
Route wikitech to MW-on-K8s

Going Live Tue 1st Oct 2024

Wikitech users have been informed that we will be disabling ldap authentication
merge and deploy de-wikitech mediawiki-config
Switch Wikitech traffic to wikikube T371358

Details

Subject	Repo	Branch	Lines +/-
reverse-proxy: Drop all public ips except cloudweb2002-dev.codfw.wmnet	operations/mediawiki-config	master	+3 -20
snapshots: Dump wikitech (labswiki) like any other wiki	operations/puppet	production	+2 -2
openstack: Stop running Wikitech jobs on cloudweb hosts	operations/puppet	production	+1 -1
hieradata: Stop monitoring Wikitech on cloudweb hosts	operations/puppet	production	+4 -5

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T375217 Complete upgrading WMCS bare metal hosts from Bullseye to Bookworm
Open	None	T376277 Reimage cloudweb hosts to bookworm
In Progress	None	T290536 Serve production traffic via Kubernetes
Resolved	jijiki	T371378 Cleanup: Wikitech code leftovers
Resolved	jijiki	T292707 ☂ Migrate Wikitech to Kubernetes
Resolved	jijiki	T363125 sustainability of wikitech.wikimedia.org
Resolved	jijiki	T371358 Traffic: Direct wikitech.wikimedia.org to mw-on-k8s
Resolved	jijiki	T371359 Migrate Wikitech's Jobqueue
Resolved	jijiki	T371374 mediawiki-config: consolidate labswiki
Resolved	jijiki	T371608 Memcached: Migrate wikitech to main memcached
Resolved	jijiki	T371537 MVP: Privately serve wikitech via mwdebug1001
Resolved	jijiki	T371360 Apache: Include Wikitech in mw-on-k8s' virtual hosts
Resolved	• Ladsgroup	T371588 wikitech self-auth: Allow wikitech to use its own internal authentication
Resolved	jijiki	T371592 LdapAuthentication: Disable extension from Wikitech
Resolved	SLyngshede-WMF	T359820 Developer Account Blocking: Migrate the one-stop Developer (un)Blocking from Wikitech to Bitu
Resolved	SLyngshede-WMF	T376991 Publically available log of blocked/unblocked accounts.
Resolved	jijiki	T373615 Communication for Wikitech/Wikimedia Developer Account migration
Resolved	jijiki	T374114 Review/update wikitech-static syncing after wikitech moves to Kubernetes

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

• taavi added a project: wikitech.wikimedia.org.Oct 7 2021, 11:12 AM

• taavi subscribed.

Marostegui subscribed.Nov 26 2021, 10:55 AM

• Ladsgroup subscribed.Nov 26 2021, 10:58 AM

Krinkle updated the task description. (Show Details)Nov 26 2021, 12:02 PM

LSobanski subscribed.Nov 26 2021, 1:12 PM

bd808 added a parent task: T237773: Move Wikitech onto the production MW cluster.Nov 28 2021, 4:32 PM

bd808 subscribed.

MoritzMuehlenhoff subscribed.Nov 29 2021, 9:03 AM

• Ladsgroup mentioned this in T201662: Change the MySQL passwords.Jan 20 2022, 1:20 PM

bd808 moved this task from Backlog to Snowflakes on the wikitech.wikimedia.org board.Apr 4 2022, 9:30 PM

jijiki moved this task from Incoming 🐫 to 🙈🙉🙊Backlog on the serviceops board.Sep 28 2022, 2:20 PM

jijiki moved this task from 🙈🙉🙊Backlog to 🌻Mediawiki on the serviceops board.Nov 9 2022, 2:16 PM

Any ETA on when this could happen?

We will soon be in next quarter OKR/planning season, we 'll post an update then if we manage to schedule it for next quarter. Just so that we have all data that we need to make the decision, is this a blocker for Data Persistence in any way?

It is not a blocker for us per se, but we got hit again: https://phabricator.wikimedia.org/T237773#8930311

In T292707#8930381, @Marostegui wrote:

It is not a blocker for us per se, but we got hit again: https://phabricator.wikimedia.org/T237773#8930311

Cool, noted. Thanks for the info.

To expand upon what Manuel said, we have to manually special-case wikitech in a lot of places which is a lot of work and when we forget to, it usually end up as an incident/outage/major issues making wikitech unusable. This has happened multiple times.

On top of that since wikitech appserver is in public network, we have to open holes in our firewall and add extra grants and that's making me uneasy.

Clement_Goubert updated the task description. (Show Details)Jun 19 2023, 10:19 AM

bd808 mentioned this in T341470: UrlShortener throws DBConnectionError exception on wikitech.Jul 10 2023, 6:03 PM

bd808 mentioned this in T237889: Install php-ldap on all MW appservers.Aug 1 2023, 5:32 PM

bd808 mentioned this in T237773: Move Wikitech onto the production MW cluster.

bd808 mentioned this in T174476: Support video on wikitech wiki.Nov 1 2023, 3:57 PM

jijiki claimed this task.Apr 11 2024, 12:36 PM

Clement_Goubert mentioned this in T362323: Move 100% of external traffic to Kubernetes.Apr 11 2024, 12:46 PM

Andrew added a parent task: T363125: sustainability of wikitech.wikimedia.org.Apr 23 2024, 3:56 AM

bd808 mentioned this in T161859: Make Wikitech an SUL wiki.Jul 22 2024, 9:19 PM

Michael subscribed.Jul 24 2024, 9:57 AM

jijiki merged a task: T237773: Move Wikitech onto the production MW cluster.Jul 26 2024, 8:21 AM

jijiki added subscribers: Andrew, • dcausse, Novem_Linguae and 13 others.

jijiki removed parent tasks: T363125: sustainability of wikitech.wikimedia.org, T237773: Move Wikitech onto the production MW cluster.Jul 26 2024, 8:33 AM

jijiki added a subtask: T363125: sustainability of wikitech.wikimedia.org.

jijiki added a subtask: T161859: Make Wikitech an SUL wiki.Jul 29 2024, 4:14 PM

jijiki renamed this task from Migrate Wikitech to Kubernetes to ☂ Migrate Wikitech to Kubernetes.Jul 30 2024, 10:00 AM

jijiki changed the task status from Open to In Progress.

jijiki raised the priority of this task from Medium to High.

jijiki updated the task description. (Show Details)

jijiki closed subtask T363125: sustainability of wikitech.wikimedia.org as Resolved.Jul 30 2024, 10:03 AM

jijiki added a subtask: T371588: wikitech self-auth: Allow wikitech to use its own internal authentication.Aug 1 2024, 10:53 AM

jijiki removed a subtask: T161859: Make Wikitech an SUL wiki.

Scott_French subscribed.Aug 1 2024, 3:53 PM

jijiki changed the status of subtask T371537: MVP: Privately serve wikitech via mwdebug1001 from Open to In Progress.Aug 8 2024, 3:54 PM

MSantos subscribed.Aug 27 2024, 1:39 PM

joanna_borun subscribed.Aug 29 2024, 1:12 PM

jijiki removed a subtask: T371608: Memcached: Migrate wikitech to main memcached.Aug 29 2024, 3:30 PM

jijiki updated the task description. (Show Details)Aug 29 2024, 3:36 PM

jijiki closed subtask T371537: MVP: Privately serve wikitech via mwdebug1001 as Resolved.Sep 10 2024, 4:02 PM

bd808 mentioned this in T375795: With XWikimediaDebug enabled, wikitech.wikimedia.org gets redirected to foundation.wikimedia.org until Wikitech is on k8s.Sep 26 2024, 8:20 PM

• taavi added a parent task: T305233: consider eliminating labweb/cloudweb hardware servers.Sep 28 2024, 1:30 PM

Mentioned in SAL (#wikimedia-operations) [2024-10-01T11:16:23Z] <effie> Switching wikitech to k8s - T292707

• aborrero awarded a token.Oct 1 2024, 11:21 AM

• Ladsgroup added a parent task: T376129: Database clean ups after migration of wikitech to production.Oct 1 2024, 11:40 AM

jijiki closed subtask T371358: Traffic: Direct wikitech.wikimedia.org to mw-on-k8s as Resolved.Oct 1 2024, 12:39 PM

jijiki closed subtask T371359: Migrate Wikitech's Jobqueue as Resolved.

jijiki closed subtask T371374: mediawiki-config: consolidate labswiki as Resolved.

jijiki reopened subtask T371374: mediawiki-config: consolidate labswiki as Stalled.

@Ladsgroup and I flipped the relevant switches today, for the time being all seems well.

• Ladsgroup closed subtask T371588: wikitech self-auth: Allow wikitech to use its own internal authentication as Resolved.Oct 1 2024, 12:46 PM

Change #1077040 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] hieradata: Stop monitoring Wikitech on cloudweb hosts

https://gerrit.wikimedia.org/r/1077040

Change #1077041 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] openstack: Stop running Wikitech jobs on cloudweb hosts

https://gerrit.wikimedia.org/r/1077041

bd808 added a parent task: T376176: Stashbot needs new owner-only OAuth grant following Wikitech config change.Oct 1 2024, 5:08 PM

jijiki removed a parent task: T376176: Stashbot needs new owner-only OAuth grant following Wikitech config change.Oct 1 2024, 7:33 PM

jijiki added a subtask: T376188: OAuth consumers registered locally at Wikitech are no longer configured to be used .

Change #1077040 merged by Effie Mouzeli:

[operations/puppet@production] hieradata: Stop monitoring Wikitech on cloudweb hosts

https://gerrit.wikimedia.org/r/1077040

Change #1077041 merged by Effie Mouzeli:

[operations/puppet@production] openstack: Stop running Wikitech jobs on cloudweb hosts

https://gerrit.wikimedia.org/r/1077041

Change #1077349 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] reverse-proxy: Drop all public ips except cloudweb2002-dev.codfw.wmnet

https://gerrit.wikimedia.org/r/1077349

jijiki removed a parent task: T376129: Database clean ups after migration of wikitech to production.Oct 2 2024, 11:17 AM

Change #1077403 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] snapshots: Dump wikitech (labswiki) like any other wiki

https://gerrit.wikimedia.org/r/1077403

Change #1077403 merged by Ladsgroup:

[operations/puppet@production] snapshots: Dump wikitech (labswiki) like any other wiki

https://gerrit.wikimedia.org/r/1077403

Change #1077349 merged by jenkins-bot:

[operations/mediawiki-config@master] reverse-proxy: Drop all public ips except cloudweb2002-dev.codfw.wmnet

https://gerrit.wikimedia.org/r/1077349

Mentioned in SAL (#wikimedia-operations) [2024-10-02T16:09:34Z] <taavi@deploy2002> Started scap sync-world: Backport for [[gerrit:1077349|reverse-proxy: Drop all public ips except cloudweb2002-dev.codfw.wmnet (T292707)]]

Mentioned in SAL (#wikimedia-operations) [2024-10-02T16:11:46Z] <taavi@deploy2002> zabe, taavi: Backport for [[gerrit:1077349|reverse-proxy: Drop all public ips except cloudweb2002-dev.codfw.wmnet (T292707)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-10-02T16:16:35Z] <taavi@deploy2002> Finished scap sync-world: Backport for [[gerrit:1077349|reverse-proxy: Drop all public ips except cloudweb2002-dev.codfw.wmnet (T292707)]] (duration: 07m 01s)