Wiki has been created

Mentioned in SAL (#wikimedia-operations) [2024-09-06T13:51:59Z] <btullis@cumin1002> START - Cookbook sre.wikireplicas.add-wiki for database bdrwiki (T371759)

Mentioned in SAL (#wikimedia-operations) [2024-09-06T13:52:10Z] <btullis@cumin1002> END (FAIL) - Cookbook sre.wikireplicas.add-wiki (exit_code=99) for database bdrwiki (T371759)

Interesting. The sre.wikireplicas.add-wiki cookbook failed when running /usr/local/sbin/maintain-views on an-redacteddb1001 with:

pymysql.err.OperationalError: (1044, "Access denied for user 'maintainviews'@'localhost' to database 'bdrwiki\\_p'")

All of the other wikireplica servers succeeded on running this command. I'll check to see what the grants are for 'maintainviews'@'localhost' and whether the bdrwiki_p database had been created on this host.

That database has definitely not been created on an-redacteddb1001.

root@an-redacteddb1001:s5[(none)]> use bdrwiki_p;
ERROR 1049 (42000): Unknown database 'bdrwiki_p'

But is is present on the other clouddb servers.

root@clouddb1016:s5[(none)]> use bdrwiki_p;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
root@clouddb1016:s5[bdrwiki_p]>

@ABran-WMF @Ladsgroup - could you help me to backtrack through the process to find out where this is going wrong, please?

an-redacteddb10054 section s5 is replicating from db1154.eqiad.wmnet:3315 and there is no replication lag. Is this the correct upstream replica?
At what stage and on which host(s) is the bdrwiki_p database actually created?

bdrwiki_p just provides the view, the actual database is bdrwiki which should be there so the replication and the data is replicated. That's fine. Just the steps to create the db is not followed (I documented them in https://wikitech.wikimedia.org/wiki/Add_a_wiki#Before_maintain_views:_DBAs'_work)

In T371759#10125938, @Ladsgroup wrote:

bdrwiki_p just provides the view, the actual database is bdrwiki which should be there so the replication and the data is replicated. That's fine. Just the steps to create the db is not followed (I documented them in https://wikitech.wikimedia.org/wiki/Add_a_wiki#Before_maintain_views:_DBAs'_work)

OK, thanks. That makes sense. I can run those commands on an-redacteddb1001 on this occasion.

I wonder if we should add this step to the sre.wikireplicas.add-wiki cookbook, for future runs?

Once that's done, create the ${wiki}_p database and granted access to labsdbuser in the cloud replicas (otherwise a bug in MariaDB will cause the scripts to fail). This what DBAs need to do on each cloud replicas (not sanitariums):

root@an-redacteddb1001:s5[bdrwiki]> create database bdrwiki_p;
Query OK, 1 row affected (0.001 sec)

root@an-redacteddb1001:s5[bdrwiki]> GRANT SELECT, SHOW VIEW ON `bdrwiki_p`.* TO 'labsdbuser';
Query OK, 0 rows affected (0.001 sec)

root@an-redacteddb1001:s5[bdrwiki]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.225 sec)

Running the cookbook again.

Mentioned in SAL (#wikimedia-operations) [2024-09-06T14:25:20Z] <btullis@cumin1002> START - Cookbook sre.wikireplicas.add-wiki for database bdrwiki (T371759)

Mentioned in SAL (#wikimedia-operations) [2024-09-06T14:51:03Z] <btullis@cumin1002> END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database bdrwiki (T371759)

Arnaud would be the right person to say whether that is being cookbooked or not.

In T371759#10125982, @BTullis wrote:

Any views on whether this creation of the ${wiki}_p database and the associated grant would be a good addition to the cookbook? If it is something that you have already automated on your side (or you already have plans to do so) then there's probably no point.

@BTullis T366146 has been created to automate this indeed :)