User Details
- User Since
- Aug 2 2021, 1:52 PM (206 w, 3 d)
- Availability
- Available
- IRC Nick
- Emperor
- LDAP User
- MVernon
- MediaWiki User
- MVernon (WMF) [ Global Accounts ]
Mon, Jul 7
All done!
Fri, Jul 4
What I assume happened is that some of the thumbnails from the first upload didn't get overwritten when the second upload was made (I purged the image this morning, which should have removed any old thumbnails), and it was those that were getting seen.
I think this may have been a caching issue - if I copy your wikitext into the Sandbox, I get what look to me to be correct thumbnails; and indeed the permalink to the cs.wp page has a thumbnail that looks correct to me.
Wed, Jul 2
@Jclark-ctr the problem with these two nodes was the same as we've had with every one of this batch of Dell servers - they arrive with junk (an EFI partition with something Windows on) on one of the hard disks.
Mon, Jun 30
These corrupt DBs have all been repaired now.
Fri, Jun 27
You can achieved automatic expiry via S3, though, using a lifecycle policy - IBM's docs have a worked example, as part of their bucket lifecycle docs. The ceph upstream docs on the topic are a bit bare-bones.
Using test-cookbook and the currently-in-review check-dbs cookbook on all the thumbnail container dbs, we find the following problems:
- wikipedia-commons-local-thumb.6b -
/srv/swift-storage/accounts1/containers/8721/17d/2211db68672f28e639decfc1f640917d/2211db68672f28e639decfc1f640917d.db on ms-be1083.eqiad.wmnet has errors: row 1745416 missing from index ix_object_deleted_name row 1745417 missing from index ix_object_deleted_name wrong # of entries in index ix_object_deleted_name /srv/swift-storage/sda3/containers/8721/17d/2211db68672f28e639decfc1f640917d/2211db68672f28e639decfc1f640917d.db on ms-be1063.eqiad.wmnet has errors: row 1745416 missing from index ix_object_deleted_name row 1745417 missing from index ix_object_deleted_name wrong # of entries in index ix_object_deleted_name /srv/swift-storage/accounts0/containers/8721/17d/2211db68672f28e639decfc1f640917d/2211db68672f28e639decfc1f640917d.db on ms-be1074.eqiad.wmnet has errors: row 1745416 missing from index ix_object_deleted_name row 1745417 missing from index ix_object_deleted_name wrong # of entries in index ix_object_deleted_name
- wikipedia-commons-local-thumb.6e
/srv/swift-storage/sdb3/containers/26973/1f3/695dccc2f2758a580e42b976670301f3/695dccc2f2758a580e42b976670301f3.db on ms-be2059.codfw.wmnet has errors: row 1632783 missing from index ix_object_deleted_name wrong # of entries in index ix_object_deleted_name /srv/swift-storage/sda3/containers/26973/1f3/695dccc2f2758a580e42b976670301f3/695dccc2f2758a580e42b976670301f3.db on ms-be2058.codfw.wmnet has errors: row 1632783 missing from index ix_object_deleted_name wrong # of entries in index ix_object_deleted_name /srv/swift-storage/accounts0/containers/26973/1f3/695dccc2f2758a580e42b976670301f3/695dccc2f2758a580e42b976670301f3.db on ms-be2076.codfw.wmnet has errors: row 1632783 missing from index ix_object_deleted_name wrong # of entries in index ix_object_deleted_name
- wikipedia-commons-local-thumb.79
/srv/swift-storage/sda3/containers/30531/b33/774332fe929445555e757c805d65eb33/774332fe929445555e757c805d65eb33.db on ms-be1067.eqiad.wmnet has errors: row 4537890 missing from index ix_object_deleted_name wrong # of entries in index ix_object_deleted_name /srv/swift-storage/sdb3/containers/30531/b33/774332fe929445555e757c805d65eb33/774332fe929445555e757c805d65eb33.db on ms-be1070.eqiad.wmnet has errors: row 4537890 missing from index ix_object_deleted_name wrong # of entries in index ix_object_deleted_name /srv/swift-storage/accounts0/containers/30531/b33/774332fe929445555e757c805d65eb33/774332fe929445555e757c805d65eb33.db on ms-be1089.eqiad.wmnet has errors: row 4537890 missing from index ix_object_deleted_name wrong # of entries in index ix_object_deleted_name
- wikipedia-commons-local-thumb.99
/srv/swift-storage/sdb3/containers/53069/95e/cf4db1352e66ef14017ee2d8a280d95e/cf4db1352e66ef14017ee2d8a280d95e.db on ms-be2064.codfw.wmnet has errors: wrong # of entries in index ix_object_deleted_name
- wikipedia-commons-local-thumb.b7
/srv/swift-storage/sdb3/containers/43276/ec7/a90c2201082e532c429f273788ee6ec7/a90c2201082e532c429f273788ee6ec7.db on ms-be1066.eqiad.wmnet has errors: row 1612987 missing from index ix_object_deleted_name wrong # of entries in index ix_object_deleted_name /srv/swift-storage/accounts1/containers/43276/ec7/a90c2201082e532c429f273788ee6ec7/a90c2201082e532c429f273788ee6ec7.db on ms-be1090.eqiad.wmnet has errors: row 1612987 missing from index ix_object_deleted_name wrong # of entries in index ix_object_deleted_name /srv/swift-storage/accounts0/containers/43276/ec7/a90c2201082e532c429f273788ee6ec7/a90c2201082e532c429f273788ee6ec7.db on ms-be1087.eqiad.wmnet has errors: row 1612987 missing from index ix_object_deleted_name wrong # of entries in index ix_object_deleted_name
- wikipedia-commons-local-thumb.bb
/srv/swift-storage/accounts0/containers/25743/3d0/648f2c55d993f0a991ca1cffd66423d0/648f2c55d993f0a991ca1cffd66423d0.db on ms-be2076.codfw.wmnet has errors: row 5310349 missing from index ix_object_deleted_name wrong # of entries in index ix_object_deleted_name
- wikipedia-commons-local-thumb.d3
/srv/swift-storage/accounts0/containers/35440/a6a/8a7070d66ad3b2c6df2c635eb41e7a6a/8a7070d66ad3b2c6df2c635eb41e7a6a.db on ms-be1079.eqiad.wmnet has errors: row 4349795 missing from index ix_object_deleted_name wrong # of entries in index ix_object_deleted_name /srv/swift-storage/accounts1/containers/35440/a6a/8a7070d66ad3b2c6df2c635eb41e7a6a/8a7070d66ad3b2c6df2c635eb41e7a6a.db on ms-be1085.eqiad.wmnet has errors: row 4234895 missing from index ix_object_deleted_name wrong # of entries in index ix_object_deleted_name /srv/swift-storage/accounts1/containers/35440/a6a/8a7070d66ad3b2c6df2c635eb41e7a6a/8a7070d66ad3b2c6df2c635eb41e7a6a.db on ms-be1078.eqiad.wmnet has errors: row 4349795 missing from index ix_object_deleted_name wrong # of entries in index ix_object_deleted_name
- wikipedia-commons-local-thumb.ea
/srv/swift-storage/accounts0/containers/18674/f11/48f22e340a0b1d0f141c98e1cc4faf11/48f22e340a0b1d0f141c98e1cc4faf11.db on ms-be1078.eqiad.wmnet has errors: row 6128861 missing from index ix_object_deleted_name wrong # of entries in index ix_object_deleted_name /srv/swift-storage/accounts0/containers/18674/f11/48f22e340a0b1d0f141c98e1cc4faf11/48f22e340a0b1d0f141c98e1cc4faf11.db on ms-be1080.eqiad.wmnet has errors: row 6128861 missing from index ix_object_deleted_name wrong # of entries in index ix_object_deleted_name
FWIW, I would advise against using Swift (the protocol) if you want to move storage to Ceph, and rather use only S3 against Ceph. I don't think the Swift protocol support in radosgw gets a lot of love these days, unlike the S3 protocol which is getting actively developed.
Mon, Jun 23
FWIW, after today's incident we ended up with both swift-rw resources depooled:
mvernon@cumin2002:~$ confctl --object-type discovery select 'dnsdisc=swift.*' get
{"eqiad": {"pooled": false, "references": [], "ttl": 300}, "tags": "dnsdisc=swift-rw"}
{"codfw": {"pooled": false, "references": [], "ttl": 300}, "tags": "dnsdisc=swift-rw"}
{"codfw": {"pooled": true, "references": [], "ttl": 300}, "tags": "dnsdisc=swift"}
{"eqiad": {"pooled": true, "references": [], "ttl": 300}, "tags": "dnsdisc=swift"}
{"codfw": {"pooled": true, "references": [], "ttl": 300}, "tags": "dnsdisc=swift-ro"}
{"eqiad": {"pooled": true, "references": [], "ttl": 300}, "tags": "dnsdisc=swift-ro"}Thu, Jun 19
Yes, I'd expect the AccessDenied when trying to list the mcv21test bucket - only the 1-month-old_kittens_32.jpg object within it is globally-readable, not the bucket itself.
@elukey You could delete each container in parallel (in a separate tmux/screen window or whatever)? That would get us some more parallelism, but less aggressively than bumping the concurrency in swift delete. I'm relaxed about it taking a few weeks, but can see that having it all under way rather than waiting weeks and then starting the next container going would be annoying...
@Jclark-ctr I've just put in T397414 to decommission (amongst others) thanos-be1003 in C4 and thanos-be1004 in D7; could the last two ms backends from this ticket go into those spaces, do you think, please?
@elukey yes, that should be fine - ping me when the deletion is done, please?
I think the answer is "you need a proxy if you want to do it externally", but it's a little "It Depends". If you set a suitable ACL, you can download objects from anywhere internal, for example:
Wed, Jun 18
[this is blocking ongoing load/drain operations for the eqiad ms cluster]
Silly question while I'm here - do you need 2 buckets, each of which ends up replicated cross-DC? [The answer might be yes, but it seemed worth checking]
FWIW, I use sudo bash ; . /etc/swift/accountfile.env, but yes. Those commands will take some time to run I expect, so run them in a screen/tmux. The thanos-swift cluster is one cluster stretched between both DCs, so those deletes will remove content from both DCs.
Jun 17 2025
@Jhancock.wm thanos-be2006 Just Worked; thanos-be2007 still had a disk with an old EFI setup on it - I knew which one from reading the run-puppet-agent output (which complained about /dev/disk/by-path/pci-0000:50:00.0-scsi-0:2:11:0-part1; blkid told me it was a vfat EFI partition and I checked it wasn't part of the installation. I think used ls -l to tell me it was /dev/sdl1 and then wiped both that and /dev/sdl with wipefs -a /dev/sdl1 ; wipefs -a /dev/sdl and then the re-image worked.
Jun 16 2025
...so ideally, delete all the old data and then you can just go ahead (and maybe let's make a rough plan for "when to delete v002"?) :)
To put a little more context on that:
root@thanos-fe1004:/home/mvernon# for b in $(swift list); do swift stat --lh "$b" | grep -E 'Container|Bytes' ; done
Container: tegola-swift-codfw-v002
Bytes: 82G
Container: tegola-swift-codfw-v003
Bytes: 0
Container: tegola-swift-container
Bytes: 59G
Container: tegola-swift-eqiad-v002
Bytes: 87G
Container: tegola-swift-eqiad-v003
Bytes: 0
Container: tegola-swift-fallback
Bytes: 38G
Container: tegola-swift-new
Bytes: 46G
Container: tegola-swift-staging-container
Bytes: 32G
Container: tegola-swift-v001
Bytes: 71GQuick question: I'm concerned about the rather vague timeline for deleting tegola-swift-eqiad-v002 and tegola-swift-codfw-v002; it'd be easier to be relaxed about your proposal if there was a timeline for going back to current-ish usage rather than x2 usage...?
Thanks @Ladsgroup :)
I've had a look, and this system looks good to me know (right number of filesystems of the right size, puppet happy, swift-recon -r and swift-dispersion-report both good). Thanks to both @Jhancock.wm and @Eevans for fixing :)
Jun 6 2025
@Jhancock.wm I've had a look at the web-iDRAC, and the serial number I found above (9120A025F1QF) does correspond to the device in slot 14, which the web iDRAC describes as " Physical Disk 0:2:14 ", which is what I'd expect to be lit up by megacli -PDLocate -PhysDrv [32:14] -a0.
My colleagues in SRE applied some filtering to problematic traffic.
This should be resolved now.
Jun 4 2025
(i.e. the kernel thinks sdf got removed, not the bad sdr)
@Jhancock.wm I fear the wrong disk has gone:
Jun 4 16:22:52 ms-be2066 kernel: [31462174.362949] sd 0:2:5:0: [sdf] tag#606 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=4s Jun 4 16:22:52 ms-be2066 kernel: [31462174.362977] sd 0:2:5:0: [sdf] tag#606 CDB: Read(16) 88 00 00 00 00 00 13 0c 24 08 00 00 00 08 00 00 Jun 4 16:22:52 ms-be2066 kernel: [31462174.362985] blk_update_request: I/O error, dev sdf, sector 319562760 op 0x0:(READ) flags 0x1000 phys_seg 1 prio class 0 Jun 4 16:22:52 ms-be2066 kernel: [31462174.364433] sd 0:2:5:0: [sdf] tag#380 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=3s Jun 4 16:22:52 ms-be2066 kernel: [31462174.364462] sd 0:2:5:0: [sdf] tag#584 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=4s Jun 4 16:22:52 ms-be2066 kernel: [31462174.364486] sd 0:2:5:0: [sdf] tag#584 CDB: Read(16) 88 00 00 00 00 02 b3 30 5a 90 00 00 00 08 00 00 Jun 4 16:22:52 ms-be2066 kernel: [31462174.364495] blk_update_request: I/O error, dev sdf, sector 11596225168 op 0x0:(READ) flags 0x1000 phys_seg 1 prio class 0 Jun 4 16:22:52 ms-be2066 kernel: [31462174.364669] XFS (sdf1): metadata I/O error in "xfs_da_read_buf+0xd9/0x130 [xfs]" at daddr 0x2b3305290 len 8 error 5 Jun 4 16:22:52 ms-be2066 kernel: [31462174.374145] XFS (sdf1): metadata I/O error in "xfs_da_read_buf+0xd9/0x130 [xfs]" at daddr 0x130c1c08 len 8 error 5 Jun 4 16:22:52 ms-be2066 kernel: [31462174.385240] sd 0:2:5:0: [sdf] tag#380 CDB: Read(16) 88 00 00 00 00 00 40 f4 99 a0 00 00 00 20 00 00 Jun 4 16:22:52 ms-be2066 kernel: [31462174.385242] blk_update_request: I/O error, dev sdf, sector 1089771936 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0 Jun 4 16:22:52 ms-be2066 kernel: [31462174.385411] XFS (sdf1): metadata I/O error in "xfs_imap_to_bp+0x61/0xb0 [xfs]" at daddr 0x40f491a0 len 32 error 5 Jun 4 16:22:55 ms-be2066 kernel: [31462176.777084] sd 0:2:5:0: [sdf] tag#580 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=5s Jun 4 16:22:55 ms-be2066 kernel: [31462176.777111] sd 0:2:5:0: [sdf] tag#580 CDB: Read(16) 88 00 00 00 00 00 8c d5 33 50 00 00 00 08 00 00 Jun 4 16:22:55 ms-be2066 kernel: [31462176.777120] blk_update_request: I/O error, dev sdf, sector 2362782544 op 0x0:(READ) flags 0x1000 phys_seg 1 prio class 0 Jun 4 16:22:55 ms-be2066 kernel: [31462176.778794] megaraid_sas 0000:18:00.0: scanning for scsi0... Jun 4 16:22:55 ms-be2066 kernel: [31462176.779347] sd 0:2:5:0: [sdf] tag#584 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=2s Jun 4 16:22:55 ms-be2066 kernel: [31462176.779353] sd 0:2:5:0: [sdf] tag#584 CDB: Read(16) 88 00 00 00 00 02 b3 30 5a 90 00 00 00 08 00 00 Jun 4 16:22:55 ms-be2066 kernel: [31462176.779359] blk_update_request: I/O error, dev sdf, sector 11596225168 op 0x0:(READ) flags 0x1000 phys_seg 1 prio class 0 Jun 4 16:22:55 ms-be2066 kernel: [31462176.779496] XFS (sdf1): metadata I/O error in "xfs_da_read_buf+0xd9/0x130 [xfs]" at daddr 0x2b3305290 len 8 error 5 Jun 4 16:22:55 ms-be2066 kernel: [31462176.782267] sd 0:2:5:0: [sdf] tag#680 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=0s Jun 4 16:22:55 ms-be2066 kernel: [31462176.782273] sd 0:2:5:0: [sdf] tag#680 CDB: Read(16) 88 00 00 00 00 02 ce fe 3b 68 00 00 00 08 00 00 Jun 4 16:22:55 ms-be2066 kernel: [31462176.782278] blk_update_request: I/O error, dev sdf, sector 12062702440 op 0x0:(READ) flags 0x1000 phys_seg 1 prio class 0 Jun 4 16:22:55 ms-be2066 kernel: [31462176.782454] XFS (sdf1): metadata I/O error in "xfs_da_read_buf+0xd9/0x130 [xfs]" at daddr 0x2cefe3368 len 8 error 5 Jun 4 16:22:55 ms-be2066 kernel: [31462176.782541] sd 0:2:5:0: [sdf] tag#674 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=0s Jun 4 16:22:55 ms-be2066 kernel: [31462176.782545] sd 0:2:5:0: [sdf] tag#674 CDB: Read(16) 88 00 00 00 00 02 ce fe 3b 68 00 00 00 08 00 00 Jun 4 16:22:55 ms-be2066 kernel: [31462176.782550] blk_update_request: I/O error, dev sdf, sector 12062702440 op 0x0:(READ) flags 0x1000 phys_seg 1 prio class 0 Jun 4 16:22:55 ms-be2066 kernel: [31462176.782641] XFS (sdf1): metadata I/O error in "xfs_da_read_buf+0xd9/0x130 [xfs]" at daddr 0x2cefe3368 len 8 error 5 Jun 4 16:22:55 ms-be2066 kernel: [31462176.782771] sd 0:2:5:0: [sdf] tag#681 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=0s Jun 4 16:22:55 ms-be2066 kernel: [31462176.782775] sd 0:2:5:0: [sdf] tag#681 CDB: Read(16) 88 00 00 00 00 02 ce fe 3b 68 00 00 00 08 00 00 Jun 4 16:22:55 ms-be2066 kernel: [31462176.782779] blk_update_request: I/O error, dev sdf, sector 12062702440 op 0x0:(READ) flags 0x1000 phys_seg 1 prio class 0 Jun 4 16:22:55 ms-be2066 kernel: [31462176.782863] XFS (sdf1): metadata I/O error in "xfs_da_read_buf+0xd9/0x130 [xfs]" at daddr 0x2cefe3368 len 8 error 5 Jun 4 16:22:55 ms-be2066 kernel: [31462176.785294] sd 0:2:5:0: [sdf] tag#296 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=0s Jun 4 16:22:55 ms-be2066 kernel: [31462176.785298] sd 0:2:5:0: [sdf] tag#296 CDB: Read(16) 88 00 00 00 00 01 b6 3e 8e e8 00 00 00 20 00 00 Jun 4 16:22:55 ms-be2066 kernel: [31462176.785300] blk_update_request: I/O error, dev sdf, sector 7352520424 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0 Jun 4 16:22:55 ms-be2066 kernel: [31462176.785351] XFS (sdf1): metadata I/O error in "xfs_imap_to_bp+0x61/0xb0 [xfs]" at daddr 0x1b63e86e8 len 32 error 5 Jun 4 16:22:55 ms-be2066 kernel: [31462176.786029] sd 0:2:5:0: [sdf] tag#692 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=0s Jun 4 16:22:55 ms-be2066 kernel: [31462176.786035] sd 0:2:5:0: [sdf] tag#692 CDB: Read(16) 88 00 00 00 00 03 8e a0 7c 98 00 00 00 08 00 00 Jun 4 16:22:55 ms-be2066 kernel: [31462176.786041] blk_update_request: I/O error, dev sdf, sector 15277784216 op 0x0:(READ) flags 0x1000 phys_seg 1 prio class 0 Jun 4 16:22:55 ms-be2066 kernel: [31462176.786137] XFS (sdf1): metadata I/O error in "xfs_da_read_buf+0xd9/0x130 [xfs]" at daddr 0x38ea07498 len 8 error 5 Jun 4 16:22:55 ms-be2066 kernel: [31462176.786289] XFS (sdf1): metadata I/O error in "xfs_da_read_buf+0xd9/0x130 [xfs]" at daddr 0x38ea07498 len 8 error 5 Jun 4 16:22:55 ms-be2066 kernel: [31462176.782550] blk_update_request: I/O error, dev sdf, sector 12062702440 op 0x0:(READ) flags 0x1000 phys_seg 1 prio class 0 Jun 4 16:22:55 ms-be2066 kernel: [31462176.782641] XFS (sdf1): metadata I/O error in "xfs_da_read_buf+0xd9/0x130 [xfs]" at daddr 0x2cefe3368 len 8 error 5 Jun 4 16:22:55 ms-be2066 kernel: [31462176.782771] sd 0:2:5:0: [sdf] tag#681 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=0s Jun 4 16:22:55 ms-be2066 kernel: [31462176.782775] sd 0:2:5:0: [sdf] tag#681 CDB: Read(16) 88 00 00 00 00 02 ce fe 3b 68 00 00 00 08 00 00 Jun 4 16:22:55 ms-be2066 kernel: [31462176.782779] blk_update_request: I/O error, dev sdf, sector 12062702440 op 0x0:(READ) flags 0x1000 phys_seg 1 prio class 0 Jun 4 16:22:55 ms-be2066 kernel: [31462176.782863] XFS (sdf1): metadata I/O error in "xfs_da_read_buf+0xd9/0x130 [xfs]" at daddr 0x2cefe3368 len 8 error 5 Jun 4 16:22:55 ms-be2066 kernel: [31462176.785294] sd 0:2:5:0: [sdf] tag#296 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=0s Jun 4 16:22:55 ms-be2066 kernel: [31462176.785298] sd 0:2:5:0: [sdf] tag#296 CDB: Read(16) 88 00 00 00 00 01 b6 3e 8e e8 00 00 00 20 00 00 Jun 4 16:22:55 ms-be2066 kernel: [31462176.785300] blk_update_request: I/O error, dev sdf, sector 7352520424 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0 Jun 4 16:22:55 ms-be2066 kernel: [31462176.785351] XFS (sdf1): metadata I/O error in "xfs_imap_to_bp+0x61/0xb0 [xfs]" at daddr 0x1b63e86e8 len 32 error 5 Jun 4 16:22:55 ms-be2066 kernel: [31462176.786029] sd 0:2:5:0: [sdf] tag#692 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=0s Jun 4 16:22:55 ms-be2066 kernel: [31462176.786035] sd 0:2:5:0: [sdf] tag#692 CDB: Read(16) 88 00 00 00 00 03 8e a0 7c 98 00 00 00 08 00 00 Jun 4 16:22:55 ms-be2066 kernel: [31462176.786041] blk_update_request: I/O error, dev sdf, sector 15277784216 op 0x0:(READ) flags 0x1000 phys_seg 1 prio class 0 Jun 4 16:22:55 ms-be2066 kernel: [31462176.786137] XFS (sdf1): metadata I/O error in "xfs_da_read_buf+0xd9/0x130 [xfs]" at daddr 0x38ea07498 len 8 error 5 Jun 4 16:22:55 ms-be2066 kernel: [31462176.786289] XFS (sdf1): metadata I/O error in "xfs_da_read_buf+0xd9/0x130 [xfs]" at daddr 0x38ea07498 len 8 error 5 Jun 4 16:22:55 ms-be2066 kernel: [31462176.788845] XFS (sdf1): log I/O error -5 Jun 4 16:22:55 ms-be2066 kernel: [31462176.799732] sd 0:2:5:0: SCSI device is removed Jun 4 16:22:55 ms-be2066 kernel: [31462176.810273] XFS (sdf1): xfs_do_force_shutdown(0x2) called from line 1211 of file fs/xfs/xfs_log.c. Return address = 00000000582ccb94 Jun 4 16:22:55 ms-be2066 kernel: [31462176.934503] XFS (sdf1): Log I/O Error Detected. Shutting down filesystem Jun 4 16:22:55 ms-be2066 kernel: [31462176.941482] XFS (sdf1): Please unmount the filesystem and rectify the problem(s) Jun 4 16:23:01 ms-be2066 kernel: [31462182.621563] XFS (sdf1): Unmounting Filesystem Jun 4 16:23:01 ms-be2066 kernel: [31462182.631276] megaraid_sas 0000:18:00.0: 4339 (802368937s/0x0021/FATAL) - Controller cache pinned for missing or offline VD 05/3 Jun 4 16:23:01 ms-be2066 kernel: [31462182.631339] megaraid_sas 0000:18:00.0: 4340 (802368937s/0x0001/FATAL) - VD 05/3 is now OFFLINE
Just to record what we talked about on IRC - the plan is for now to put two nodes into A (one each in A2 and A7), and at the same time drain ms-be1061 (in B2) and ms-be1062 (in C2) with a view to putting the remaining two nodes there if no other space in A-D becomes available beforehand.
This is blocking other work on ms-codfw at the moment.
Jun 3 2025
Thanks :)
Looks like you've just finished the codfw 3x ones, so I looked:
wikipedia-commons-local-thumb.30 838,472 objects 96,830,126,986 bytes
wikipedia-commons-local-thumb.3f 836,193 objects 95,360,984,940 bytes
I'm not inclined to move this right now, I think - thanos-swift already gives you S3 protocol support and cross-DC replication, and a very large number of small objects is not a super-great fit to apus right now (we don't have solid-state storage for bucket indexes and suchlike; and our experience with gitlab is that cross-DC replication is done per-object, so lots of small objects is slow compared to fewer larger ones).
@Jhancock.wm @Jclark-ctr I've now re-imaged thanos-be2008 and thanos-be2009 OK. The problem in both cases was that there was one of the hdds with an existing partition on it - it had a single vfat partition with EFI on it (and subdirectories Boot Dell Microsoft PEBoot); the puppet that sets up swift (and also thanos) disks won't over-write existing data, so gets stuck. I wiped the partition and drive and then all was good.
@Jhancock.wm thanks for this; I tried running puppet on thanos-be2009 and the problem was that /dev/sdi1 had an EFI partition on it (presumably either because it was shipped like that, or maybe the result of a previous failed isntall); I wiped that and puppet now runs to completion, so I'll try another reimage.
Jun 2 2025
I've checked these objects in swift, and they are both present and correct in both clusters (I checked with swift stat wikipedia-commons-local-public.65 6/65/Басков_переулок_19_СПб_01.jpg and swift stat wikipedia-commons-local-public.89 8/89/Басков_переулок_19_СПб_02.jpg respectively), with timestamps of 07:07 UTC this morning.
I think the "check the file is in a consistent (presumed-to-be-absent) state" operation is intentional, and probably replicated across other file changing call paths; not least because we get tickets sometimes when these checks fail...
May 28 2025
Just apropos the sizes, I took one at random (wikipedia-commons-local-thumb.f9), and whilst eqiad is bigger, it's not a lot bigger:
eqiad: 9,285,690 objects 1,307,136,946,396 bytes
codfw: 7,434,712 objects 1,027,498,001,890 bytes
@elukey OK, I've set that up for you. Quota is 3T, but can be adjusted as needed - it's really there so we can keep track of how much apus storage we've promised to people.
May 27 2025
Yeah, the apus cluster isn't ideal for buckets with a very large number of objects in (if we wanted to start aiming to support such a use case, we'd want to use some SSD/NVME specifically for bucket indexes, which I think would involve some hardware hassle), so if it's straightforward to keep the artifacts bucket to a smaller number that'd be nice.
May 23 2025
I had a look in the swift logs for the associated item (per logstash, https://ms-fe.svc.codfw.wmnet/v1/AUTH_mw/wikipedia-commons-local-temp.85/8/85/1bszz9c7w23k.h5b3mo.6438344.webm.5), and get exactly one hit, in eqiad:
May 21 16:51:30 ms-fe1013 proxy-server: 10.67.188.58 10.64.48.149 21/May/2025/16/51/30 PUT /v1/AUTH_mw/wikipedia-commons-local-temp.85/8/85/1bszz9c7w23k.h5b3mo.6438344.webm.5 HTTP/1.0 201 - wikimedia/multi-http-client%20v1.1 AUTH_tk3261723f1... 16777216 - d5807b8ad0ae1b0af9f9cf957a42740c txd4becbe641444c8ba1f63-00682e0492 - 0.1807 - - 1747846290.054478645 1747846290.235161781 0
...so that request took 0.1807 seconds in eqiad, and was of 16,777,216 bytes.
@Ladsgroup can you let me know when one of the current batch has finished, please? Now we've done the thumbnail defrag stuff, I'd like to re-asses (for the purposes of wondering about a cache system) how big a freshly-deleted thumb container is (and how it grows).
A few TB of quota shouldn't be a problem; how many objects per bucket are you looking at? We get better performance out of fewer larger objects.
@Jclark-ctr @wiki_willy any idea when that might be or if there's anywhere else these servers could go? Once they're installed I'll be able to drain the nodes they're replacing (but that process typically takes ~3 weeks) and they can be decommissioned, but I'd rather avoid having to drain them first (partly because of the delay, and also because of the capacity reduction that would involve)...
@Jclark-ctr the problem (at least on thanos-be1006 where I started) is that the disks aren't visible to the operating system, because they're "Ready" not non-RAID, and indeed the OS looks to have been installed on virtual disks not non-RAID disks. These systems are all meant to be set up as JBOD...
May 22 2025
Ah, the bucket is gone from eqiad, but codfw is still catching up:
root@moss-be2001:/# radosgw-admin bucket sync status --bucket=gitlab-artifacts
realm 5d3dbc7a-7bbf-4412-a33b-124dfd79c774 (apus)
zonegroup 19f169b0-5e44-4086-9e1b-0df871fbea50 (apus_zg)
zone acc58620-9fac-476b-aee3-0f640100a3bb (codfw)
bucket :gitlab-artifacts[64f0dd71-48bf-45aa-9741-69a51c083556.75705.1])
current time 2025-05-22T09:33:40ZMay 21 2025
@Jelto both buckets deleted.
May 20 2025
May 17 2025
May 16 2025
@Jclark-ctr the only BIOS/iDRAC changes I made were to set all the hdds to be non-RAID. Everything else was fixing puppet/preseed configs.
@Jhancock.wm we had some fun with the eqiad equivalent system, but did get it properly installed (and the preseed done such that this system should also preseed OK).
@Jclark-ctr system imaged OK now. I don't know if you have more you want to do before closing this ticket out?
Thanks for this! I was able to get in via install_console, and have a look. None of the hdds were available - I had to boot into BIOS and convert them all to non-RAID from there (for some reason attempting this via the iDRAC doesn't work).
May 15 2025
Oh, right, yes, we want the OS on that (which I thought was going to be presented to the OS as a single device, doing RAID-1 in hardware), sorry.
@Jclark-ctr EFI booting is fine (I thought I'd said as much on a previous ticket, but may have missed it); I don't want the OS on the NVME drive, the OS should go on the SSDs on the boss card.
May 14 2025
@Jclark-ctr the boss card should be left as RAID 1, thank you (but all the other drives should be JBOD).
May 13 2025
[A brief aside: data-persistence sometimes need to use this script for restoring images we had to fish out of backups (or one of the ms clusters if an image was only uploaded to one); we could "just" upload the image to ms directly with the swift CLI tool, but have avoided doing so in the past because it's not clear what (if any) metadata (either in swift or in MW) would also need to be updated for this to work.]
@VRiley-WMF apologies, but any chance you can get thanos-fe1007 to at least PXE-boot OK, please? I don't think I can usefully make progress here.
May 12 2025
thanos-fe1007 looks like it's not even trying to PXE at the moment, so maybe the 10g card needs setting up to PXE on this system too?