Identify failed drive

zfs is reporting read and write errors:

  pool: pool0
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
    attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
    using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-9P
  scan: scrub in progress since Sun Sep 29 02:42:02 2019
    970G scanned out of 44.4T at 7.43M/s, (scan is slow, no estimated time)
    0B repaired, 2.14% done
config:

    NAME                        STATE     READ WRITE CKSUM
    pool0                       ONLINE       0     0     0
      raidz2-0                  ONLINE       0     0     0
        scsi-35000cca252119064  ONLINE       0     0     0
        scsi-35000cca2525e92c8  ONLINE       0     0     0
        scsi-35000cca2525e970c  ONLINE       0     0     0
        scsi-35000cca2525eb8bc  ONLINE      37   768     0
        scsi-35000cca2525ec8e8  ONLINE       0     0     0
        scsi-35000cca2525ed36c  ONLINE       0     0     0
        scsi-35000cca2525eda08  ONLINE       0     0     0
        scsi-35000cca2525ef9bc  ONLINE       0     0     0
        scsi-35000cca2525f4f6c  ONLINE       0     0     0
        scsi-35000cca2525fd754  ONLINE       0     0     0

dmesg can confirm these errors:

# dmesg -T | grep blk_update_request
[Sat Sep 28 11:52:11 2019] blk_update_request: I/O error, dev sdf, sector 949800376
[Sat Sep 28 11:52:11 2019] blk_update_request: I/O error, dev sdf, sector 949799600
[Sat Sep 28 11:52:11 2019] blk_update_request: I/O error, dev sdf, sector 949800120
[Sat Sep 28 11:52:11 2019] blk_update_request: I/O error, dev sdf, sector 949799088
[Sat Sep 28 11:52:11 2019] blk_update_request: I/O error, dev sdf, sector 949798792
[Sat Sep 28 11:52:11 2019] blk_update_request: I/O error, dev sdf, sector 949798808
[Sat Sep 28 11:52:11 2019] blk_update_request: I/O error, dev sdf, sector 949798800
[Sat Sep 28 11:52:11 2019] blk_update_request: I/O error, dev sdf, sector 949798688
[Sat Sep 28 11:52:11 2019] blk_update_request: I/O error, dev sdf, sector 949798824
[Sat Sep 28 11:52:11 2019] blk_update_request: I/O error, dev sdf, sector 949798656

Time to change disk. 1/12 disks is Faulty. The Problem I have no clue which one.

ledctl to the rescue:

apt install -y ledmon
ledctl failure=/dev/disk/by-id/scsi-35000cca2525eb8bc

Now I can safely pull the faulty disk.

---

Last update: March 22, 2021