Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DegradedArray on server3 #115

Open
clonm opened this issue Sep 27, 2018 · 1 comment
Open

DegradedArray on server3 #115

clonm opened this issue Sep 27, 2018 · 1 comment
Labels
broken

Comments

@clonm
Copy link
Member

clonm commented Sep 27, 2018

Looks like a disk failed. See also #114. Need to go through and figure out which serial number is the failed disk, and swap it out.
@clonm clonm added the broken label Sep 27, 2018
@clonm
Copy link
Member Author

clonm commented Oct 13, 2018

Output of lsblk -fs:

NAME      FSTYPE      LABEL     UUID                                   MOUNTPOINT
fd0                                                                    
sda1      ext4                  6cf195d0-3d61-44f1-adfe-32e2be0ef589   /
└─sda                                                                  
sda2      swap                  f9d85b58-993d-4f14-8b9b-a7473cc6e5ed   [SWAP]
└─sda                                                                  
sdg1                                                                   
└─sdg                                                                  
sr0                                                                    
vg0-srv   ext4                  234aaade-0a00-4034-94fd-327e4656de6a   /srv
├─md0     LVM2_member           a3cbng-LNOD-tFIN-b7Jk-7I4Q-AsBt-J8ALjQ 
│ ├─sdb1  linux_raid_ server3:0 6343cc3a-1818-08ad-6e15-c8d9b83e4c2f   
│ │ └─sdb                                                              
│ └─sdc1  linux_raid_ server3:0 6343cc3a-1818-08ad-6e15-c8d9b83e4c2f   
│   └─sdc                                                              
├─md1     LVM2_member           2QUdr1-zHAf-YdIZ-HnLs-Cd1e-FApB-XrYsZb 
│ ├─sdd1  linux_raid_ server3:1 2408dcd6-a135-1e73-4cb1-e6a2a64a2aa6   
│ │ └─sdd                                                              
│ └─sde1  linux_raid_ server3:1 2408dcd6-a135-1e73-4cb1-e6a2a64a2aa6   
│   └─sde                                                              
├─md2     LVM2_member           lft9mX-bY4A-0LVs-tBr4-ei9M-Nwb0-0rge10 
│ ├─sdf1  linux_raid_ server3:2 f4d64eb3-fda4-f669-3f53-ee91ddb8ebf9   
│ │ └─sdf                                                              
│ └─sdh1  linux_raid_ server3:2 f4d64eb3-fda4-f669-3f53-ee91ddb8ebf9   
│   └─sdh                                                              
├─md3     LVM2_member           RrgDwD-8TsR-UblH-Qvjf-U5M2-C2Up-3Kr7E2 
│ ├─sdi1  linux_raid_ server3:3 94cb4f84-8506-bfd2-233f-4c6710d70379   
│ │ └─sdi                                                              
│ └─sdk1  linux_raid_ server3:3 94cb4f84-8506-bfd2-233f-4c6710d70379   
│   └─sdk                                                              
├─md5     LVM2_member           HLKlVn-3wuK-yh2Y-P0Ux-hLvq-9Rz3-1EIUDr 
│ └─sdj1  linux_raid_ server3:5 a5a813fd-d87a-a785-02f8-8bfde370b59d   
│   └─sdj                                                              
├─md6     LVM2_member           dCLeqW-x55j-zf2h-3Vjx-sIV3-FqOC-KmBeK6 
│ ├─sdl1  linux_raid_ server3:6 8c1bcb52-e684-f0f5-aae0-28928b9144cb   
│ │ └─sdl                                                              
│ └─sdm1  linux_raid_ server3:6 8c1bcb52-e684-f0f5-aae0-28928b9144cb   
│   └─sdm                                                              
└─md7     LVM2_member           ryStBk-SRMn-v2fG-2bEr-NHev-uRTO-MccbS7 
  ├─sdn1  linux_raid_ server3:7 53c6b4d3-ecd7-4f8f-514d-8002f73c57be   
  │ └─sdn                                                              
  └─sdo1  linux_raid_ server3:7 53c6b4d3-ecd7-4f8f-514d-8002f73c57be   
    └─sdo

So, sdg is bad. Here's that part of lshw

              *-disk:11
                   description: SCSI Disk
                   product: 9650SE-16M DISK
                   vendor: AMCC
                   physical id: 0.5.0
                   bus info: scsi@2:0.5.0
                   logical name: /dev/sdg
                   version: 4.10
                   serial: 3QD0A7EC57C4970016E9
                   size: 698GiB (749GB)
                   capabilities: partitioned partitioned:dos
                   configuration: ansiversion=5 logicalsectorsize=512 sectorsize=512 signature=00078bb4
                 *-volume
                      description: Linux raid autodetect partition
                      physical id: 1
                      bus info: scsi@2:0.5.0,1
                      logical name: /dev/sdg1
                      capacity: 698GiB
                      capabilities: primary multi

so it's disk 11. hence, smartctl -a -d 3ware,11 /dev/twa0:

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD30EFRX-68EUZN0
Serial Number:    WD-WMC4N0228885
LU WWN Device Id: 5 0014ee 0ae542bbf
Firmware Version: 80.00A80
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sat Oct 13 12:02:14 2018 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (41940) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 420) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x703d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   238   177   021    Pre-fail  Always       -       3100
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       53
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   057   057   000    Old_age   Always       -       31693
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       53
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       52
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       0
194 Temperature_Celsius     0x0022   119   109   000    Old_age   Always       -       31
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      4760         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
broken
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@clonm