lkml.org 
[lkml]   [2008]   [Nov]   [1]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: FS corruption after I/O errors
    Vladislav Bolkhovitin wrote:
    > Nick Piggin wrote:
    >> On Wednesday 29 October 2008 06:38, Vladislav Bolkhovitin wrote:
    >>> Nick Piggin wrote:
    >>>> On Saturday 25 October 2008 03:10, Vladislav Bolkhovitin wrote:
    >>>>> Hi,
    >>>>>
    >>>>> During recent debugging session of my SCSI target SCST
    >>>>> (http://scst.sf.net) I noticed many
    >>>>>
    >>>>> WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()
    >>>>>
    >>>>> messages in kernel log on the initiator. I attached the full log of
    >>>>> several of them.
    >>>>>
    >>>>> My target was buggy and I was working on fixing it, but I suppose Linux
    >>>>> should handle such failures more gracefully. In all the cases the target
    >>>>> had one type of failure: it "ate" a SCSI command and never returned
    >>>>> result of it.
    >>>> Right. This is one of the warnings I see in my fault-injection testing.
    >>>> It is fixed by my patch to clean up and improve the page and buffer
    >>>> error handling in the vm/fs.
    >>> Can you specify which patch you referring? Is it in 2.6.27?
    >> It's just an RFC at the moment which I posted to fsdevel. Not in 2.6.27.
    >
    > I see. I'm looking forward to see it in 2.6.28 or .29. This is really a
    > needed work.
    >
    > BTW, have you even seen in your fault-injection testing that after
    > receiving a failure from a SCSI device during heavy load ext3 file
    > system mounted on it gets corrupted and journal replay on remount
    > doesn't repair it, only manual e2fsck helps? I've many times seen that,
    > including cases when the target was remaining up and fully functional.
    > See, e.g., "MOANING MODE ON" part in
    > http://marc.info/?l=linux-scsi&m=121932252324432&w=2. I haven't checked
    > that case since then, although I see such corruptions quite often. But
    > in all them I can't so clearly say that it isn't a target's failure.

    I've just checked it with 2.6.27. The situation greatly improved and
    dbench was able to complete several runs under constant TASK_ABORTED
    "bombarding" (TASK RESET task management commands using "sg_reset -b"
    each 31 seconds from another "connection" to that device via qla2xxx
    initiator driver. You can see those resets in the attached log). But
    when then I unmounted the affected partition, e2fsck found errors on it.
    See attachments for details. The target all the times was fine and
    completely healthy.


    [root@inif w]# e2fsck -f -y /dev/sdc
    e2fsck 1.38 (30-Jun-2005)
    Pass 1: Checking inodes, blocks, and sizes
    Pass 2: Checking directory structure
    Pass 3: Checking directory connectivity
    Pass 4: Checking reference counts
    Pass 5: Checking group summary information
    /dev/sdc: 69413/640000 files (0.2% non-contiguous), 316508/1280000 blocks
    [root@inif w]# mount -t ext3 /dev/sdc /mnt
    [root@inif dbench-mod]# while true; do ./dbench 50; done
    50 clients started
    .........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+...................................+..........................................................................+.................................................................+.+.................................................+.....................................+................................................................................................................++...........................++....................++..........................+.......................+......................................+.................................................................................................+.........................++++.+...+................................+...++.+........++..........+.................++++......................+++....+.++.....+.....+.....................+......+++.++++**************************************************
    Throughput 27.5109 MB/sec (NB=34.3887 MB/sec 275.109 MBit/sec)
    Average latency 7.166755 ms
    Maximum latency 22972.843000 ms
    Latency > 2 sec. 0.091621 %
    Latency > 10 sec. 0.016009 %
    50 clients started
    ............................................................................................................write failed on handle 4290
    ..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+............................................................................................................................................................................................................................................................................................................................................................................................................................++.........+......................................write failed on handle 4279
    ........................+.................................................+...................................................+...............+.........+...............................+.............++..............................++...........................+++........................+...+............+......+...+.+......+.....+...............++.....+....................+.....+..+++.+............+....+++......+.+...++.....+........+........+.+++++**************************************************
    Throughput 27.5921 MB/sec (NB=34.4902 MB/sec 275.921 MBit/sec)
    Average latency 7.097532 ms
    Maximum latency 20892.153000 ms
    Latency > 2 sec. 0.087833 %
    Latency > 10 sec. 0.016437 %
    50 clients started
    ....................................................................................write failed on handle 4290
    ..(4727) nb_stat: CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP size=260096 No such file or directory
    (4729) open CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP failed for handle 4328 (File exists)
    (4730) nb_close: handle 4328 was not open
    (4739) nb_stat: CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP size=260096 No such file or directory
    (4743) nb_stat: CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP size=260096 No such file or directory
    (4745) open CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP failed for handle 4329 (File exists)
    (4751) open CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP failed for handle 4330 (File exists)
    (4752) nb_read: handle 4330 was not open size=24 ofs=2156
    (4754) nb_read: handle 4330 was not open size=276 ofs=2139
    (4755) nb_read: handle 4330 was not open size=14 ofs=2136
    (4757) nb_read: handle 4330 was not open size=7 ofs=2136
    (4759) nb_read: handle 4330 was not open size=12 ofs=2138
    (4761) nb_read: handle 4330 was not open size=8 ofs=2137
    (4763) nb_read: handle 4330 was not open size=15 ofs=2135
    (4765) nb_read: handle 4330 was not open size=16 ofs=0
    (4767) nb_read: handle 4330 was not open size=2048 ofs=16
    (4768) nb_read: handle 4330 was not open size=2048 ofs=2064
    (4769) nb_read: handle 4330 was not open size=2048 ofs=257
    (4770) nb_read: handle 4330 was not open size=2048 ofs=385
    (4771) nb_read: handle 4330 was not open size=2048 ofs=513
    (4772) nb_read: handle 4330 was not open size=2048 ofs=641
    (4773) nb_read: handle 4330 was not open size=2048 ofs=769
    (4774) nb_read: handle 4330 was not open size=2048 ofs=897
    (4775) nb_read: handle 4330 was not open size=2048 ofs=1025
    (4776) nb_read: handle 4330 was not open size=2048 ofs=1153
    (4777) nb_read: handle 4330 was not open size=2048 ofs=1281
    (4778) nb_read: handle 4330 was not open size=2048 ofs=1409
    (4779) nb_read: handle 4330 was not open size=408 ofs=1537
    ..(4899) nb_close: handle 4330 was not open
    (4964) nb_read: handle 4329 was not open size=1386 ofs=2157
    (4971) nb_read: handle 4329 was not open size=4096 ofs=2157
    (4972) nb_read: handle 4329 was not open size=4096 ofs=6253
    (4973) nb_read: handle 4329 was not open size=4096 ofs=10349
    (4974) nb_read: handle 4329 was not open size=4096 ofs=14445
    (4975) nb_read: handle 4329 was not open size=4096 ofs=18541
    (4976) nb_read: handle 4329 was not open size=256 ofs=22637
    (4977) nb_read: handle 4329 was not open size=4096 ofs=22893
    (4978) nb_read: handle 4329 was not open size=4096 ofs=26989
    (4979) nb_read: handle 4329 was not open size=4096 ofs=31085
    (4980) nb_read: handle 4329 was not open size=4096 ofs=35181
    (4981) nb_read: handle 4329 was not open size=4096 ofs=39277
    (4982) nb_read: handle 4329 was not open size=4096 ofs=43373
    (4983) nb_read: handle 4329 was not open size=4096 ofs=47469
    (4986) nb_read: handle 4329 was not open size=1024 ofs=2157
    (4987) nb_read: handle 4329 was not open size=4688 ofs=1667
    (4988) nb_read: handle 4329 was not open size=3516 ofs=6355
    (4989) nb_read: handle 4329 was not open size=16384 ofs=9871
    (4990) nb_read: handle 4329 was not open size=3771 ofs=26255
    (4991) nb_read: handle 4329 was not open size=2048 ofs=30026
    (4992) nb_read: handle 4329 was not open size=1024 ofs=32074
    (4993) nb_read: handle 4329 was not open size=6652 ofs=33098
    (4994) nb_read: handle 4329 was not open size=16384 ofs=39750
    (4995) nb_read: handle 4329 was not open size=10496 ofs=56134
    (4996) nb_read: handle 4329 was not open size=1784 ofs=66630
    (4997) nb_read: handle 4329 was not open size=90 ofs=68414
    (4998) nb_read: handle 4329 was not open size=273 ofs=68504
    (4999) nb_read: handle 4329 was not open size=1600 ofs=68777
    (5000) nb_read: handle 4329 was not open size=640 ofs=70377
    (5001) nb_read: handle 4329 was not open size=1280 ofs=71017
    (5002) nb_read: handle 4329 was not open size=8464 ofs=72297
    (5003) nb_read: handle 4329 was not open size=158 ofs=80761
    (5004) nb_read: handle 4329 was not open size=912 ofs=80919
    (5005) nb_read: handle 4329 was not open size=1980 ofs=81831
    (5006) nb_read: handle 4329 was not open size=2256 ofs=83811
    (5007) nb_read: handle 4329 was not open size=400 ofs=86067
    (5008) nb_read: handle 4329 was not open size=252 ofs=86467
    (5009) nb_read: handle 4329 was not open size=42 ofs=86719
    (5011) nb_read: handle 4329 was not open size=725 ofs=86761
    (5012) nb_read: handle 4329 was not open size=512 ofs=2009
    (5013) nb_read: handle 4329 was not open size=9 ofs=3895
    (5015) nb_read: handle 4329 was not open size=46 ofs=3781
    (5017) nb_read: handle 4329 was not open size=92 ofs=4059
    (5018) nb_read: handle 4329 was not open size=92 ofs=3781
    (5019) nb_read: handle 4329 was not open size=184 ofs=3986
    (5020) nb_read: handle 4329 was not open size=16384 ofs=3987
    (5021) nb_read: handle 4329 was not open size=16384 ofs=3902
    (5022) nb_read: handle 4329 was not open size=10240 ofs=3781
    (5023) nb_read: handle 4329 was not open size=16384 ofs=3821
    (5024) nb_read: handle 4329 was not open size=935 ofs=3891
    (5025) nb_read: handle 4329 was not open size=10240 ofs=4060
    (5026) nb_read: handle 4329 was not open size=184 ofs=4059
    (5027) nb_read: handle 4329 was not open size=184 ofs=256
    (5028) nb_read: handle 4329 was not open size=1026 ofs=256
    (5029) nb_read: handle 4329 was not open size=326 ofs=3901
    (5030) nb_read: handle 4329 was not open size=652 ofs=3899
    (5031) nb_read: handle 4329 was not open size=2 ofs=256
    (5033) nb_read: handle 4329 was not open size=4948 ofs=258
    (5034) nb_close: handle 4329 was not open
    ............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+...+......................................................................+..................................+.............................+.....................+++.........................................................................................................................+....................+...............+.....................................+......+......+.+...............................+.+....+.......................+.................+..................++....+.+..+..........++..+...........+.........+.........+..........................++++........+++++.............................+.++...........+..++++++**************************************************
    Throughput 27.8199 MB/sec (NB=34.7749 MB/sec 278.199 MBit/sec)
    Average latency 6.940010 ms
    Maximum latency 18993.291000 ms
    Latency > 2 sec. 0.081048 %
    Latency > 10 sec. 0.015581 %
    50 clients started
    ...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+.................................................................+....................................................................................................................................................+..................+...+.............................................................................................................................................................................................................+..................+.......................................................+................++.......+............+.....+...............................+..+...............++.+.................................+............+.....++..++.......................+.........+...............++...........................+++.........+++.+...............+....+...+.....+........+......++....++++++.++**************************************************
    Throughput 27.0223 MB/sec (NB=33.7779 MB/sec 270.223 MBit/sec)
    Average latency 6.739319 ms
    Maximum latency 18699.530000 ms
    Latency > 2 sec. 0.090834 %
    Latency > 10 sec. 0.013650 %
    50 clients started
    .........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+..............................................................................................................................+............................................................................+...................................................................................................................................................................+.+................................................................+.................................................++++...........................+..+....+..........++...........+....................+.....+.............................................+.............+................+...+..........+++....+....+............................+........+..+.....+....+...+........+........+++...............+++++..++++.++++**************************************************
    Throughput 27.2861 MB/sec (NB=34.1077 MB/sec 272.861 MBit/sec)
    Average latency 7.051013 ms
    Maximum latency 23801.490000 ms
    Latency > 2 sec. 0.091263 %
    Latency > 10 sec. 0.014722 %
    50 clients started
    .......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................^C
    [root@inif w]# umount /mnt
    [root@inif w]# e2fsck -f -y /dev/sdc
    e2fsck 1.38 (30-Jun-2005)
    Pass 1: Checking inodes, blocks, and sizes
    Pass 2: Checking directory structure
    Pass 3: Checking directory connectivity
    Pass 4: Checking reference counts
    Pass 5: Checking group summary information
    Block bitmap differences: -(1250434--1250435) -(1273360--1273370)
    Fix? yes

    Free blocks count wrong for group #38 (32253, counted=32266).
    Fix? yes

    Free blocks count wrong (1003250, counted=1003263).
    Fix? yes


    /dev/sdc: ***** FILE SYSTEM WAS MODIFIED *****
    /dev/sdc: 68842/640000 files (0.1% non-contiguous), 276737/1280000 blocks
    Nov 1 16:27:35 inif kernel: kjournald starting. Commit interval 5 seconds
    Nov 1 16:27:35 inif kernel: EXT3 FS on sdc, internal journal
    Nov 1 16:27:35 inif kernel: EXT3-fs: mounted filesystem with ordered data mode.
    Nov 1 16:29:32 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:29:32 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:29:32 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:29:37 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    Nov 1 16:30:18 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:30:18 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:30:18 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:30:19 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:30:19 inif kernel: end_request: I/O error, dev sdc, sector 9376352
    Nov 1 16:30:19 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:30:19 inif kernel: end_request: I/O error, dev sdc, sector 9995808
    Nov 1 16:30:19 inif kernel: EXT3-fs error (device sdc): ext3_find_entry: reading directory #608098 offset 0
    Nov 1 16:30:23 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    Nov 1 16:31:04 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:31:04 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:31:04 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:31:05 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:31:05 inif kernel: end_request: I/O error, dev sdc, sector 367432
    Nov 1 16:31:05 inif kernel: __ratelimit: 62 callbacks suppressed
    Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45929
    Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45930
    Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45931
    Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45932
    Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45933
    Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45934
    Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45935
    Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45936
    Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45937
    Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45938
    Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:31:09 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    Nov 1 16:31:22 inif kernel: JBD: Detected IO errors while flushing file data on sdc
    Nov 1 16:31:54 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:31:54 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:31:54 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:31:59 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    Nov 1 16:32:40 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:32:40 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:32:40 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:32:45 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    Nov 1 16:33:26 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:33:26 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:33:26 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:33:31 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    Nov 1 16:34:12 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:34:12 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:34:12 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:34:13 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:34:13 inif kernel: end_request: I/O error, dev sdc, sector 9621088
    Nov 1 16:34:13 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:34:13 inif kernel: end_request: I/O error, dev sdc, sector 10145456
    Nov 1 16:34:13 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:34:13 inif kernel: end_request: I/O error, dev sdc, sector 7095792
    Nov 1 16:34:17 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    Nov 1 16:34:58 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:34:58 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:34:58 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9530232
    Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9798712
    Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9798728
    Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9798736
    Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 10059240
    Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9798720
    Nov 1 16:35:03 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    Nov 1 16:35:44 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:35:44 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:35:44 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:35:49 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    Nov 1 16:36:30 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:36:30 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:36:30 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:36:31 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:36:31 inif kernel: end_request: I/O error, dev sdc, sector 10032752
    Nov 1 16:36:35 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    Nov 1 16:37:16 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:37:16 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:37:16 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8862288
    Nov 1 16:37:17 inif kernel: __ratelimit: 59 callbacks suppressed
    Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107786
    Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107787
    Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107788
    Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107789
    Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107790
    Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107791
    Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107792
    Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107793
    Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107794
    Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107795
    Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8892560
    Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8893456
    Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8894480
    Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8895512
    Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8896536
    Nov 1 16:37:21 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    Nov 1 16:37:29 inif kernel: JBD: Detected IO errors while flushing file data on sdc
    Nov 1 16:38:02 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:38:02 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:38:02 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9213944
    Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9280464
    Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9280472
    Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9280480
    Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9184784
    Nov 1 16:38:03 inif kernel: EXT3-fs error (device sdc): ext3_find_entry: reading directory #560038 offset 0
    Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 10159136
    Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 303616
    Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9586656
    Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9641352
    Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 10061776
    Nov 1 16:38:03 inif kernel: EXT3-fs error (device sdc): ext3_find_entry: reading directory #576172 offset 0
    Nov 1 16:38:07 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    Nov 1 16:38:48 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:38:48 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:38:48 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:38:53 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    Nov 1 16:39:34 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:39:34 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:39:34 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:39:39 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    Nov 1 16:40:20 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:40:20 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:40:20 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:40:25 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    Nov 1 16:41:06 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:41:06 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:41:06 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:41:11 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    Nov 1 16:41:52 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:41:52 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:41:52 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:41:53 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:41:53 inif kernel: end_request: I/O error, dev sdc, sector 647544
    Nov 1 16:41:53 inif kernel: __ratelimit: 639 callbacks suppressed
    Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80943
    Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80944
    Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80945
    Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80946
    Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80947
    Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80948
    Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80949
    Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80950
    Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80951
    Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80952
    Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:41:57 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    Nov 1 16:42:05 inif kernel: JBD: Detected IO errors while flushing file data on sdc
    Nov 1 16:42:38 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:42:38 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:42:38 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:42:43 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    Nov 1 16:43:24 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:43:24 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:43:24 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:43:25 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:43:25 inif kernel: end_request: I/O error, dev sdc, sector 478480
    Nov 1 16:43:25 inif kernel: __ratelimit: 118 callbacks suppressed
    Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59810
    Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59811
    Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59812
    Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59813
    Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59814
    Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59815
    Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59816
    Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59817
    Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59818
    Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59819
    Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:43:29 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    Nov 1 16:44:10 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:44:10 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:44:10 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:44:15 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    Nov 1 16:44:56 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:44:56 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:44:56 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:44:57 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:44:57 inif kernel: end_request: I/O error, dev sdc, sector 9353840
    Nov 1 16:44:57 inif kernel: __ratelimit: 44 callbacks suppressed
    Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169230
    Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169231
    Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169232
    Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169233
    Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169234
    Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169235
    Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169236
    Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169237
    Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169238
    Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169239
    Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:45:01 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    Nov 1 16:45:11 inif kernel: JBD: Detected IO errors while flushing file data on sdc
    Nov 1 16:45:42 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:45:42 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:45:42 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:45:47 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    Nov 1 16:46:28 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:46:28 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:46:28 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9226672
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9426536
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9832040
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9895976
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9896048
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9946768
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 279576
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 1048576
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 7097712
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9102152
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 10190752
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 10199216
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 295416
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9057504
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9115608
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9289416
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9529432
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9577376
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9672304
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9763984
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9789792
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9811504
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9826536
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9115656
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9624224
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9928144
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 10003472
    Nov 1 16:46:29 inif kernel: EXT3-fs error (device sdc): ext3_free_branches: Read failure, inode=517632, block=1250434
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 10186880
    Nov 1 16:46:29 inif kernel: EXT3-fs error (device sdc): ext3_free_branches: Read failure, inode=560474, block=1273360
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 332944
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 451768
    Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9208392
    Nov 1 16:46:29 inif kernel: EXT3-fs error (device sdc): ext3_find_entry: reading directory #560267 offset 0
    Nov 1 16:46:33 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    Nov 1 16:47:14 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:47:14 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:47:14 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:47:15 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:47:15 inif kernel: end_request: I/O error, dev sdc, sector 429128
    Nov 1 16:47:15 inif kernel: __ratelimit: 67 callbacks suppressed
    Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 53641
    Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:47:15 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:47:15 inif kernel: end_request: I/O error, dev sdc, sector 10138096
    Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267262
    Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267263
    Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267264
    Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267265
    Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267266
    Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267267
    Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267268
    Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267269
    Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267270
    Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:47:15 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:47:15 inif kernel: end_request: I/O error, dev sdc, sector 964064
    Nov 1 16:47:19 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    Nov 1 16:47:28 inif kernel: JBD: Detected IO errors while flushing file data on sdc
    Nov 1 16:47:33 inif kernel: JBD: Detected IO errors while flushing file data on sdc
    Nov 1 16:48:00 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:48:00 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:48:00 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:48:05 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    Nov 1 16:48:46 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:48:46 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:48:46 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:48:47 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:48:47 inif kernel: end_request: I/O error, dev sdc, sector 10082504
    Nov 1 16:48:47 inif kernel: __ratelimit: 164 callbacks suppressed
    Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260313
    Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260314
    Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260315
    Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260316
    Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260317
    Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260318
    Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260319
    Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260320
    Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260321
    Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260322
    Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:48:47 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:48:47 inif kernel: end_request: I/O error, dev sdc, sector 10086400
    Nov 1 16:48:51 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    Nov 1 16:49:00 inif kernel: JBD: Detected IO errors while flushing file data on sdc
    Nov 1 16:49:32 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:49:32 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:49:32 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:49:33 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:49:33 inif kernel: end_request: I/O error, dev sdc, sector 9777376
    Nov 1 16:49:33 inif kernel: __ratelimit: 132 callbacks suppressed
    Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222172
    Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222173
    Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222174
    Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222175
    Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222176
    Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222177
    Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222178
    Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222179
    Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222180
    Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222181
    Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:49:37 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    Nov 1 16:49:46 inif kernel: JBD: Detected IO errors while flushing file data on sdc
    Nov 1 16:50:18 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:50:18 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:50:18 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:50:19 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:50:19 inif kernel: end_request: I/O error, dev sdc, sector 9728192
    Nov 1 16:50:19 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:50:19 inif kernel: end_request: I/O error, dev sdc, sector 9350872
    Nov 1 16:50:23 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    Nov 1 16:51:04 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
    Nov 1 16:51:04 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
    Nov 1 16:51:04 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
    Nov 1 16:51:05 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:51:05 inif kernel: end_request: I/O error, dev sdc, sector 9709760
    Nov 1 16:51:05 inif kernel: __ratelimit: 54 callbacks suppressed
    Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213720
    Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213721
    Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213722
    Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213723
    Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213724
    Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213725
    Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213726
    Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213727
    Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213728
    Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213729
    Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
    Nov 1 16:51:05 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
    Nov 1 16:51:05 inif kernel: end_request: I/O error, dev sdc, sector 9709864
    Nov 1 16:51:09 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
    \
     
     \ /
      Last update: 2008-11-01 14:11    [W:0.089 / U:1.368 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site