lkml.org 
[lkml]   [2008]   [Nov]   [1]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: FS corruption after I/O errors
Vladislav Bolkhovitin wrote:
> Nick Piggin wrote:
>> On Wednesday 29 October 2008 06:38, Vladislav Bolkhovitin wrote:
>>> Nick Piggin wrote:
>>>> On Saturday 25 October 2008 03:10, Vladislav Bolkhovitin wrote:
>>>>> Hi,
>>>>>
>>>>> During recent debugging session of my SCSI target SCST
>>>>> (http://scst.sf.net) I noticed many
>>>>>
>>>>> WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()
>>>>>
>>>>> messages in kernel log on the initiator. I attached the full log of
>>>>> several of them.
>>>>>
>>>>> My target was buggy and I was working on fixing it, but I suppose Linux
>>>>> should handle such failures more gracefully. In all the cases the target
>>>>> had one type of failure: it "ate" a SCSI command and never returned
>>>>> result of it.
>>>> Right. This is one of the warnings I see in my fault-injection testing.
>>>> It is fixed by my patch to clean up and improve the page and buffer
>>>> error handling in the vm/fs.
>>> Can you specify which patch you referring? Is it in 2.6.27?
>> It's just an RFC at the moment which I posted to fsdevel. Not in 2.6.27.
>
> I see. I'm looking forward to see it in 2.6.28 or .29. This is really a
> needed work.
>
> BTW, have you even seen in your fault-injection testing that after
> receiving a failure from a SCSI device during heavy load ext3 file
> system mounted on it gets corrupted and journal replay on remount
> doesn't repair it, only manual e2fsck helps? I've many times seen that,
> including cases when the target was remaining up and fully functional.
> See, e.g., "MOANING MODE ON" part in
> http://marc.info/?l=linux-scsi&m=121932252324432&w=2. I haven't checked
> that case since then, although I see such corruptions quite often. But
> in all them I can't so clearly say that it isn't a target's failure.

I've just checked it with 2.6.27. The situation greatly improved and
dbench was able to complete several runs under constant TASK_ABORTED
"bombarding" (TASK RESET task management commands using "sg_reset -b"
each 31 seconds from another "connection" to that device via qla2xxx
initiator driver. You can see those resets in the attached log). But
when then I unmounted the affected partition, e2fsck found errors on it.
See attachments for details. The target all the times was fine and
completely healthy.

[root@inif w]# e2fsck -f -y /dev/sdc
e2fsck 1.38 (30-Jun-2005)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sdc: 69413/640000 files (0.2% non-contiguous), 316508/1280000 blocks
[root@inif w]# mount -t ext3 /dev/sdc /mnt
[root@inif dbench-mod]# while true; do ./dbench 50; done
50 clients started
.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+...................................+..........................................................................+.................................................................+.+.................................................+.....................................+................................................................................................................++...........................++....................++..........................+.......................+......................................+.................................................................................................+.........................++++.+...+................................+...++.+........++..........+.................++++......................+++....+.++.....+.....+.....................+......+++.++++**************************************************
Throughput 27.5109 MB/sec (NB=34.3887 MB/sec 275.109 MBit/sec)
Average latency 7.166755 ms
Maximum latency 22972.843000 ms
Latency > 2 sec. 0.091621 %
Latency > 10 sec. 0.016009 %
50 clients started
............................................................................................................write failed on handle 4290
..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+............................................................................................................................................................................................................................................................................................................................................................................................................................++.........+......................................write failed on handle 4279
........................+.................................................+...................................................+...............+.........+...............................+.............++..............................++...........................+++........................+...+............+......+...+.+......+.....+...............++.....+....................+.....+..+++.+............+....+++......+.+...++.....+........+........+.+++++**************************************************
Throughput 27.5921 MB/sec (NB=34.4902 MB/sec 275.921 MBit/sec)
Average latency 7.097532 ms
Maximum latency 20892.153000 ms
Latency > 2 sec. 0.087833 %
Latency > 10 sec. 0.016437 %
50 clients started
....................................................................................write failed on handle 4290
..(4727) nb_stat: CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP size=260096 No such file or directory
(4729) open CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP failed for handle 4328 (File exists)
(4730) nb_close: handle 4328 was not open
(4739) nb_stat: CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP size=260096 No such file or directory
(4743) nb_stat: CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP size=260096 No such file or directory
(4745) open CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP failed for handle 4329 (File exists)
(4751) open CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP failed for handle 4330 (File exists)
(4752) nb_read: handle 4330 was not open size=24 ofs=2156
(4754) nb_read: handle 4330 was not open size=276 ofs=2139
(4755) nb_read: handle 4330 was not open size=14 ofs=2136
(4757) nb_read: handle 4330 was not open size=7 ofs=2136
(4759) nb_read: handle 4330 was not open size=12 ofs=2138
(4761) nb_read: handle 4330 was not open size=8 ofs=2137
(4763) nb_read: handle 4330 was not open size=15 ofs=2135
(4765) nb_read: handle 4330 was not open size=16 ofs=0
(4767) nb_read: handle 4330 was not open size=2048 ofs=16
(4768) nb_read: handle 4330 was not open size=2048 ofs=2064
(4769) nb_read: handle 4330 was not open size=2048 ofs=257
(4770) nb_read: handle 4330 was not open size=2048 ofs=385
(4771) nb_read: handle 4330 was not open size=2048 ofs=513
(4772) nb_read: handle 4330 was not open size=2048 ofs=641
(4773) nb_read: handle 4330 was not open size=2048 ofs=769
(4774) nb_read: handle 4330 was not open size=2048 ofs=897
(4775) nb_read: handle 4330 was not open size=2048 ofs=1025
(4776) nb_read: handle 4330 was not open size=2048 ofs=1153
(4777) nb_read: handle 4330 was not open size=2048 ofs=1281
(4778) nb_read: handle 4330 was not open size=2048 ofs=1409
(4779) nb_read: handle 4330 was not open size=408 ofs=1537
..(4899) nb_close: handle 4330 was not open
(4964) nb_read: handle 4329 was not open size=1386 ofs=2157
(4971) nb_read: handle 4329 was not open size=4096 ofs=2157
(4972) nb_read: handle 4329 was not open size=4096 ofs=6253
(4973) nb_read: handle 4329 was not open size=4096 ofs=10349
(4974) nb_read: handle 4329 was not open size=4096 ofs=14445
(4975) nb_read: handle 4329 was not open size=4096 ofs=18541
(4976) nb_read: handle 4329 was not open size=256 ofs=22637
(4977) nb_read: handle 4329 was not open size=4096 ofs=22893
(4978) nb_read: handle 4329 was not open size=4096 ofs=26989
(4979) nb_read: handle 4329 was not open size=4096 ofs=31085
(4980) nb_read: handle 4329 was not open size=4096 ofs=35181
(4981) nb_read: handle 4329 was not open size=4096 ofs=39277
(4982) nb_read: handle 4329 was not open size=4096 ofs=43373
(4983) nb_read: handle 4329 was not open size=4096 ofs=47469
(4986) nb_read: handle 4329 was not open size=1024 ofs=2157
(4987) nb_read: handle 4329 was not open size=4688 ofs=1667
(4988) nb_read: handle 4329 was not open size=3516 ofs=6355
(4989) nb_read: handle 4329 was not open size=16384 ofs=9871
(4990) nb_read: handle 4329 was not open size=3771 ofs=26255
(4991) nb_read: handle 4329 was not open size=2048 ofs=30026
(4992) nb_read: handle 4329 was not open size=1024 ofs=32074
(4993) nb_read: handle 4329 was not open size=6652 ofs=33098
(4994) nb_read: handle 4329 was not open size=16384 ofs=39750
(4995) nb_read: handle 4329 was not open size=10496 ofs=56134
(4996) nb_read: handle 4329 was not open size=1784 ofs=66630
(4997) nb_read: handle 4329 was not open size=90 ofs=68414
(4998) nb_read: handle 4329 was not open size=273 ofs=68504
(4999) nb_read: handle 4329 was not open size=1600 ofs=68777
(5000) nb_read: handle 4329 was not open size=640 ofs=70377
(5001) nb_read: handle 4329 was not open size=1280 ofs=71017
(5002) nb_read: handle 4329 was not open size=8464 ofs=72297
(5003) nb_read: handle 4329 was not open size=158 ofs=80761
(5004) nb_read: handle 4329 was not open size=912 ofs=80919
(5005) nb_read: handle 4329 was not open size=1980 ofs=81831
(5006) nb_read: handle 4329 was not open size=2256 ofs=83811
(5007) nb_read: handle 4329 was not open size=400 ofs=86067
(5008) nb_read: handle 4329 was not open size=252 ofs=86467
(5009) nb_read: handle 4329 was not open size=42 ofs=86719
(5011) nb_read: handle 4329 was not open size=725 ofs=86761
(5012) nb_read: handle 4329 was not open size=512 ofs=2009
(5013) nb_read: handle 4329 was not open size=9 ofs=3895
(5015) nb_read: handle 4329 was not open size=46 ofs=3781
(5017) nb_read: handle 4329 was not open size=92 ofs=4059
(5018) nb_read: handle 4329 was not open size=92 ofs=3781
(5019) nb_read: handle 4329 was not open size=184 ofs=3986
(5020) nb_read: handle 4329 was not open size=16384 ofs=3987
(5021) nb_read: handle 4329 was not open size=16384 ofs=3902
(5022) nb_read: handle 4329 was not open size=10240 ofs=3781
(5023) nb_read: handle 4329 was not open size=16384 ofs=3821
(5024) nb_read: handle 4329 was not open size=935 ofs=3891
(5025) nb_read: handle 4329 was not open size=10240 ofs=4060
(5026) nb_read: handle 4329 was not open size=184 ofs=4059
(5027) nb_read: handle 4329 was not open size=184 ofs=256
(5028) nb_read: handle 4329 was not open size=1026 ofs=256
(5029) nb_read: handle 4329 was not open size=326 ofs=3901
(5030) nb_read: handle 4329 was not open size=652 ofs=3899
(5031) nb_read: handle 4329 was not open size=2 ofs=256
(5033) nb_read: handle 4329 was not open size=4948 ofs=258
(5034) nb_close: handle 4329 was not open
............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+...+......................................................................+..................................+.............................+.....................+++.........................................................................................................................+....................+...............+.....................................+......+......+.+...............................+.+....+.......................+.................+..................++....+.+..+..........++..+...........+.........+.........+..........................++++........+++++.............................+.++...........+..++++++**************************************************
Throughput 27.8199 MB/sec (NB=34.7749 MB/sec 278.199 MBit/sec)
Average latency 6.940010 ms
Maximum latency 18993.291000 ms
Latency > 2 sec. 0.081048 %
Latency > 10 sec. 0.015581 %
50 clients started
...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+.................................................................+....................................................................................................................................................+..................+...+.............................................................................................................................................................................................................+..................+.......................................................+................++.......+............+.....+...............................+..+...............++.+.................................+............+.....++..++.......................+.........+...............++...........................+++.........+++.+...............+....+...+.....+........+......++....++++++.++**************************************************
Throughput 27.0223 MB/sec (NB=33.7779 MB/sec 270.223 MBit/sec)
Average latency 6.739319 ms
Maximum latency 18699.530000 ms
Latency > 2 sec. 0.090834 %
Latency > 10 sec. 0.013650 %
50 clients started
.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+..............................................................................................................................+............................................................................+...................................................................................................................................................................+.+................................................................+.................................................++++...........................+..+....+..........++...........+....................+.....+.............................................+.............+................+...+..........+++....+....+............................+........+..+.....+....+...+........+........+++...............+++++..++++.++++**************************************************
Throughput 27.2861 MB/sec (NB=34.1077 MB/sec 272.861 MBit/sec)
Average latency 7.051013 ms
Maximum latency 23801.490000 ms
Latency > 2 sec. 0.091263 %
Latency > 10 sec. 0.014722 %
50 clients started
.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................^C
[root@inif w]# umount /mnt
[root@inif w]# e2fsck -f -y /dev/sdc
e2fsck 1.38 (30-Jun-2005)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences: -(1250434--1250435) -(1273360--1273370)
Fix? yes
Free blocks count wrong for group #38 (32253, counted=32266).
Fix? yes

Free blocks count wrong (1003250, counted=1003263).
Fix? yes


/dev/sdc: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sdc: 68842/640000 files (0.1% non-contiguous), 276737/1280000 blocksNov 1 16:27:35 inif kernel: kjournald starting. Commit interval 5 seconds
Nov 1 16:27:35 inif kernel: EXT3 FS on sdc, internal journal
Nov 1 16:27:35 inif kernel: EXT3-fs: mounted filesystem with ordered data mode.
Nov 1 16:29:32 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:29:32 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:29:32 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:29:37 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:30:18 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:30:18 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:30:18 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:30:19 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:30:19 inif kernel: end_request: I/O error, dev sdc, sector 9376352
Nov 1 16:30:19 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:30:19 inif kernel: end_request: I/O error, dev sdc, sector 9995808
Nov 1 16:30:19 inif kernel: EXT3-fs error (device sdc): ext3_find_entry: reading directory #608098 offset 0
Nov 1 16:30:23 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:31:04 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:31:04 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:31:04 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:31:05 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:31:05 inif kernel: end_request: I/O error, dev sdc, sector 367432
Nov 1 16:31:05 inif kernel: __ratelimit: 62 callbacks suppressed
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45929
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45930
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45931
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45932
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45933
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45934
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45935
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45936
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45937
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45938
Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:31:09 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:31:22 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov 1 16:31:54 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:31:54 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:31:54 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:31:59 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:32:40 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:32:40 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:32:40 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:32:45 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:33:26 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:33:26 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:33:26 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:33:31 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:34:12 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:34:12 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:34:12 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:34:13 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:34:13 inif kernel: end_request: I/O error, dev sdc, sector 9621088
Nov 1 16:34:13 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:34:13 inif kernel: end_request: I/O error, dev sdc, sector 10145456
Nov 1 16:34:13 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:34:13 inif kernel: end_request: I/O error, dev sdc, sector 7095792
Nov 1 16:34:17 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:34:58 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:34:58 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:34:58 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9530232
Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9798712
Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9798728
Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9798736
Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 10059240
Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9798720
Nov 1 16:35:03 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:35:44 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:35:44 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:35:44 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:35:49 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:36:30 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:36:30 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:36:30 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:36:31 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:36:31 inif kernel: end_request: I/O error, dev sdc, sector 10032752
Nov 1 16:36:35 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:37:16 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:37:16 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:37:16 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8862288
Nov 1 16:37:17 inif kernel: __ratelimit: 59 callbacks suppressed
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107786
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107787
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107788
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107789
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107790
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107791
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107792
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107793
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107794
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107795
Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8892560
Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8893456
Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8894480
Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8895512
Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8896536
Nov 1 16:37:21 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:37:29 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov 1 16:38:02 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:38:02 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:38:02 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9213944
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9280464
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9280472
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9280480
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9184784
Nov 1 16:38:03 inif kernel: EXT3-fs error (device sdc): ext3_find_entry: reading directory #560038 offset 0
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 10159136
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 303616
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9586656
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9641352
Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 10061776
Nov 1 16:38:03 inif kernel: EXT3-fs error (device sdc): ext3_find_entry: reading directory #576172 offset 0
Nov 1 16:38:07 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:38:48 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:38:48 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:38:48 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:38:53 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:39:34 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:39:34 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:39:34 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:39:39 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:40:20 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:40:20 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:40:20 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:40:25 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:41:06 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:41:06 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:41:06 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:41:11 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:41:52 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:41:52 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:41:52 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:41:53 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:41:53 inif kernel: end_request: I/O error, dev sdc, sector 647544
Nov 1 16:41:53 inif kernel: __ratelimit: 639 callbacks suppressed
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80943
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80944
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80945
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80946
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80947
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80948
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80949
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80950
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80951
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80952
Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:41:57 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:42:05 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov 1 16:42:38 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:42:38 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:42:38 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:42:43 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:43:24 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:43:24 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:43:24 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:43:25 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:43:25 inif kernel: end_request: I/O error, dev sdc, sector 478480
Nov 1 16:43:25 inif kernel: __ratelimit: 118 callbacks suppressed
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59810
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59811
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59812
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59813
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59814
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59815
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59816
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59817
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59818
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59819
Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:43:29 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:44:10 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:44:10 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:44:10 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:44:15 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:44:56 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:44:56 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:44:56 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:44:57 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:44:57 inif kernel: end_request: I/O error, dev sdc, sector 9353840
Nov 1 16:44:57 inif kernel: __ratelimit: 44 callbacks suppressed
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169230
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169231
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169232
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169233
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169234
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169235
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169236
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169237
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169238
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169239
Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:45:01 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:45:11 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov 1 16:45:42 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:45:42 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:45:42 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:45:47 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:46:28 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:46:28 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:46:28 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9226672
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9426536
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9832040
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9895976
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9896048
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9946768
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 279576
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 1048576
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 7097712
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9102152
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 10190752
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 10199216
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 295416
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9057504
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9115608
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9289416
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9529432
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9577376
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9672304
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9763984
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9789792
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9811504
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9826536
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9115656
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9624224
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9928144
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 10003472
Nov 1 16:46:29 inif kernel: EXT3-fs error (device sdc): ext3_free_branches: Read failure, inode=517632, block=1250434
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 10186880
Nov 1 16:46:29 inif kernel: EXT3-fs error (device sdc): ext3_free_branches: Read failure, inode=560474, block=1273360
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 332944
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 451768
Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9208392
Nov 1 16:46:29 inif kernel: EXT3-fs error (device sdc): ext3_find_entry: reading directory #560267 offset 0
Nov 1 16:46:33 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:47:14 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:47:14 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:47:14 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:47:15 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:47:15 inif kernel: end_request: I/O error, dev sdc, sector 429128
Nov 1 16:47:15 inif kernel: __ratelimit: 67 callbacks suppressed
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 53641
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:47:15 inif kernel: end_request: I/O error, dev sdc, sector 10138096
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267262
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267263
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267264
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267265
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267266
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267267
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267268
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267269
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267270
Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:47:15 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:47:15 inif kernel: end_request: I/O error, dev sdc, sector 964064
Nov 1 16:47:19 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:47:28 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov 1 16:47:33 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov 1 16:48:00 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:48:00 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:48:00 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:48:05 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:48:46 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:48:46 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:48:46 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:48:47 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:48:47 inif kernel: end_request: I/O error, dev sdc, sector 10082504
Nov 1 16:48:47 inif kernel: __ratelimit: 164 callbacks suppressed
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260313
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260314
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260315
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260316
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260317
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260318
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260319
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260320
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260321
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260322
Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:48:47 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:48:47 inif kernel: end_request: I/O error, dev sdc, sector 10086400
Nov 1 16:48:51 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:49:00 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov 1 16:49:32 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:49:32 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:49:32 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:49:33 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:49:33 inif kernel: end_request: I/O error, dev sdc, sector 9777376
Nov 1 16:49:33 inif kernel: __ratelimit: 132 callbacks suppressed
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222172
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222173
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222174
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222175
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222176
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222177
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222178
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222179
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222180
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222181
Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:49:37 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:49:46 inif kernel: JBD: Detected IO errors while flushing file data on sdc
Nov 1 16:50:18 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:50:18 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:50:18 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:50:19 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:50:19 inif kernel: end_request: I/O error, dev sdc, sector 9728192
Nov 1 16:50:19 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:50:19 inif kernel: end_request: I/O error, dev sdc, sector 9350872
Nov 1 16:50:23 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Nov 1 16:51:04 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED.
Nov 1 16:51:04 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0).
Nov 1 16:51:04 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0).
Nov 1 16:51:05 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:51:05 inif kernel: end_request: I/O error, dev sdc, sector 9709760
Nov 1 16:51:05 inif kernel: __ratelimit: 54 callbacks suppressed
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213720
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213721
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213722
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213723
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213724
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213725
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213726
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213727
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213728
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213729
Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc
Nov 1 16:51:05 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK
Nov 1 16:51:05 inif kernel: end_request: I/O error, dev sdc, sector 9709864
Nov 1 16:51:09 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
\
 
 \ /
  Last update: 2008-11-01 14:11    [from the cache]
©2003-2011 Jasper Spaans