Messages in this thread | | | Date | Fri, 21 Aug 2009 12:05:10 +0100 | Subject | Re: 2.6.28.9: EXT3/NFS inodes corruption | From | Daniel J Blueman <> |
| |
Hi Sylvain,
On Tue, Jul 28, 2009 at 5:45 PM, Sylvain Rochet<gradator@gradator.net> wrote: > On Tue, Jul 28, 2009 at 09:40:42AM -0700, Daniel J Blueman wrote: >> On Apr 20, 5:30 pm, Sylvain Rochet <grada...@gradator.net> wrote: >> > Hi, >> > >> > We(TuxFamily) are having some inodes corruptions on a NFS server. >> > >> > So, let's start with the facts. >> > >> > ==== NFS Server >> > >> > Linux bazooka 2.6.28.9 #1 SMP Mon Mar 30 12:58:22 CEST 2009 x86_64 GNU/Linux >> [snip] >> >> Can you do a 'lspci -v' on the server please? > > Of course yes. > > I attached the 'lspci -v' of the previous and the current storage > server.
The reason I ask, I was chasing data corruption across the PCIe bus with some high-performance Quadrics interconnect adapters a while ago. The reproducer involved multiple outstanding main memory read requests to related addresses and a small block of data would be returned from the wrong offset.
In the end, I found the nVidia CK804 (also MCP55) HT->PCIe bridge was at fault and later found disk corruption when doing heavy rsyncs to network. This was never publicly acknowledged, but I guess it illustrates the need for some micro-tests to verify data-soundness under duress; it took a day (and petabytes of data) of the production I/O workload to get this data corruption, and 3 seconds with the right reproducer, (still non-trivial to catch on a PCIe protocol analyser).
Sometime I'll develop a stress-test driver for a common SATA or network controller to drive it's DMA engine with I/O patterns to and from main memory, checking the data integrity every few seconds; this could be generalised with OpenGL nicely for graphics cards on workstations I imagine.
Daniel -- Daniel J Blueman -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |