lkml.org 
[lkml]   [1999]   [Nov]   [25]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
SubjectRe: Are SMP spinlocks safe in WB cached mem?
Date
From

Robert Redelmeier <redelm@ev1.net> wrote:

> Having an SMP box, and noting all the SMP crash reports I did some
> light reading :) of the IA32 System Programming Manual. I came
> up with a scenario for P6 SMP spinlock corruption/theft:
>
> CPU0 has and clears spinlock SL. The write is still in cache not mem
> CPU1 tries to read SL.
> CPU0 sees the read and helpfully signals HITM# and passes the line.
> CPU1 starts receiving the line and sets SL
> CPU0 "simultaneously" sets SL in it's cache line.
>
> Both CPUs think they have SL, and will cause nonreentrancy chaos.
> Of course, this is going to be _very_ rare. But spinlocks spin,
> and spinlocks are often called again. But what is "simultaneous"?
> The 40-60 ns it takes to transfer the cache line--24 cpu clocks
> with a 6x multiplier? The same busclock? The same CPU clock?

If this didn't work, you'd see data corruption and crashes on a
much larger scale. I don't know the details of the bus signalling
protocol, but I'm pretty darn sure that only one processor is allowed
to own a cache-line in the M or E states at any one time.

When debugging this stuff, even 1-cycle windows tend to show up in 4-way
SMP stress-testing... we even test multiple different SMP OSes.

We've got 8-way systems that happily run for weeks/months of uptime.

I myself ran a 4-way Linux SMP box with uptimes in the 5+ month
area.


...
> Ref [1]&[2] are very clear that LOCK only goes as far as cache
> (L1 or L2?) and memory writes can be long delayed for write-back
> cached memory. [3] describes the HITM# mechanism.
>
> Unless HITM# also sets the sent cache line to MESI "invalid",
> then the simultaneous write could happen. Most likely, it sets
> shared. CPU0 can certainly read a clear SL in it's cache. And
> CPU1 just got it as it's supposed to be (shared?). The chipset is
> supposed to write the HITM# to DRAM (if the BX isn't throttled).
> First write sets "exclusive" on itself, "invalid" on the other,
> and goes to DRAM. But what if writes are "simultaneous"?

I think you're confusing how MESI works.

First of all, for the LOCK to only go as far as the cache, then
the line must be in a either of the M or E states. If it is in
an S state, it must get into one of the M or E states first.


> Reading between the lines of the full Chapters 7 & 9, write-back
> cached DRAM has higher performance, but is not fully SMP coherent.

I think they were referring to how hardware devices would see the
bus. See following comments.


> Fortunately, a solution is also hinted at: use write-thru or
> uncachable attributes (PCD & PWT) for critical memory.
>
> Is this correct? or have I misread something?

The WB memory type is very definitely the one you want to use. It
is coherent, just not from the perspective of hardware devices
on the bus. Using WT or UC would add complexity (since you have to
set it up), and simply lose you performance.


...
> [2] The ia32 System Pgming Manual section 7.2.4 says:
> >* For areas of memory where weak ordering is acceptable, the write
> > back (WB) memory type can be chosen. Here, reads can be performed
> > speculatively and writes can be buffered and combined. For this type
> > of memory, cache locking is performed on atomic (locked) operations
> > that do not split across cache lines, which helps to reduce the
> > performance penalty associated with the use of the typical synchronization
> > instructions, such as XCHG, that lock the bus during the entire
> > read-modify-write operation. With the WB memory type, the XCHG
> > instruction locks the cache instead of the bus if the memory access
> > is contained within a cache line.

I'd guess the comment about "Weak Ordering" here is referring to how
hardware devices would see it (hence the comment on the fact that
reads can be done speculatively and writes can be bufferred and
combined... if you only see the signals on the bus and don't disambiguate
what a cacheable image would look like, it would be Weak).

IA32 uses "Processor Ordering" (as I've surely bored the members
of this list by saying enough times in the last day or 2 ;) for WB memory
with respect to other processors.

For hardware devices, you have to use UC to make sure the actual bus
cycles are the right ones, since the very act of doing a read or
a write of a particular size, for example, might be recognized and
used by a particular device.


Erich Boleyn
PMD IA32 Architecture
Intel
--
Erich Boleyn
PMD Architecture
<esboleyn@ichips.intel.com>

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2005-03-22 13:55    [W:0.059 / U:0.860 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site