On Sun, 2011-12-11 at 10:22 +0800, Ming Lei wrote:
> On Sun, Dec 11, 2011 at 3:49 AM, Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:
> > On Sat, 2011-12-10 at 23:57 +0800, Ming Lei wrote:
> >
> >> CPU0                  CPU1
> >>
> >> atomic_set(v)
> >> smp_mb()
> >>                               smp_mb()
> >>                               atomic_dec_and_test(v)
> >>
> >> Without the barrier after atomic_set, CPU1 may see a stale
> >> value of v first, then decrease it, so may miss a release operation.
> >
> > Your example is doubly broken. If there's concurrency possible with
> > atomic_set() you've lost.
> 
> kref_init is guaranteed to be run only one time __before__ executing
> kref_get/kref_put.

If used properly, yes. But in that case you still don't need the
barrier. Whatever means you use to make the object visible to other CPUs
will include a barrier.

> > Lets change it to kref_get() aka atomic_inc():
> >
> >        CPU0            CPU1
> >
> >        atomic_inc()
> >                        atomic_dec_and_test()
> >
> > and
> >
> >                        atomic_dec_and_test()
> >        atomic_inc()
> >
> > For if the first is possible, then so is the second.
> 
> Yes, both are reasonable.
> 
> >
> > This illustrates that no matter how many barriers you put in, you're
> > still up shit creek without no paddle because the kref_put() can come in
> > before you do the kref_get(), making the kref_get() the invalid
> > operation.
> 
> So one smp_mb__before_atomic_inc should be added before atomic_inc
> to make sure that CPU0 can see the uptodate ref, right?

No.

Assume v == 1:

	CPU0		CPU1

			atomic_dec_and_test(); /* --v == 0 */
				kfree()

	smp_mb__before_atomic_inc()
	atomic_inc(); <-- OOPS!


You still got an access to already freed memory. There is no amount of
memory barriers that will solve this problem.

> But the initial value of kref is 1, so seems we don't need to consider
> the 0-refs.

There's a dec in there, isn't it. How much is 1-1?