[lkml]   [1999]   [Aug]   [8]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
SubjectRe: IPSEC transport mode w/2.2.x kernels and large packets
Date (Richard Guy Briggs) writes:

> Large packet handling was not checked at the time and was tripped over
> afterwards. When large packets hit a 2.2.x kernel with ipsec and were
> passed to ip_fragment(), the machine oopsed, sometimes hard. The
> output function passed to ip_fragment() was skb->dst->output(). Once
> it was changed to neigh_compat_output(), as was used in the place of
> dev_queue_xmit() from 2.0.x to 2.2.x, the problem went away *for me*
> and got worse for others. This started my quest as to what function
> should be used for output, and lead into cflow diagrams...

Simple. Not _your_ functions, but the functions returned by the
routing table lookung and the neighbour state machine. Trying
to muck with dst->output is recipe for desaster

> There have been reports of severe crashes under the 2.2 kernels with
> large packets. I was seeing fairly regular crashes under 2.2.7 which
> would start with a small oops and keep rolling to larger and larger
> oopses until they finally exhausted themselves. I upgraded to 2.2.10
> and those problems largely went away, with only the occasional crash.
> Others have reported exactly the opposite behaviour with relative
> stability under 2.2.7 and crashing consistently under 2.2.10. There
> have been many reports on this list and linux-kernel about
> instability, locking up, tcp stalling and such for many of the 2.2.x
> series kernels, so we are not certain it is anything we are doing.
> There was a report on the list that 2.2.11pre4 is more stable. I am
> trying that, but still have some unexplained behaviour, which might be
> the 2.0.36 machine at the other end, which is not able to re-assemble
> the fragments for some reason. Investigation pending.

It is easy to blame the kernel stack of course. Although there are bugs,
you should be aware that about 80% of the reports are caused
because the users use some equipment that can't deal with the new
SACK/timestamp/window scaling TCP options.

> At the same time, I did a FreeS/WAN presentation at the Ottawa Linux
> Symposium and in discussions with a number of other developpers,
> describing the problem elucidated some of the issues surrounding doing
> transport mode IPSEC. It confirmed that we need to move our
> processing up the stack, before things are fragmented. In particular,
> Alan Cox suggested setting the MTU to the maximum allowed:0xfff0, then
> fragmenting after encryption to the MTU of the physical device.

So when one fragment gets dropped the whole packet is lost ? This is the
"NFS trap" and works badly over WANs, because you would disturb path
MTU discovery. It is important that packets with DF get a icmp error
when they exceed the effective size of the tunnel.

> This seems to work under the 2.0.x kernels since before I call
> dev_queue_xmit() or ip_fragment(), I can change the ownership of the
> skb from the virtual device to the physical device. In the process of
> testing all this, I started getting some skb's that were not getting
> skb_copy_expand()ed properly. For some reason,
> skb->truesize-sizeof(sk_buff) was not quite right. I have changed it
> to be the same as that used in the 2.2.x kernels by Marc Boucher and I
> am starting to see some 'cannot get free page' messages, which I am
> not certain are from that change or the maximum mtu change. It only
> seems to happen with connections directly to the gateway, and not
> connections *through* the gateway that use ipsec.

skb->truesize is broken in 2.2 (it should contains sizeof(sk_buff) but
it does not ATM). Why are you using that anyways? You should use

> Under 2.2.x kernels, this is not so simple, since even if I change the
> ownership of the skb, skb->dev, the skb still has a dst cache entry
> pointer, skb->dst, which still assumes the old device, skb->dst->dev.
> How do we change the dst entry? We can't do another routing table
> lookup since that is unreliable and prone to route stealing, besides
> we are trying to move away from that method. Do we blow it away? It
> would then need to do a routing lookup. How do we tell it to use the
> new device without corrupting the dst cache? The only way I could
> come up with was to patch net/ipv4/ip_output.c:ip_fragment() so that
> if skb->dst->dev, the device pointed to in the dst cache pointer, and
> skb->dev, the device pointed to by the skb, were different, it would
> take its MTU information from skb->dev->mtu, rather than where it does
> now, from skb->dst->pmtu.

Why is another routing table lookup unreliable? For me it seems to
be the cleanest way (e.g. ip_gre does it too). I don't know what you
mean with route stealing. Trying to play your own games with the dst
is not a good idea, see the dst as encapsulated object you shouldn't

This is like TV. I don't like TV.

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to
Please read the FAQ at

 \ /
  Last update: 2005-03-22 13:53    [W:0.095 / U:0.860 seconds]
©2003-2018 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site