Messages in this thread | | | Subject | Re: IPSEC transport mode w/2.2.x kernels and large packets | From | Andi Kleen <> | Date | 07 Aug 1999 11:08:03 +0200 |
| |
rgb@conscoop.ottawa.on.ca (Richard Guy Briggs) writes:
> Large packet handling was not checked at the time and was tripped over > afterwards. When large packets hit a 2.2.x kernel with ipsec and were > passed to ip_fragment(), the machine oopsed, sometimes hard. The > output function passed to ip_fragment() was skb->dst->output(). Once > it was changed to neigh_compat_output(), as was used in the place of > dev_queue_xmit() from 2.0.x to 2.2.x, the problem went away *for me* > and got worse for others. This started my quest as to what function > should be used for output, and lead into cflow diagrams...
Simple. Not _your_ functions, but the functions returned by the routing table lookung and the neighbour state machine. Trying to muck with dst->output is recipe for desaster
> There have been reports of severe crashes under the 2.2 kernels with > large packets. I was seeing fairly regular crashes under 2.2.7 which > would start with a small oops and keep rolling to larger and larger > oopses until they finally exhausted themselves. I upgraded to 2.2.10 > and those problems largely went away, with only the occasional crash. > Others have reported exactly the opposite behaviour with relative > stability under 2.2.7 and crashing consistently under 2.2.10. There > have been many reports on this list and linux-kernel about > instability, locking up, tcp stalling and such for many of the 2.2.x > series kernels, so we are not certain it is anything we are doing. > There was a report on the list that 2.2.11pre4 is more stable. I am > trying that, but still have some unexplained behaviour, which might be > the 2.0.36 machine at the other end, which is not able to re-assemble > the fragments for some reason. Investigation pending.
It is easy to blame the kernel stack of course. Although there are bugs, you should be aware that about 80% of the reports are caused because the users use some equipment that can't deal with the new SACK/timestamp/window scaling TCP options.
> > At the same time, I did a FreeS/WAN presentation at the Ottawa Linux > Symposium and in discussions with a number of other developpers, > describing the problem elucidated some of the issues surrounding doing > transport mode IPSEC. It confirmed that we need to move our > processing up the stack, before things are fragmented. In particular, > Alan Cox suggested setting the MTU to the maximum allowed:0xfff0, then > fragmenting after encryption to the MTU of the physical device.
So when one fragment gets dropped the whole packet is lost ? This is the "NFS trap" and works badly over WANs, because you would disturb path MTU discovery. It is important that packets with DF get a icmp error when they exceed the effective size of the tunnel.
> This seems to work under the 2.0.x kernels since before I call > dev_queue_xmit() or ip_fragment(), I can change the ownership of the > skb from the virtual device to the physical device. In the process of > testing all this, I started getting some skb's that were not getting > skb_copy_expand()ed properly. For some reason, > skb->truesize-sizeof(sk_buff) was not quite right. I have changed it > to be the same as that used in the 2.2.x kernels by Marc Boucher and I > am starting to see some 'cannot get free page' messages, which I am > not certain are from that change or the maximum mtu change. It only > seems to happen with connections directly to the gateway, and not > connections *through* the gateway that use ipsec.
skb->truesize is broken in 2.2 (it should contains sizeof(sk_buff) but it does not ATM). Why are you using that anyways? You should use skb->len.
> > Under 2.2.x kernels, this is not so simple, since even if I change the > ownership of the skb, skb->dev, the skb still has a dst cache entry > pointer, skb->dst, which still assumes the old device, skb->dst->dev. > How do we change the dst entry? We can't do another routing table > lookup since that is unreliable and prone to route stealing, besides > we are trying to move away from that method. Do we blow it away? It > would then need to do a routing lookup. How do we tell it to use the > new device without corrupting the dst cache? The only way I could > come up with was to patch net/ipv4/ip_output.c:ip_fragment() so that > if skb->dst->dev, the device pointed to in the dst cache pointer, and > skb->dev, the device pointed to by the skb, were different, it would > take its MTU information from skb->dev->mtu, rather than where it does > now, from skb->dst->pmtu.
Why is another routing table lookup unreliable? For me it seems to be the cleanest way (e.g. ip_gre does it too). I don't know what you mean with route stealing. Trying to play your own games with the dst is not a good idea, see the dst as encapsulated object you shouldn't touch.
-Andi -- This is like TV. I don't like TV.
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
| |