Messages in this thread | | | From | Richard Guy Briggs <> | Subject | Re: IPSEC transport mode w/2.2.x kernels and large packets | Date | Sat, 7 Aug 1999 11:52:31 -0400 (EDT) |
| |
-----BEGIN PGP SIGNED MESSAGE-----
> rgb@conscoop.ottawa.on.ca (Richard Guy Briggs) writes:
[I wish you had not removed the Cc:'s and crosspost addresses! ...particularly linux-ipsec@clinet.fi]
> > Large packet handling was not checked at the time and was tripped over > > afterwards. When large packets hit a 2.2.x kernel with ipsec and were > > passed to ip_fragment(), the machine oopsed, sometimes hard. The > > output function passed to ip_fragment() was skb->dst->output(). Once > > it was changed to neigh_compat_output(), as was used in the place of > > dev_queue_xmit() from 2.0.x to 2.2.x, the problem went away *for me* > > and got worse for others. This started my quest as to what function > > should be used for output, and lead into cflow diagrams... > > Simple. Not _your_ functions, but the functions returned by the > routing table lookung and the neighbour state machine. Trying > to muck with dst->output is recipe for desaster
I didn't mess with dst->output, I simply replaced the argument to ip_forward(..., skb->dst->output) with ip_forward(..., neigh_compat_output)
> > There have been reports of severe crashes under the 2.2 kernels with > > large packets. I was seeing fairly regular crashes under 2.2.7 which > > would start with a small oops and keep rolling to larger and larger > > oopses until they finally exhausted themselves. I upgraded to 2.2.10 > > and those problems largely went away, with only the occasional crash. > > Others have reported exactly the opposite behaviour with relative > > stability under 2.2.7 and crashing consistently under 2.2.10. There > > have been many reports on this list and linux-kernel about > > instability, locking up, tcp stalling and such for many of the 2.2.x > > series kernels, so we are not certain it is anything we are doing. > > There was a report on the list that 2.2.11pre4 is more stable. I am > > trying that, but still have some unexplained behaviour, which might be > > the 2.0.36 machine at the other end, which is not able to re-assemble > > the fragments for some reason. Investigation pending. > > It is easy to blame the kernel stack of course. Although there are bugs, > you should be aware that about 80% of the reports are caused > because the users use some equipment that can't deal with the new > SACK/timestamp/window scaling TCP options.
Can you explain this in more detail? I am not touching the upper-layer protocol headers at all. I don't need to know about them. If the 2.2.10 box can talk to a 2.0.36 box, then that layer should still work. I just need to muck with the mss for that route.
> > At the same time, I did a FreeS/WAN presentation at the Ottawa Linux > > Symposium and in discussions with a number of other developpers, > > describing the problem elucidated some of the issues surrounding doing > > transport mode IPSEC. It confirmed that we need to move our > > processing up the stack, before things are fragmented. In particular, > > Alan Cox suggested setting the MTU to the maximum allowed:0xfff0, then > > fragmenting after encryption to the MTU of the physical device. > > So when one fragment gets dropped the whole packet is lost ? This is the > "NFS trap" and works badly over WANs, because you would disturb path > MTU discovery. It is important that packets with DF get a icmp error > when they exceed the effective size of the tunnel.
Yes, I know it can cause bad performance. In fact, maybe what should be done is to send back an ICMP message only for tunnel mode packets, and not for transport mode. I am not presently differentiating.
> > This seems to work under the 2.0.x kernels since before I call > > dev_queue_xmit() or ip_fragment(), I can change the ownership of the > > skb from the virtual device to the physical device. In the process of > > testing all this, I started getting some skb's that were not getting > > skb_copy_expand()ed properly. For some reason, > > skb->truesize-sizeof(sk_buff) was not quite right. I have changed it > > to be the same as that used in the 2.2.x kernels by Marc Boucher and I > > am starting to see some 'cannot get free page' messages, which I am > > not certain are from that change or the maximum mtu change. It only > > seems to happen with connections directly to the gateway, and not > > connections *through* the gateway that use ipsec. > > skb->truesize is broken in 2.2 (it should contains sizeof(sk_buff) but > it does not ATM). Why are you using that anyways? You should use > skb->len.
Yes, I know, that is why Marc did not use it. I was using it in 2.0, since skb_copy_expand is a customised skb_copy(). In fact, skb->len is not complete enough. I want to make the skb larger by 'headroom' plus 'tailroom' over the (obviously too small, otherwise I would have used it) existing skb. This is why I use skb->end - skb->head, which includes all of the existing headroom and tailroom which is needed for bits that have been or will be stripped off for processing.
> > Under 2.2.x kernels, this is not so simple, since even if I change the > > ownership of the skb, skb->dev, the skb still has a dst cache entry > > pointer, skb->dst, which still assumes the old device, skb->dst->dev. > > How do we change the dst entry? We can't do another routing table > > lookup since that is unreliable and prone to route stealing, besides > > we are trying to move away from that method. Do we blow it away? It > > would then need to do a routing lookup. How do we tell it to use the > > new device without corrupting the dst cache? The only way I could > > come up with was to patch net/ipv4/ip_output.c:ip_fragment() so that > > if skb->dst->dev, the device pointed to in the dst cache pointer, and > > skb->dev, the device pointed to by the skb, were different, it would > > take its MTU information from skb->dev->mtu, rather than where it does > > now, from skb->dst->pmtu. > > Why is another routing table lookup unreliable? For me it seems to > be the cleanest way (e.g. ip_gre does it too). I don't know what you > mean with route stealing. Trying to play your own games with the dst > is not a good idea, see the dst as encapsulated object you shouldn't > touch.
Right, I am not trying to play with the dst cache, just use the right entry.
Take the following example:
A-----B=======C==========D=========E------F
A, F are endhosts which know nothing of IPSEC
B, E are security GWs, one foot in a trusted network, the other in the big, bad internet
C, D are the nexthop routers from the security GWs
- ----- is an unprotected local link from an endhost to a security GW
===== is an IPSEC tunnel
Say we have a tunnel mode connection set up from B to E to cover traffic from A to F. Say we also have a tunnel or transport mode connection set up from B to E to cover packets with source and destination addresses of B or E (which includes previously encapsulated packets from A to F and vice versa). The routing table will look like:
Destination Gateway Iface A-B_subnet - eth1 B - lo B-C_subnet - eth0 default C eth0 E C ipsec0 E-F_subnet C ipsec0
Since the packet destination has not changed for any packets going from B to E or the device and next hop have not changed for any packets going from A to F, then B to E, we don't have any physical I/F to point to in the dst cache.
> -Andi
slainte mhath, RGB - -- The first Ottawa Linux Symposium was a huge success! <ottawalinuxsymposium.org> This SunRayce was a wet one! DroughtRelief_99? -- <www.sunrayce.com/sunrayce/> Richard Guy Briggs -- PGP key available Auto-Free Ottawa! Canada <http://www.conscoop.ottawa.on.ca/rgb/> </www.flora.org/afo/> Prevent Internet Wiretapping! -- FreeS/WAN:<www.xs4all.nl/~freeswan> Thanks for voting Green! -- <green.ca> Marillion:<www.marillion.co.uk>
-----BEGIN PGP SIGNATURE----- Version: 2.6.3i Charset: noconv
iQCVAwUBN6xWPd+sBuIhFagtAQFdYgQAnESbL6fTVicfJ9Cv/V2+84lDiUKmGM1m ts5XYlj2Dy30cwWieTzimAbrt/0941DUGSIqGg7HeUTae3KU3YpU7DHZnNKipOFM o1SapfaS60qETL28ct9yWhhUvZLINBXYWKXPtWMkvaAY3R0XXo9bbBtTr/8lWGZC 1WGzWrkSZ38= =MJ5x -----END PGP SIGNATURE-----
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
| |