lkml.org 
[lkml]   [2015]   [Jun]   [12]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    SubjectRe: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() )
    From
    On Fri, Jun 12, 2015 at 10:40 AM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
    > On Fri, 2015-06-12 at 10:10 -0400, Trond Myklebust wrote:
    >> On Thu, Jun 11, 2015 at 11:49 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
    >> >
    >> > I recently upgraded my main server to 4.0.4 from 3.19.5 and rkhunter
    >> > started reporting a hidden port on my box.
    >> >
    >> > Running unhide-tcp I see this:
    >> >
    >> > # unhide-tcp
    >> > Unhide-tcp 20121229
    >> > Copyright © 2012 Yago Jesus & Patrick Gouin
    >> > License GPLv3+ : GNU GPL version 3 or later
    >> > http://www.unhide-forensics.info
    >> > Used options:
    >> > [*]Starting TCP checking
    >> >
    >> > Found Hidden port that not appears in ss: 946
    >> > [*]Starting UDP checking
    >> >
    >> > This scared the hell out of me as I'm thinking that I have got some kind
    >> > of NSA backdoor hooked into my server and it is monitoring my plans to
    >> > smuggle Kinder Überraschung into the USA from Germany. I panicked!
    >> >
    >> > Well, I wasted the day writing modules to first look at all the sockets
    >> > opened by all processes (via their file descriptors) and posted their
    >> > port numbers.
    >> >
    >> > http://rostedt.homelinux.com/private/tasklist.c
    >> >
    >> > But this port wasn't there either.
    >> >
    >> > Then I decided to look at the ports in tcp_hashinfo.
    >> >
    >> > http://rostedt.homelinux.com/private/portlist.c
    >> >
    >> > This found the port but no file was connected to it, and worse yet,
    >> > when I first ran it without using probe_kernel_read(), it crashed my
    >> > kernel, because sk->sk_socket pointed to a freed socket!
    >> >
    >> > Note, each boot, the hidden port is different.
    >> >
    >> > Finally, I decided to bring in the big guns, and inserted a
    >> > trace_printk() into the bind logic, to see if I could find the culprit.
    >> > After fiddling with it a few times, I found a suspect:
    >> >
    >> > kworker/3:1H-123 [003] ..s. 96.696213: inet_bind_hash: add 946
    >> >
    >> > Bah, it's a kernel thread doing it, via a work queue. I then added a
    >> > trace_dump_stack() to find what was calling this, and here it is:
    >> >
    >> > kworker/3:1H-123 [003] ..s. 96.696222: <stack trace>
    >> > => inet_csk_get_port
    >> > => inet_addr_type
    >> > => inet_bind
    >> > => xs_bind
    >> > => sock_setsockopt
    >> > => __sock_create
    >> > => xs_create_sock.isra.18
    >> > => xs_tcp_setup_socket
    >> > => process_one_work
    >> > => worker_thread
    >> > => worker_thread
    >> > => kthread
    >> > => kthread
    >> > => ret_from_fork
    >> > => kthread
    >> >
    >> > I rebooted, and examined what happens. I see the kworker binding that
    >> > port, and all seems well:
    >> >
    >> > # netstat -tapn |grep 946
    >> > tcp 0 0 192.168.23.9:946 192.168.23.22:55201 ESTABLISHED -
    >> >
    >> > But waiting for a bit, the connection goes into a TIME_WAIT, and then
    >> > it just disappears. But the bind to the port does not get released, and
    >> > that port is from then on, taken.
    >> >
    >> > This never happened with my 3.19 kernels. I would bisect it but this is
    >> > happening on my main server box which I usually only reboot every other
    >> > month doing upgrades. It causes too much disturbance for myself (and my
    >> > family) as when this box is offline, basically the rest of my machines
    >> > are too.
    >> >
    >> > I figured this may be enough information to see if you can fix it.
    >> > Otherwise I can try to do the bisect, but that's not going to happen
    >> > any time soon. I may just go back to 3.19 for now, such that rkhunter
    >> > stops complaining about the hidden port.
    >> >
    >>
    >> The only new thing that we're doing with 4.0 is to set SO_REUSEPORT on
    >> the socket before binding the port (commit 4dda9c8a5e34: "SUNRPC: Set
    >> SO_REUSEPORT socket option for TCP connections"). Perhaps there is an
    >> issue with that?
    >
    > Strange, because the usual way to not have time-wait is to use SO_LINGER
    > with linger=0
    >
    > And apparently xs_tcp_finish_connecting() has this :
    >
    > sock_reset_flag(sk, SOCK_LINGER);
    > tcp_sk(sk)->linger2 = 0;

    Are you sure? I thought that SO_LINGER is more about controlling how
    the socket behaves w.r.t. waiting for the TCP_CLOSE state to be
    achieved (i.e. about aborting the FIN state negotiation early). I've
    never observed an effect on the TCP time-wait states.

    > Are you sure SO_REUSEADDR was not the thing you wanted ?

    Yes. SO_REUSEADDR has the problem that it requires you bind to
    something other than 0.0.0.0, so it is less appropriate for outgoing
    connections; the RPC code really should not have to worry about
    routing and routability of a particular source address.

    Cheers
    Trond


    \
     
     \ /
      Last update: 2015-06-12 17:21    [W:7.133 / U:0.052 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site