tcp: eliminate negative reordering in tcp_clean_rtx_queue

[ Upstream commit bafbb9c73241760023d8981191ddd30bb1c6dbac ]

tcp_ack() can call tcp_fragment() which may dededuct the
value tp->fackets_out when MSS changes. When prior_fackets
is larger than tp->fackets_out, tcp_clean_rtx_queue() can
invoke tcp_update_reordering() with negative values. This
results in absurd tp->reodering values higher than
sysctl_tcp_max_reordering.

Note that tcp_update_reordering indeeds sets tp->reordering
to min(sysctl_tcp_max_reordering, metric), but because
the comparison is signed, a negative metric always wins.

Fixes: c7caf8d3ed7a ("[TCP]: Fix reord detection due to snd_una covered holes")
Reported-by: Rebecca Isaacs <risaacs@google.com>
Signed-off-by: Soheil Hassas Yeganeh <soheil@google.com>
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index 22335d8..01336aa 100644 (file)
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -3233,7 +3233,7 @@ static int tcp_clean_rtx_queue(struct sock *sk, int prior_fackets,
                        int delta;
 
                        /* Non-retransmitted hole got filled? That's reordering */
-                       if (reord < prior_fackets)
+                       if (reord < prior_fackets && reord <= tp->fackets_out)
                                tcp_update_reordering(sk, tp->fackets_out - reord, 0);
 
                        delta = tcp_is_fack(tp) ? pkts_acked :