xfrm: Don't allow esn with disabled anti replay detection

Unlike the standard case, disabled anti replay detection needs some
nontrivial extra treatment on ESN. RFC 4303 states:

Note: If a receiver chooses to not enable anti-replay for an SA, then
the receiver SHOULD NOT negotiate ESN in an SA management protocol.
Use of ESN creates a need for the receiver to manage the anti-replay
window (in order to determine the correct value for the high-order
bits of the ESN, which are employed in the ICV computation), which is
generally contrary to the notion of disabling anti-replay for an SA.

So return an error if an ESN state with disabled anti replay detection
is inserted for now and add the extra treatment later if we need it.

Signed-off-by: Steffen Klassert <steffen.klassert@secunet.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

1 files changed, 3 insertions, 0 deletions

diff --git a/net/xfrm/xfrm_replay.c b/net/xfrm/xfrm_replay.c
index e8a781422feb..47f1b8638df9 100644
--- a/net/xfrm/xfrm_replay.c
+++ b/net/xfrm/xfrm_replay.c
@@ -535,6 +535,9 @@ int xfrm_init_replay(struct xfrm_state *x)
                     replay_esn->bmp_len * sizeof(__u32) * 8)
                         return -EINVAL;
 
+        if ((x->props.flags & XFRM_STATE_ESN) && replay_esn->replay_window == 0)
+                return -EINVAL;
+
         if ((x->props.flags & XFRM_STATE_ESN) && x->replay_esn)
                 x->repl = &xfrm_replay_esn;
         else