crypto: hash - Remove VLA usage

In the quest to remove all stack VLA usage from the kernel[1], this
removes the VLAs in SHASH_DESC_ON_STACK (via crypto_shash_descsize())
by using the maximum allowable size (which is now more clearly captured
in a macro), along with a few other cases. Similar limits are turned into
macros as well.

A review of existing sizes shows that SHA512_DIGEST_SIZE (64) is the
largest digest size and that sizeof(struct sha3_state) (360) is the
largest descriptor size. The corresponding maximums are reduced.

[1] https://lkml.kernel.org/r/CA+55aFzCG-zNmZwX4A2FQpadafLfEzK6CC=qPXydAacU1RqZWA@mail.gmail.com

Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

1 files changed, 2 insertions, 2 deletions

diff --git a/crypto/ahash.c b/crypto/ahash.c
index a64c143165b1..78aaf2158c43 100644
--- a/crypto/ahash.c
+++ b/crypto/ahash.c
@@ -550,8 +550,8 @@ static int ahash_prepare_alg(struct ahash_alg *alg)
 {
         struct crypto_alg *base = &alg->halg.base;
 
-        if (alg->halg.digestsize > PAGE_SIZE / 8 ||
-            alg->halg.statesize > PAGE_SIZE / 8 ||
+        if (alg->halg.digestsize > HASH_MAX_DIGESTSIZE ||
+            alg->halg.statesize > HASH_MAX_STATESIZE ||
             alg->halg.statesize == 0)
                 return -EINVAL;