Fix internal memory leaks from OPENSSL_MALLOC_FAILURES

There is a rarely used feature that can be enabled
with `./config enable-crypto-mdebug` when additionally
the environment variable OPENSSL_MALLOC_FAILURES is used.
It turns out to be possible that CRYPTO_zalloc may
create a leak when the memory is allocated and then
the shouldfail happens, then the memory is lost.
Likewise when OPENSSL_realloc is used with size=0,
then the memory is to be free'd but here the shouldfail
check is too early, and the failure may prevent the
memory to be freed thus creating a bogus memory leak.

Reviewed-by: Matt Caswell <[email protected]>
Reviewed-by: Tomas Mraz <[email protected]>
Reviewed-by: Hugo Landau <[email protected]>
(Merged from openssl#21944)

crypto/mem.c

@@ -214,7 +214,6 @@ void *CRYPTO_zalloc(size_t num, const char *file, int line)
     void *ret;
 
     ret = CRYPTO_malloc(num, file, line);
-    FAILTEST();
     if (ret != NULL)
         memset(ret, 0, num);
 
@@ -227,7 +226,6 @@ void *CRYPTO_realloc(void *str, size_t num, const char *file, int line)
     if (realloc_impl != CRYPTO_realloc)
         return realloc_impl(str, num, file, line);
 
-    FAILTEST();
     if (str == NULL)
         return CRYPTO_malloc(num, file, line);
 
@@ -236,6 +234,7 @@ void *CRYPTO_realloc(void *str, size_t num, const char *file, int line)
         return NULL;
     }
 
+    FAILTEST();
     return realloc(str, num);
 }