malloc: checked_request2size failure deadlocks

For some rarely cases(almost App bugs), calling malloc with
a very largre size, checked_request2size check will fail,set
ENOMEM, and return 0 to caller.

But this will let __malloc_lock futex locked and owned by the
caller. In multithread circumstance, other thread calling
malloc/calloc will NOT succeed and get locked.

Signed-off-by: Zhiqiang Zhang <zhangzhiqiang.zhang@huawei.com>
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>

1 files changed, 3 insertions, 2 deletions

diff --git a/libc/stdlib/malloc-standard/malloc.c b/libc/stdlib/malloc-standard/malloc.c
index 2abb5bbdd..fd33b50c7 100644
--- a/libc/stdlib/malloc-standard/malloc.c
+++ b/libc/stdlib/malloc-standard/malloc.c
@@ -832,8 +832,6 @@ void* malloc(size_t bytes)
     }
 #endif
 
-    __MALLOC_LOCK;
-    av = get_malloc_state();
     /*
        Convert request size to internal form by adding (sizeof(size_t)) bytes
        overhead plus possibly more to obtain necessary alignment and/or
@@ -845,6 +843,9 @@ void* malloc(size_t bytes)
 
     checked_request2size(bytes, nb);
 
+    __MALLOC_LOCK;
+    av = get_malloc_state();
+
     /*
        Bypass search if no frees yet
        */