Make __syscall_error return long, as expected by syscall() callers

The return type of syscall() is long so __syscall_error, which is jumped
to by syscall handlers to stash an error number into errno, must return
long too otherwhise it returs 4294967295L instead of -1L. For example,
syscall for x86_64 is defined in libc/sysdeps/linux/x86_64/syscall.S as

    syscall:
        movq %rdi, %rax         /* Syscall number -> rax.  */
        movq %rsi, %rdi         /* shift arg1 - arg5.  */
        movq %rdx, %rsi
        movq %rcx, %rdx
        movq %r8, %r10
        movq %r9, %r8
        movq 8(%rsp),%r9        /* arg6 is on the stack.  */
        syscall                 /* Do the system call.  */
        cmpq $-4095, %rax       /* Check %rax for error.  */
        jae __syscall_error     /* Branch forward if it failed.  */
        ret                     /* Return to caller.  */

In libc/sysdeps/linux/x86_64/__syscall_error.c, __syscall_error is
defined as

    int __syscall_error(void) attribute_hidden;
    int __syscall_error(void)
    {
        register int err_no __asm__ ("%rcx");
        __asm__ ("mov %rax, %rcx\n\t"
                 "neg %rcx");
        __set_errno(err_no);
        return -1;
    }

So __syscall_error returns -1 as a 32-bit int in a 64-bit register, %rax
(0x00000000ffffffff, whose decimal value is decimal 4294967295) and a
test like this always returns false:

    if (syscall(number, ...) == -1)
        foo();

Fix the error by making __syscall_error return a long, like syscall().

The problem can be circumvented by the caller by coercing the returned
value to int before comparing it to -1:

    if ((int) syscall(number, ...) == -1)
        foo();

The same problem probably occurs on other 64-bit systems but so far only
x86_64 was tested, so this change must be considered experimental.

Signed-off-by: Carlos Santos <unixmania@gmail.com>

1 files changed, 2 insertions, 2 deletions

diff --git a/libpthread/nptl/sysdeps/unix/sysv/linux/__syscall_error.c b/libpthread/nptl/sysdeps/unix/sysv/linux/__syscall_error.c
index 5e109a83b..af26cf6ab 100644
--- a/libpthread/nptl/sysdeps/unix/sysv/linux/__syscall_error.c
+++ b/libpthread/nptl/sysdeps/unix/sysv/linux/__syscall_error.c
@@ -10,8 +10,8 @@
 
 /* This routine is jumped to by all the syscall handlers, to stash
  * an error number into errno.  */
-int __syscall_error(int err_no) attribute_hidden;
-int __syscall_error(int err_no)
+long __syscall_error(int err_no) attribute_hidden;
+long __syscall_error(int err_no)
 {
 	__set_errno(err_no);
 	return -1;