uClibc thread-safety analysis By Steve Thayer with updates by Erik Andersen Introduction: The purpose of this document is to identify the things that need to be done to the uClibc C library in order to make it thread-safe. The goal is to be able to use a pthreads thread implementation under uClinux, using the uClibc C library. To help identify the things that require changing, I used David R. Butenhof's book Programming With POSIX Threads, the source code for the glibc 2.1.3 C library, and the source code for the C library included in the Proventhreads distribution. References: Butenhof, David R., Programming With POSIX Threads, Addison Wesley Longman, Inc., Reading, MA, ISBN 0-201-63392-2, 1997. The GNU C library is available from the Free Software Foundation http://www.gnu.org/software/libc/libc.html Proventhreads is part of the Inferno Operating system. http://www.vitanuova.com/inferno/index.html 1. Stdio: 1.1 Buffer access mutexes The following functions are required in order to protect shared I/O buffers from being accessed by more than one thread at a time. None of these functions are currently implemented in the uClibc library, so they must be added. flockfile ftrylockfile funlockfile 1.2 Functions that must use buffer access mutexes, according to Butenhof The following functions are identified by Butenhof as needing to use buffer access mutexes. This does not represent all functions that need to use the mutexes. getc getchar putc putchar 1.3 Functions from glibc (libio) that use buffer access mutexes The following functions are functions found in glibc that currently use the buffer access mutexes. Comments in brackets represent status of uClibc with regard to these functions. Most of these functions aren't even supported under uClibc, so no work is required. The rest may require the access mutex. (These must be analyzed on a case-by-case basis.) clearerr feof ferror fputc fputwc freopen freopen64 fseek fseeko fseeko64 ftello ftello64 fwide getc getchar getwc getwchar iofclose iofflush iofgetpos iofgetpos64 iofgets iofgetws iofputs iofputws iofread iofsetpos iofsetpos64 ioftell iofwrite iogetdelim iogets ioputs ioseekoff ioseekpos iosetbuffer iosetvbuf ioungetc ioungetwc oldiofclose oldiofgetpos oldiofgetpos64 oldiofsetpos oldiofsetpos64 peekc putc putchar putwc putwchar rewind 1.4 Functions from Proventhreads that use buffer access mutexes See description above. This applies only to the C library included in the proventhreads distribution. clearerr fclose fflush fgetc __getc fgetline fgetpos fgetpos64 fgets fpurge fputc __putc fputs fread freopen fscanf fseek ftell fwalk fwrite getc getchar putc putchar puts putw refill rewind scanf setvbuf ungetc vfprintf vscanf 1.5 Unlocked buffer access These functions get used in situations where speed is important, and locking isn't necessary, or for use in a section of code that is already locked. For example, a for-loop that makes multiple calls to getc could make an exlicit call to flockfile outside the loop, and then use getc_unlocked within the loop for speed. That way, the lock only needs to be grabbed and released once, rather than for each call. getc_unlocked 2. Thread-safe functions: There are some functions in the C library standard that are simply not thread-safe and cannot be thread-safe using their current interface. For example, any function that returns a pointer to static data, or requires static context between calls. To resolve this problem, new functions were added to the standard that perform the same basic task, but use a new interface that does not require static data. These functions share their name with the old unsafe functions, but have an "_r" appended to the name. By definition, these functions are reentrant and thread-safe. It is important to note that these functions to do not depend on and can be used even if threading is not supported. 2.1 User and terminal identification: getlogin_r ctermid ttyname_r 2.2 Directory searching readdir_r 2.3 String token strtok_r 2.4 Time representation asctime_r ctime_r gmtime_r localtime_r 2.5 Random number generation rand_r <--- getgrnam_r <--- getpwuid_r getpwnam_r 2.7 Additional thread-safe functions implemented by glibc The following is a list of additional functions implemented by glibc that also provide thread-safe functionality. Most of these functions are not supported by uClibc, so there is no need to implement the thread-safe version. Those that do apply, but have not been implemented yet are highlighted. fgetpwent_r getttyname_r getmntent_r ecvt_r fcvt_r qecvt_r qfcvt_r hcreate_r hdestroy_r hsearch_r getspent_r getspnam_r sgetspent_r fgetspent_r gethostbyaddr_r gethostbyname2_r gethostbyname_r gethostent_r <--- getnetbyaddr_r <--- getnetent_r <--- getnetbyname_r <--- getprotobynumber_r <--- getprotoent_r <--- getprotobyname_r <--- getservbyname_r getservbyport_r getservent_r getrpcent_r <--- getrpcbyname_r <--- getrpcbynumber_r ether_aton_r ether_ntoa_r getnetgrent_r getaliasent_r getaliasbyname_r nscd_getpw_r nscd_getgr_r nscd_gethst_r getutent_r <--- getutent_r_unknown getutid_r_unknown getutline_r_unknown getutid_r <--- getutline_r <--- getutent_r_file getutid_r_file getutline_r_file internal_getut_r getutent_r_daemon getutid_r_daemon getutline_r_daemon ptsname_r __fgetpwent_r __ttyname_r __getmntent_r __getspent_r __getspnam_r __sgetspent_r __fgetspent_r __gethostbyaddr_r __gethostbyname2_r __gethostbyname_r __gethostent_r __getnetent_r __getnetbyname_r __getprotobynumber_r __getprotoent_r __getprotobyname_r __getservbyname_r __getservbyport_r __getrpcent_r __getservent_r __getrpcbyname_r __getrpcbynumber_r __getnetgrent_r __internal_getnetgrent_r __getaliasent_r __getaliasbyname_r __nscd_getpwnam_r __nscd_getpwuid_r __nscd_getgrgid_r __nscd_getgrnam_r __nscd_gethostbyaddr_r __nscd_gethostbyname2_r __nscd_gethostbyname_r __getutent_r __getutid_r __getutline_r __ptsname_r 2.8 Additional thread-safe functions implemented by Proventhreads See description above. This applies only to the C library included in the proventhreads distribution. inet_ntoa_r gethostbyaddr_r gethostbyname_r gethostent_r getnetbyaddr_r <--- getnetbyname_r <--- getnetent_r <--- getprotobynumber_r <--- getprotoent_r <--- getprotobyname_r <--- getservbyname_r getservbyport_r getservent_r 3. List of functions in uClibc that use static data structures The following is a list of static data structures found in uClibc, and the functions that use them. In most cases, static data is not thread-safe, since multiple threads can access the same data at the same time. This is an attempt to identify all of the static data that is not considered thread-safe. Some of these problems will get resolved by the changes outlines above. crypt/crypt.c: static struct crypt_data __crypt_data; crypt setkey encrypt -------------------------------------------------------------------- crypt/des.c: static u_char inv_key_perm[64]; static u_char inv_comp_perm[56]; static u_char u_sbox[8][64]; static u_char u_key_perm[56]; static u_char un_pbox[32]; static u_int32_t en_keysl[16], en_keysr[16]; static u_int32_t de_keysl[16], de_keysr[16]; static u_int32_t ip_maskl[8][256], ip_maskr[8][256]; static u_int32_t fp_maskl[8][256], fp_maskr[8][256]; static u_int32_t key_perm_maskl[8][128], key_perm_maskr[8][128]; static u_int32_t comp_maskl[8][128], comp_maskr[8][128]; static u_int32_t saltbits; __des_crypt_r <--- This needs to be fixed to actually be reentrant. Right now it just fakes it, which is pretty lame.... All the above should be moved into a 'struct des_context' or some such... -------------------------------------------------------------------- crypt/md5.c: static const char __md5__magic[]; static const unsigned char __md5_itoa64[]; __md5_crypt_r No problems here... -------------------------------------------------------------------- inet/addr.c: static char buf[16]; inet_ntoa inet_ntoa_r -------------------------------------------------------------------- inet/getnetent.c: static FILE *netf = NULL; static char line[BUFSIZ+1]; static struct netent net; static char *net_aliases[MAXALIASES]; setnetent endnetent getnetent getnetent_r NOTE: setnetent and endnetent are not implemented in glibc. -------------------------------------------------------------------- inet/getproto.c: static FILE *protof = NULL; static char line[BUFSIZ+1]; static struct protoent proto; static char *proto_aliases[MAXALIASES]; static int proto_stayopen; setprotoent endprotoent getprotoent getprotoent_r getprotobyname getprotobyname_r getprotobynumber getprotobynumber_r NOTE: setprotoent and endprotoent are not implemented in glibc. -------------------------------------------------------------------- inet/getservice.c: static FILE *servf = NULL; static char line[BUFSIZ+1]; static struct servent serv; static char *serv_aliases[MAXALIASES]; static int serv_stayopen; setservent endservent getservent getservent_r getservbyname getservbyname_r getservbyport getservbyport_r NOTE: setservent and endservent are not implemented in glibc. -------------------------------------------------------------------- static struct hostent h; static char buf[xxx]; gethostbyname static struct hostent h; static char buf[xxx]; gethostbyname2 static struct hostent h; static char buf[xxx]; gethostbyaddr static int __stay_open; static FILE * __gethostent_fp; static struct hostent h; static char buf[xxx]; gethostent sethostent endhostent -------------------------------------------------------------------- inet/rpc/auth_none.c: static struct auth_ops ops static struct authnone_private authnone_create <--- authnone_marshal <--- NOTE: This file makes a lot of use of this static variable and also a global allocated authentication structure. Care should be taken in fixing this to make it thread-safe. -------------------------------------------------------------------- inet/rpc/auth_unix.c: static struct auth_ops auth_unix_ops authunix_create <--- marshal_new_auth <--- NOTE: This file makes a lot of use of this static variable and also a global allocated authentication structure. Care should be taken in fixing this to make it thread-safe. -------------------------------------------------------------------- inet/rpc/bindresvport.c: static short port; bindresvport <--- -------------------------------------------------------------------- inet/rpc/clnt_perror.c: static char *buf; static struct rpc_errtab rpc_errlist[] static struct auth_errtab auth_errlist[] NOTE: These static structures all have #if 0 around them, so they do not get compiled in. Hopefully, we don't have to worry about them right now, but prehaps a comment should be added to the code indicating that it is not thread-safe. -------------------------------------------------------------------- inet/rpc/clnt_raw.c: static struct clntraw_private static struct clnt_ops client_ops clntraw_create <--- clntraw_call <--- clntraw_freeres <--- NOTE: This file makes a lot of use of these static variables and also a global allocated client structure. Care should be taken in fixing this to make it thread-safe. -------------------------------------------------------------------- inet/rpc/clnt_simple.c: static struct callrpc_private callrpc <--- -------------------------------------------------------------------- inet/rpc/clnt_tcp.c: static struct clnt_ops tcp_ops clnttcp_create NOTE: This static structure is just a group of function pointers. It could probably be made const, but this might affect the function signature. This should be investigated further. -------------------------------------------------------------------- inet/rpc/clnt_udp.c: static struct clnt_ops udp_ops clntudp_bufcreate NOTE: This static structure is just a group of function pointers. It could probably be made const, but this might affect the function signature. This should be investigated further. -------------------------------------------------------------------- inet/rpc/getrpcent.c: static char RPCDB[] <--- NOTE: This is okay, but should use the const keyword. -------------------------------------------------------------------- inet/rpc/pmap_clnt.c: static struct timeval timeout <--- static struct timeval tottimeout <--- NOTE: These are okay, but should use the const keyword. -------------------------------------------------------------------- inet/rpc/pmap_getport.c: static struct timeval timeout <--- static struct timeval tottimeout <--- NOTE: These are okay, but should use the const keyword. -------------------------------------------------------------------- inet/rpc/pmap_rmt.c: static struct timeval timeout <--- NOTE: This is okay, but should use the const keyword. -------------------------------------------------------------------- inet/rpc/rpc_dtablesize.c: static int size; _rpc_dtablesize <--- -------------------------------------------------------------------- inet/rpc/rpc_prot.c: static struct xdr_discrim reply_dscrm[3] <--- NOTE: This is okay, but should use the const keyword. -------------------------------------------------------------------- inet/rpc/svc.c: static SVCXPRT **xports; static SVCXPRT *xports[NOFILE]; static struct svc_callout xprt_register <--- xprt_unregister <--- svc_getreqset <--- svc_register <--- svc_unregister <--- svc_callout <--- NOTE: This is intricate code, and care should be taken when making this thread-safe. -------------------------------------------------------------------- inet/rpc/svc_auth.c: static struct svcauthsw <--- NOTE: This is okay, but should use the const keyword. -------------------------------------------------------------------- net/rpc/svc_raw.c: static struct svcraw_private static struct xp_ops server_ops svcraw_create <--- svcraw_recv <--- svcraw_reply <--- svcraw_getargs <--- svcraw_freeargs <--- NOTE: This is intricate code, and care should be taken when making this thread-safe. -------------------------------------------------------------------- inet/rpc/svc_simple.c: static struct proglst static SVCXPRT *transp; registerrpc <--- universal <--- NOTE: This is intricate code, and care should be taken when making this thread-safe. -------------------------------------------------------------------- inet/rpc/svc_tcp.c: static struct xp_ops svctcp_op static struct xp_ops svctcp_rendezvous_op svctcp_create <--- makefd_xprt <--- NOTE: This static structure is just a group of function pointers. It could probably be made const, but this might affect the function signature. This should be investigated further. static struct timeval wait_per_try readtcp <--- NOTE: This looks like a bug. This static timeout value is passed by reference to a select() call. According to the linux man page for select: "On Linux, timeout is modified to reflect the amount of time not slept; most other implementations do not do this. This causes problems both when Linux code which reads timeout is ported to other operating systems, and when code is ported to Linux that reuses a struct timeval for multiple selects in a loop without reinitializing it. Consider timeout to be undefined after select returns." Unless the implementation of select is different than that of Linux, this needs to be fixed! -------------------------------------------------------------------- inet/rpc/svc_udp.c: static struct xp_ops svcudp_op (1) svcudp_bufcreate 1: This static structure is just a group of function pointers. It could probably be made const, but this might affect the function signature. This should be investigated further. -------------------------------------------------------------------- inet/rpc/xdr.c: static char xdr_zero[BYTES_PER_XDR_UNIT] <--- NOTE: This is okay, but should use the const keyword. static u_long crud[BYTES_PER_XDR_UNIT] NOTE: This looks like it doesn't matter what's in this array. -------------------------------------------------------------------- inet/rpc/xdr_float.c: static struct sgl_limits <--- static struct dbl_limits <--- NOTE: These are okay, but should use the const keyword. -------------------------------------------------------------------- inet/rpc/xdr_mem.c: static struct xdr_ops xdrmem_ops (1) xdrmem_create 1: This static structure is just a group of function pointers. It could probably be made const, but this might affect the function signature. This should be investigated further. -------------------------------------------------------------------- inet/rpc/xdr_rec.c: static struct xdr_ops xdrrec_ops (1) xdrrec_create 1: This static structure is just a group of function pointers. It could probably be made const, but this might affect the function signature. This should be investigated further. -------------------------------------------------------------------- inet/rpc/xdr_stdio.c: static struct xdr_ops xdrstdio_ops (1) xdrstdio_create 1: This static structure is just a group of function pointers. It could probably be made const, but this might affect the function signature. This should be investigated further. -------------------------------------------------------------------- ld.so-1/d-link/boot1.c: static char *_dl_malloc_addr static char *_dl_mmap_zero static char *_dl_not_lazy static char *_dl_warn static char *_dl_trace_loaded_objects _dl_boot <--- _dl_malloc <--- _dl_fixup <--- These are all part of the shared library loader, and are not used by applications directly. Therefore, fixing these is not a high priority item. -------------------------------------------------------------------- ld.so-1/d-link/readelflib1.c: static caddr_t _dl_cache_addr static size_t _dl_cache_size _dl_map_cache <--- _dl_unmap_cache <--- _dl_load_shared_library <--- These are all part of the shared library loader, and are not used by applications directly. Therefore, fixing these is not a high priority item. -------------------------------------------------------------------- ld.so-1/d-link/string.h: static char local[22] _dl_simple_ltoa <--- _dl_simple_ltoahex <--- These are all part of the shared library loader, and are not used by applications directly. Therefore, fixing these is not a high priority item. -------------------------------------------------------------------- ld.so-1/d-link/arm/elfinterp.c: static char *_dl_reltypes[] <--- NOTE: This is okay, but should use the const keyword. -------------------------------------------------------------------- ld.so-1/d-link/i386/elfinterp.c: static char *_dl_reltypes[] <--- NOTE: This is okay, but should use the const keyword. These are all part of the shared library loader, and are not used by applications directly. Therefore, fixing these is not a high priority item. -------------------------------------------------------------------- ld.so-1/d-link/m68k/elfinterp.c: static char *_dl_reltypes[] <--- NOTE: This is okay, but should use the const keyword. These are all part of the shared library loader, and are not used by applications directly. Therefore, fixing these is not a high priority item. -------------------------------------------------------------------- ld.so-1/d-link/sparc/elfinterp.c: static char *_dl_reltypes[] <--- NOTE: This is okay, but should use the const keyword. These are all part of the shared library loader, and are not used by applications directly. Therefore, fixing these is not a high priority item. -------------------------------------------------------------------- ld.so-1/libdl/dlib.c: static int dl_init _dlopen <--- static int __attribute__ ((unused)) foobar1 (1) NOTE: The comment for this says it all: "This is a real hack." ;-) static char *type[] <--- NOTE: This is okay, but should use the const keyword. These are all part of the shared library loader, and are not used by applications directly. Therefore, fixing these is not a high priority item. -------------------------------------------------------------------- ld.so-1/util/ldconfig.c: static header_t magic <--- NOTE: This is okay, but should use the const keyword. static liblist_t *lib_head cache_dolib <--- cache_write <--- This is not actually part of the C library, and is not built by default, so fixing these is not a high priority item. -------------------------------------------------------------------- misc/internals/tempname.c: static uint64_t value; __gen_tempname This is supposed to be random trash, so the more threading collisions the better... -------------------------------------------------------------------- misc/locale/locale.c: static char C_LOCALE_NAME[]="C"; <--- NOTE: This is okay, but should use the const keyword. static struct SAV_LOADED_LOCALE sav_loaded_locale [1] static struct SAV_LOADED_LOCALE * old_locale setlocale <--- NOTE: Can different threads use different locales? I don't see why not. -------------------------------------------------------------------- misc/locale/localeconv.c: static struct lconv result; localeconv <--- NOTE: This function returns a pointer to a static data structure. static char *blank = ""; <--- static char *decimal = "."; <--- NOTE: These are okay, but should use the const keyword. -------------------------------------------------------------------- misc/mntent/mntent.c: static char buff[MNTMAXSTR]; static struct mntent mnt; getmntent getmntent_r -------------------------------------------------------------------- misc/regex/regex.c: static char re_syntax_table[CHAR_SET_SIZE]; static int done = 0; init_syntax_table <--- static int debug; <--- NOTE: This is used to turn on debugging, and is used in several functions. It will need to be fixed if you want differing debug levels per thread. static char reg_unset_dummy; <--- static fail_stack_type fail_stack; regex_compile <--- <--- <--- <--- <--- <--- <--- <--- <--- <--- <--- <--- <--- <--- <--- static int regs_allocated_size; regs_grow_registers <--- static register_info_type *reg_info; static register_info_type *reg_info_dummy; static unsigned failure_id; static struct re_pattern_buffer re_comp_buf; <--- NOTE: This is just a NASTY file for static variables. A lot of work needs to be done here to clean this up. But I'm not even sure if it matters. This code is taken directly from glibc. This code is also very large (adds over 30k to the C library all by itself). This file needs a complete rewrite. -------------------------------------------------------------------- misc/syslog/syslog.c: static pthread_once__t _once_block = pthread_once_init; static pthread_mutex_t _syslog_mutex; NOTE: I think these are okay. ;-) static int LogFile = -1; static int connected; static int LogStat = 0; static int LogFacility = LOG_USER; static int LogMask = 0xff; static char truncate_msg[12] static struct sockaddr SyslogAddr; NOTE: These are already protected. -------------------------------------------------------------------- misc/time/asctime.c: static char timebuf[26]; asctime -------------------------------------------------------------------- misc/time/ctime.c: static char cbuf[26]; ctime -------------------------------------------------------------------- misc/time/gmtime.c: static struct tm tmb; gmtime -------------------------------------------------------------------- misc/time/localtime.c: static struct tm tmb; localtime -------------------------------------------------------------------- misc/time/mktime.c: static tz_rule tz_rules[2]; tzset <--- static time_t localtime_offset; mktime <--- -------------------------------------------------------------------- misc/time/tm_conv.c: static int moffset[] <--- NOTE: This is okay, but should use the const keyword. -------------------------------------------------------------------- misc/utmp/utent.c: static int static_fd = -1; static struct utmp static_utmp; static const char default_file_name[] = _PATH_UTMP; static const char *static_ut_name = (const char *) default_file_name; setutent endutent getutent getutid getutline pututline utmpname __getutent -------------------------------------------------------------------- pwd_grp/__getgrent.c: static char line_buff[GR_MAX_LINE_LEN]; static char *members[GR_MAX_MEMBERS]; static char *line_buff = NULL; static char **members = NULL; static struct group group; __getgrent <--- -------------------------------------------------------------------- pwd_grp/fgetpwent.c: static char line_buff[PWD_BUFFER_SIZE]; static struct passwd pwd; fgetpwent -------------------------------------------------------------------- pwd_grp/getpwnam.c: static char line_buff[PWD_BUFFER_SIZE]; static struct passwd pwd; getpwnam -------------------------------------------------------------------- pwd_grp/getpwuid.c: static char line_buff[PWD_BUFFER_SIZE]; static struct passwd pwd; getpwuid -------------------------------------------------------------------- pwd_grp/grent.c: static int grp_fd = -1; setgrent <--- endgrent <--- getgrent <--- -------------------------------------------------------------------- pwd_grp/pwent.c: static int pw_fd = -1; setpwent <--- endpwent <--- getpwent_r <--- NOTE: Yeah, this looks weird, but getpwent_r isn't completely thread-safe. static char line_buff[PWD_BUFFER_SIZE]; static struct passwd pwd; getpwent <--- -------------------------------------------------------------------- stdio/tmpnam.c: static char tmpnam_buffer[L_tmpnam]; tmpnam -------------------------------------------------------------------- stdlib/atexit.c: static vfuncp __atexit_table[__UCLIBC_MAX_ATEXIT]; static int __atexit_count = 0; atexit_handler <--- atexit <--- -------------------------------------------------------------------- stdlib/bsearch.c: static int _bsearch; bsearch -------------------------------------------------------------------- stdlib/setenv.c: static char **last_environ; __add_to_environ unsetenv setenv clearenv putenv all ok now. -------------------------------------------------------------------- stdlib/random.c: static long int seed1 = 1; static long int seed2 = 1; static long int seed3 = 1; random (1) srandom (1) 1: I'm not sure if it matters if these are static, since they are random number seeds. Who cares if more than one thread changes their value? -------------------------------------------------------------------- stdlib/malloc/avlmacro.h static objname *__Avl_##objname##pr##_new_node; Avl_Tree_no_replace <--- NOTE: This will take a bit of study to figure out if it needs fixing. -------------------------------------------------------------------- stdlib/malloc/malloc.c: //static mutex_t malloc_lock = MUTEX_INITIALIZER; <--- NOTE: Basically, thread support in malloc is broken and must be fixed. It looks like the infrastructure is there, but more investigation is required. -------------------------------------------------------------------- stdlib/malloc-930716/malloc.c: A bunch of stuff malloc realloc free etc -------------------------------------------------------------------- string/strerror.c: static char retbuf[48]; static char retbuf[33]; strerror <--- strerror_r <--- -------------------------------------------------------------------- string/strsignal.c: static char retbuf[28]; strsignal <--- main <--- -------------------------------------------------------------------- string/strtok.c: static char *save = 0; strtok strtok_r -------------------------------------------------------------------- termios/tcgetsid.c: static int tiocgsid_does_not_work; tcgestsid <--- -------------------------------------------------------------------- termios/ttyname.c: static char dev[] = "/dev"; ttyname <--- NOTE: This is okay, but should use the const keyword. static char name[NAME_MAX]; ttyname <--- -------------------------------------------------------------------- test/testsuite.h: static int failures error_msg <--- done_testing <--- init_testsuite <--- -------------------------------------------------------------------- unistd/getopt.c: static int sp = 1; getopt