Modify

Opened 5 years ago

Closed 4 years ago

Last modified 4 years ago

#13024 closed defect (fixed)

uClibc's (v)asprintf wastes a lot of memory

Reported by: anonymous Owned by: developers
Priority: normal Milestone: Barrier Breaker 14.07
Component: packages Version: Trunk
Keywords: Cc:

Description

I'd had some OOM problem during some tests of programs using asprintf on OpenWRT. This was relative unexpected because they use around 32 KiB of memory (heap) on eglibc. Problem seems to be the glibc compat UCLIBC_HAS_GLIBC_CUSTOM_STREAMS. Enabling it also enables an alternative implementation of (v)asprintf using the streams.

This problem can also be reproduced on PC (even with the current git version v0.9.32-rc3-854-g0eb3076) with a small program (see attached file). The expected memory usage would be ~40 Bytes, but the program needs over 8000 Bytes (over 4000 per string).

Disabling the open_memstream implementation in ./libc/stdio/vasprintf.c worked around this problem for me (did it by replacing "#ifdef __UCLIBC_HAS_GLIBC_CUSTOM_STREAMS__" with "#ifdef DISABLED_AS_BUG_WORKAROUND__UCLIBC_HAS_GLIBC_CUSTOM_STREAMS__")

Attachments (1)

malloc.c (398 bytes) - added by anonymous 5 years ago.

Download all attachments as: .zip

Change History (4)

Changed 5 years ago by anonymous

comment:1 Changed 4 years ago by anonymous

malloc.c compiles when I add an include for malloc.h

Issue seems valid (tested with AA on ar71xx)

The reason for this is that vasprintf (used by asprintf) uses open_memstream which allocates memory BUFSIZ at a time. The size of BUFSIZ can be configured and is set to 4096 in my build environment.

$ grep -n "open_memstream" attitude_adjustment-12.09/build_dir/toolchain-mips_r2_gcc-4.6-linaro_uClibc-0.9.33.2/uClibc-0.9.33.2/libc/stdio/vasprintf.c 
18:#warning CONSIDER: avoid open_memstream call?
36:     if ((f = open_memstream(buf, &size)) != NULL) {
$ grep -nA 7 "open_memstream" attitude_adjustment-12.09/build_dir/toolchain-mips_r2_gcc-4.6-linaro_uClibc-0.9.33.2/uClibc-0.9.33.2/libc/stdio/open_memstream.c 
131:FILE *open_memstream(char **__restrict bufloc, size_t *__restrict sizeloc)
132-{
133-    register __oms_cookie *cookie;
134-    register FILE *fp;
135-
136-    if ((cookie = malloc(sizeof(__oms_cookie))) != NULL) {
137-            if ((cookie->buf = malloc(cookie->len = BUFSIZ)) == NULL) {
138-                    goto EXIT_cookie;
--
165:libc_hidden_def(open_memstream)
166-#endif
$ grep -n STDIO_BUFSIZ attitude_adjustment-12.09/build_dir/toolchain-mips_r2_gcc-4.6-linaro_uClibc-0.9.33.2/uClibc-0.9.33.2/.config
181:# UCLIBC_HAS_STDIO_BUFSIZ_NONE is not set
182:# UCLIBC_HAS_STDIO_BUFSIZ_256 is not set
183:# UCLIBC_HAS_STDIO_BUFSIZ_512 is not set
184:# UCLIBC_HAS_STDIO_BUFSIZ_1024 is not set
185:# UCLIBC_HAS_STDIO_BUFSIZ_2048 is not set
186:UCLIBC_HAS_STDIO_BUFSIZ_4096=y
187:# UCLIBC_HAS_STDIO_BUFSIZ_8192 is not set

So the allocation of memory would be less if a smaller BUFSIZ would be selected. But the problem seems to be that unused heap allocated memory by open_memstream can not be used by further calls to that function. Maybe adding a malloc_trim(0);

In uClibc-0.9.33.2/docs/Glibc_vs_uClibc_Differences.txt, it says under section stdio:

3) BUFSIZ is configurable, but no attempt is made at automatic tuning of internal
   buffer sizes for stdio streams.  In fact, the stdio code in general sacrifices
   sophistication/performace for minimal size.

comment:2 Changed 4 years ago by nbd

  • Resolution set to fixed
  • Status changed from new to closed

fixed in r40774

comment:3 Changed 4 years ago by jow

  • Milestone changed from Attitude Adjustment 12.09 to Barrier Breaker 14.07

Milestone Attitude Adjustment 12.09 deleted

Add Comment

Modify Ticket

Action
as closed .
The resolution will be deleted. Next status will be 'reopened'.
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.