Modify

Opened 8 years ago

Closed 7 years ago

Last modified 4 years ago

#6418 closed defect (fixed)

memory leak on brcm47xx

Reported by: acoul <alex at ozo.com> Owned by: hauke
Priority: high Milestone: Barrier Breaker 14.07
Component: kernel Version:
Keywords: memleak memory leak brcm47xx slab Cc:

Description

this is an issue that at some time was resolved, I am not sure if it's a code issue or config option, and reappeared again. Russell Senior has been beaten by this here

On trunk r18920, jffs2 image with gcc-4.4.2, binutils-2.20 on kernel 2.6.32.2, without the usb and the led kernel options and no extra drivers loaded here is what I get:

BusyBox v1.15.3 (2009-12-24 12:46:31 EET) built-in shell (ash)
Enter 'help' for a list of built-in commands.

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 KAMIKAZE (bleeding edge, r18920) ------------------
  * 10 oz Vodka       Shake well with ice and strain
  * 10 oz Triple sec  mixture into 10 shot glasses.
  * 10 oz lime juice  Salute!
 ---------------------------------------------------
root@OpenWrt:/# free
              total         used         free       shared      buffers
  Mem:        13676        12912          764            0            0
 Swap:            0            0            0
Total:        13676        12912          764
root@OpenWrt:/# dmesg 
Linux version 2.6.32.2 (alex@motard) (gcc version 4.4.2 (GCC) ) #2 Thu Dec 24 13:24:12 EET 2009
CPU revision is: 00024000 (Broadcom BCM4710)
ssb: Core 0 found: SDRAM (cc 0x803, rev 0x00, vendor 0x4243)
ssb: Core 1 found: ILine 20 (cc 0x801, rev 0x00, vendor 0x4243)
ssb: Core 2 found: Fast Ethernet (cc 0x806, rev 0x00, vendor 0x4243)
ssb: Core 3 found: V90 (cc 0x807, rev 0x00, vendor 0x4243)
ssb: Core 4 found: USB 1.1 Hostdev (cc 0x808, rev 0x00, vendor 0x4243)
ssb: Core 5 found: PCI (cc 0x804, rev 0x00, vendor 0x4243)
ssb: Core 6 found: MIPS (cc 0x805, rev 0x00, vendor 0x4243)
ssb: Core 7 found: EXTIF (cc 0x811, rev 0x00, vendor 0x4243)
ssb: Core 8 found: Fast Ethernet (cc 0x806, rev 0x00, vendor 0x4243)
ssb: Initializing MIPS core...
ssb: set_irq: core 0x0806, irq 4 => 4
ssb: set_irq: core 0x0801, irq 5 => 2
ssb: set_irq: core 0x0806, irq 7 => 5
ssb: after irq reconfiguration
ssb: core 0x0803, irq : 2(S)  3  4  5  6  D  I*
ssb: core 0x0801, irq : 2(S)* 3  4  5  6  D  I 
ssb: core 0x0806, irq : 2(S)  3  4* 5  6  D  I 
ssb: core 0x0807, irq : 2(S)  3  4  5  6  D* I 
ssb: core 0x0808, irq : 2(S)  3* 4  5  6  D  I 
ssb: core 0x0804, irq : 2(S)  3  4  5  6* D  I 
ssb: core 0x0805, irq : 2(S)  3  4  5  6  D  I*
ssb: core 0x0811, irq : 2(S)  3  4  5  6  D* I 
ssb: core 0x0806, irq : 2(S)  3  4  5* 6  D  I 
ssb: Sonics Silicon Backplane found at address 0x18000000
Serial init done.
Determined physical RAM map:
 memory: 01000000 @ 00000000 (usable)
Initrd not found or empty - disabling initrd
Zone PFN ranges:
  Normal   0x00000000 -> 0x00001000
Movable zone start PFN for each node
early_node_map[1] active PFN ranges
    0: 0x00000000 -> 0x00001000
On node 0 totalpages: 4096
free_area_init_node: node 0, pgdat 8025f260, node_mem_map 802a1000
  Normal zone: 32 pages used for memmap
  Normal zone: 0 pages reserved
  Normal zone: 4064 pages, LIFO batch:0
Built 1 zonelists in Zone order, mobility grouping off.  Total pages: 4064
Kernel command line: root=/dev/mtdblock2 rootfstype=squashfs,jffs2 noinitrd console=ttyS0,115200
PID hash table entries: 64 (order: -4, 256 bytes)
Dentry cache hash table entries: 2048 (order: 1, 8192 bytes)
Inode-cache hash table entries: 1024 (order: 0, 4096 bytes)
Enabling BCM4710A0 cache workarounds.
Primary instruction cache 8kB, VIPT, 2-way, linesize 16 bytes.
Primary data cache 4kB, 2-way, VIPT, no aliases, linesize 16 bytes
Memory: 13548k/16384k available (2032k kernel code, 2836k reserved, 393k data, 128k init, 0k highmem)
SLUB: Genslabs=7, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Hierarchical RCU implementation.
NR_IRQS:128
Calibrating delay loop... 82.43 BogoMIPS (lpj=164864)
Mount-cache hash table entries: 512
devtmpfs: initialized
NET: Registered protocol family 16
bio: create slab <bio-0> at 0
Switching to clocksource MIPS
Switched to NOHz mode on CPU #0
ssb: PCIcore in host mode found
PCI: Fixing up bridge 0000:00:00.0
PCI: Setting latency timer of device 0000:00:00.0 to 64
PCI: Fixing up device 0000:00:00.0
PCI: Fixing latency timer of device 0000:00:00.0 to 168
pci 0000:00:00.0: reg 10 32bit mmio: [0x000000-0x001fff]
pci 0000:00:00.0: reg 14 32bit mmio pref: [0x000000-0x7ffffff]
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 512 (order: 0, 4096 bytes)
TCP bind hash table entries: 512 (order: -1, 2048 bytes)
TCP: Hash tables configured (established 512 bind 512)
TCP reno registered
NET: Registered protocol family 1
squashfs: version 4.0 (2009/01/31) Phillip Lougher
Registering mini_fo version $Id$
JFFS2 version 2.2. (NAND) (SUMMARY)  © 2001-2006 Red Hat, Inc.
msgmni has been set to 26
alg: No test for stdrng (krng)
io scheduler noop registered
io scheduler deadline registered (default)
Serial: 8250/16550 driver, 2 ports, IRQ sharing enabled
flash init: 0x1fc00000 0x00400000
Physically mapped flash: Found 1 x16 devices at 0x0 in 16-bit bank
 Amd/Fujitsu Extended Query Table at 0x0040
number of CFI chips: 1
cfi_cmdset_0002: Disabling erase-suspend-program due to code brokenness.
Flash device: 0x400000 at 0x1fc00000
bootloader size: 262144
Creating 4 MTD partitions on "Physically mapped flash":
0x000000000000-0x000000040000 : "cfe"
0x000000040000-0x0000003f0000 : "linux"
0x000000110000-0x0000003f0000 : "rootfs"
mtd: partition "rootfs" set to be root filesystem
split_squashfs: no squashfs found in "Physically mapped flash"
0x0000003f0000-0x000000400000 : "nvram"
b44.c:v2.0
eth0: Broadcom 44xx/47xx 10/100BaseT Ethernet 00:90:96:45:03:e8
b44: eth1: BUG!  Timeout waiting for bit 00000002 of register 42c to clear.
eth1: Broadcom 44xx/47xx 10/100BaseT Ethernet 00:90:96:45:03:e9
BCM47xx Watchdog Timer enabled (30 seconds, nowayout)
TCP westwood registered
NET: Registered protocol family 17
802.1Q VLAN Support v1.8 Ben Greear <greearb@candelatech.com>
All bugs added by David S. Miller <davem@redhat.com>
VFS: Mounted root (jffs2 filesystem) readonly on device 31:2.
devtmpfs: mounted
Freeing unused kernel memory: 128k freed
Please be patient, while OpenWrt loads ...
b44: eth0: Link is up at 100 Mbps, full duplex.
b44: eth0: Flow control is off for TX and off for RX.
b44: eth0: powering down PHY
b44: eth0: Link is up at 100 Mbps, full duplex.
b44: eth0: Flow control is off for TX and off for RX.
root@OpenWrt:/# cat /proc/meminfo 
MemTotal:          13676 kB
MemFree:             876 kB
Buffers:               0 kB
Cached:              692 kB
SwapCached:            0 kB
Active:              600 kB
Inactive:            668 kB
Active(anon):        104 kB
Inactive(anon):      504 kB
Active(file):        496 kB
Inactive(file):      164 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:           588 kB
Mapped:              456 kB
Shmem:                32 kB
Slab:              10960 kB
SReclaimable:        128 kB
SUnreclaim:        10832 kB
KernelStack:         192 kB
PageTables:          120 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:        6836 kB
Committed_AS:       2152 kB
VmallocTotal:    1048372 kB
VmallocUsed:         316 kB
VmallocChunk:    1048052 kB
root@OpenWrt:/# cat /proc/interrupts 
           CPU0       
  4:      11672            MIPS  eth0
  7:    5411506            MIPS  timer

ERR:          0
root@OpenWrt:/# uptime
 17:51:43 up 1 day, 17:51, load average: 0.07, 0.09, 0.08

Attachments (0)

Change History (15)

comment:1 Changed 8 years ago by acoul <alex at ozo.com>

here is the relative older thread about this issue on the Greek openwrt forum.

comment:2 Changed 8 years ago by acoul <alex at ozo.com>

this ticket is also valid for latest brcm47xx snapshot. it affects both jffs2 & squashfs images. to recreate, just flash a device and let it do nothing for 2-3 days. After that the device will die from memory starvation.

Latest working/tested image was r17456 running BusyBox v1.13.4

Linux version 2.6.30.5 (alex@motard) (gcc version 4.3.4 (GCC) ) #1 Tue Sep 1 09:43:39 EEST 2009

I've tested BusyBox v1.13.4 on latest trunk but the issue remains. I've also tested kernel 2.6.31.10 and the problem remains.

here are the running processes:

root@OpenWrt:/# ps auxwww
  PID USER       VSZ STAT COMMAND
    1 root      1348 S    init       
    2 root         0 SW   [kthreadd]
    3 root         0 SW   [ksoftirqd/0]
    4 root         0 SW   [events/0]
    5 root         0 SW   [khelper]
    8 root         0 SW   [async/mgr]
   23 root         0 SW   [sync_supers]
   25 root         0 SW   [bdi-default]
   27 root         0 SW   [kblockd/0]
   58 root         0 SW   [kswapd0]
   59 root         0 SW   [aio/0]
   60 root         0 SW   [crypto/0]
   71 root         0 SW   [mtdblockd]
  181 root         0 SWN  [jffs2_gcd_mtd2]
  342 root      1124 S    /usr/sbin/dropbear -p 22 -P /var/run/dropbear.1.pid 
  345 root      1340 S    telnetd -l /bin/login.sh 
11164 root      1352 S    /bin/ash --login 
11634 root         0 Z    [ash]
11639 root      1340 R    ps auxwww 

comment:3 Changed 8 years ago by acoul <alex at ozo.com>

this is how the router becomes before it collapses from memory starvation:

BusyBox v1.15.3 (2009-12-29 13:31:01 EET) built-in shell (ash)
Enter 'help' for a list of built-in commands.

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 KAMIKAZE (bleeding edge, r18957) ------------------
  * 10 oz Vodka       Shake well with ice and strain
  * 10 oz Triple sec  mixture into 10 shot glasses.
  * 10 oz lime juice  Salute!
 ---------------------------------------------------
root@OpenWrt:/# ps auxwww
  PID USER       VSZ STAT COMMAND
    1 root      1348 S    init       
    2 root         0 SW   [kthreadd]
    3 root         0 SW   [ksoftirqd/0]
    4 root         0 SW   [events/0]
    5 root         0 SW   [khelper]
    8 root         0 SW   [async/mgr]
   23 root         0 SW   [sync_supers]
   25 root         0 SW   [bdi-default]
   27 root         0 SW   [kblockd/0]
   61 root         0 DW   [kswapd0]
   62 root         0 SW   [aio/0]
   63 root         0 SW   [crypto/0]
   74 root         0 SW   [mtdblockd]
  181 root         0 SWN  [jffs2_gcd_mtd2]
  342 root      1124 S    /usr/sbin/dropbear -p 22 -P /var/run/dropbear.1.pid 
  345 root      1344 S    telnetd -l /bin/login.sh 
30809 root      1344 S    /bin/ash --login 
30832 root      1340 R    ps auxwww 
root@OpenWrt:/# cat /proc/meminfo 

MemTotal:          13652 kB
MemFree:             664 kB
Buffers:               0 kB
Cached:              644 kB
SwapCached:            0 kB
Active:              316 kB
Inactive:            576 kB
Active(anon):        116 kB
Inactive(anon):      236 kB
Active(file):        200 kB
Inactive(file):      340 kB
Unevictable:          92 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:           352 kB
Mapped:              248 kB
Shmem:                 0 kB
Slab:              11496 kB
SReclaimable:        168 kB
SUnreclaim:        11328 kB
KernelStack:         152 kB
PageTables:           72 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:        6824 kB
Committed_AS:       1348 kB
VmallocTotal:    1048372 kB
VmallocUsed:         316 kB
VmallocChunk:    1048052 kB

Slab and SUnreclaim are way too high here.

root@OpenWrt:/# uptime

 19:27:13 up 1 day, 19:27, load average: 3.19, 1.71, 0.73
root@OpenWrt:/# cat /proc/interrupts 

           CPU0       
  4:     135725            MIPS  eth0
  7:    2695276            MIPS  timer

ERR:          0

comment:4 Changed 8 years ago by acoul <alex at ozo.com>

this thread has some good input on how to trace memory leaks.

comment:5 Changed 8 years ago by acoul <alex at ozo.com>

here is another pointer about this issue

comment:6 Changed 8 years ago by acoul <alex at ozo.com>

root@OpenWrt:/# uptime

 20:14:27 up 20:14, load average: 0.16, 0.22, 0.18
root@OpenWrt:/# cat /proc/interrupts 

           CPU0       
  4:       8452            MIPS  eth0
  7:    3376327            MIPS  timer

ERR:          0
root@OpenWrt:/# cat /proc/softirqs 

                CPU0       
      HI:          0
   TIMER:    3305247
  NET_TX:      15229
  NET_RX:       8507
   BLOCK:          0
BLOCK_IOPOLL:          0
 TASKLET:          0
   SCHED:          0
 HRTIMER:         58
     RCU:    1435777
root@OpenWrt:/# cat /proc/pagetypeinfo 

Page block order: 10
Pages per block:  1024

Free pages count per migrate type at order       0      1      2      3      4      5      6      7      8      9     10 
Node    0, zone   Normal, type    Unmovable      7      8      4      1      2      0      1      0      1      1      0 
Node    0, zone   Normal, type  Reclaimable      0      0      0      0      0      0      0      0      0      0      0 
Node    0, zone   Normal, type      Movable      0      0      0      0      0      0      0      0      0      0      0 
Node    0, zone   Normal, type      Reserve      0      0      0      0      0      0      0      0      0      0      0 
Node    0, zone   Normal, type      Isolate      0      0      0      0      0      0      0      0      0      0      0 

Number of blocks type     Unmovable  Reclaimable      Movable      Reserve      Isolate 
Node 0, zone   Normal            4            0            0            0            0 
root@OpenWrt:/# cat /proc/vmstat 

nr_free_pages 910
nr_inactive_anon 0
nr_active_anon 81
nr_inactive_file 255
nr_active_file 235
nr_unevictable 0
nr_mlock 0
nr_anon_pages 84
nr_mapped 131
nr_file_pages 490
nr_dirty 0
nr_writeback 0
nr_slab_reclaimable 68
nr_slab_unreclaimable 1780
nr_page_table_pages 15
nr_kernel_stack 19
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 0
root@OpenWrt:/# cat /proc/slab_allocators 

jffs2_inode_cache: 1 800ddca4
jffs2_inode_cache: 319 800de198
jffs2_node_frag: 497 800d8fa0
jffs2_refblock: 78 800d9824
jffs2_full_dnode: 582 800dbc30
jffs2_full_dnode: 2 800dd3d4
jffs2_i: 142 800e3bb0
UNIX: 2 8016aab0
tcp_bind_bucket: 2 801ae560
ip_fib_hash: 8 801e0848
ip_dst_cache: 11 80180ba0
arp_cache: 1 801844bc
RAW: 2 8016aab0
TCP: 3 8016aab0
blkdev_queue: 1 800fb7f4
blkdev_requests: 4 8005bb9c
bio-0: 2 8005bb9c
biovec-256: 2 8005bb9c
sock_inode_cache: 11 80165148
skbuff_head_cache: 200 80170b14
proc_inode_cache: 91 800c58a8
radix_tree_node: 29 8010a35c
bdev_cache: 3 800b46d8
sysfs_dir_cache: 745 800d31e4
mnt_cache: 15 8009e3a0
filp: 38 80085c00
inode_cache: 240 8009a610
dentry: 534 8009886c
vm_area_struct: 41 8001f21c
vm_area_struct: 6 800759fc
vm_area_struct: 3 80076658
vm_area_struct: 45 80076a74
vm_area_struct: 3 8008b108
mm_struct: 3 8001ea80
mm_struct: 2 8001f0dc
fs_cache: 4 800a8f04
files_cache: 5 8009b404
signal_cache: 19 8001fe18
sighand_cache: 18 8001fdcc
sighand_cache: 1 8008a6c4
task_struct: 19 8001f7f4
cred_jar: 27 80040f08
anon_vma: 45 8007a170
pid: 73396 80036bbc
idr_layer_cache: 83 801066dc
root@OpenWrt:/# cat /proc/slabinfo 
slabinfo - version: 2.1 (statistics)
# name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail> : globalstat <listallocs> <maxobjs> <grown> <reaped> <error> <maxfreeable> <nodeallocs> <remotefrees> <alienoverflow> : cpustat <allochit> <allocmiss> <freehit> <freemiss>
jffs2_inode_cache    320    390     48   78    1 : tunables   32   16    0 : slabdata      5      5      0 : globalstat     344    335     5    0                                  0    0    0    0    0 : cpustat    298     22      0      0
jffs2_node_frag      497    546     48   78    1 : tunables   32   16    0 : slabdata      7      7      0 : globalstat     622    512     7    0                                  0    0    0    0    0 : cpustat    771     41    313      2
jffs2_refblock        78     84    272   14    1 : tunables   32   16    0 : slabdata      6      6      0 : globalstat      84     84     6    0                                  0    0    0    0    0 : cpustat     72      6      0      0
jffs2_tmp_dnode        0      0     56   67    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat     474    131     8    8                                  0    1    0    0    0 : cpustat    839     31    849     21
jffs2_raw_inode        0      0     96   40    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat      80     16     5    5                                  0    0    0    0    0 : cpustat    495      5    500      0
jffs2_raw_dirent       0      0     64   59    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat      16     16     1    1                                  0    0    0    0    0 : cpustat      1      1      2      0
jffs2_full_dnode     584    644     40   92    1 : tunables   32   16    0 : slabdata      7      7      0 : globalstat     714    599     7    0                                  0    0    0    0    0 : cpustat    826     47    288      1
jffs2_i              142    144    328   12    1 : tunables   32   16    0 : slabdata     12     12      0 : globalstat     188    144    12    0                                  0    0    0    0    0 : cpustat    123     20      1      0
kioctx                 0      0    184   21    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0                                  0    0    0    0    0 : cpustat      0      0      0      0
kiocb                  0      0    144   27    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0                                  0    0    0    0    0 : cpustat      0      0      0      0
fasync_cache           0      0     40   92    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0                                  0    0    0    0    0 : cpustat      0      0      0      0
nsproxy                0      0     48   78    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0                                  0    0    0    0    0 : cpustat      0      0      0      0
posix_timers_cache      0      0    128   30    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0                                 0    0    0    0    0 : cpustat      0      0      0      0
uid_cache              0      0     64   59    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0                                  0    0    0    0    0 : cpustat      0      0      0      0
UNIX                  10     10    384   10    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      10     10     1    0                                  0    0    0    0    0 : cpustat 440343      1 440342      0
ip_mrt_cache           0      0    104   37    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0                                  0    0    0    0    0 : cpustat      0      0      0      0
UDP-Lite               0      0    480    8    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0                                  0    0    0    0    0 : cpustat      0      0      0      0
tcp_bind_bucket        2     78     48   78    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      16     16     1    0                                  0    0    0    0    0 : cpustat      1      1      0      0
inet_peer_cache        0      0     72   53    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat      32     16     2    2                                  0    0    0    0    0 : cpustat      0      2      2      0
ip_fib_alias           0      0     40   92    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0                                  0    0    0    0    0 : cpustat      0      0      0      0
ip_fib_hash            8     59     64   59    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      32     16     1    0                                  0    0    0    0    0 : cpustat     18      2     12      0
ip_dst_cache          14     28    272   14    1 : tunables   32   16    0 : slabdata      2      2      0 : globalstat   14651     28    32   30                                  0    0    0    0    0 : cpustat    390   1360   1736      0
arp_cache              1     26    152   26    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat     256     17    15   14                                  0    0    0    0    0 : cpustat      1     16     16      0
RAW                    2      8    464    8    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat       8      8     1    0                                  0    0    0    0    0 : cpustat      1      1      0      0
UDP                    0      0    480    8    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat      24      8     3    3                                  0    0    0    0    0 : cpustat     19      3     22      0
tw_sock_TCP            0      0    128   30    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat      32     16     2    2                                  0    0    0    0    0 : cpustat      0      2      2      0
request_sock_TCP       0      0     96   40    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat      64     16     4    4                                  0    0    0    0    0 : cpustat      0      4      4      0
TCP                    3      3   1064    3    1 : tunables   24   12    0 : slabdata      1      1      0 : globalstat       9      3     1    0                                  0    0    0    0    0 : cpustat      2      6      5      0
eventpoll_pwq          0      0     64   59    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0                                  0    0    0    0    0 : cpustat      0      0      0      0
eventpoll_epi          0      0     96   40    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0                                  0    0    0    0    0 : cpustat      0      0      0      0
blkdev_queue           1      3   1192    3    1 : tunables   24   12    0 : slabdata      1      1      0 : globalstat       3      3     1    0                                  0    0    0    0    0 : cpustat      0      1      0      0
blkdev_requests        4     17    224   17    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      29     17     1    0                                  0    0    0    0    0 : cpustat      8      2      6      0
blkdev_ioc             0      0     72   53    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0                                  0    0    0    0    0 : cpustat      0      0      0      0
bio-0                  2     27    144   27    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      32     18     1    0                                  0    0    0    0    0 : cpustat     96      2     96      0
biovec-256             2      2   3096    1    1 : tunables   24   12    0 : slabdata      2      2      0 : globalstat       2      2     2    0                                  0    0    0    0    0 : cpustat      0      2      0      0
biovec-128             0      0   1560    2    1 : tunables   24   12    0 : slabdata      0      0      0 : globalstat       0      0     0    0                                  0    0    0    0    0 : cpustat      0      0      0      0
biovec-64              0      0    792    5    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0                                  0    0    0    0    0 : cpustat      0      0      0      0
biovec-16              0      0    216   18    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0                                  0    0    0    0    0 : cpustat      0      0      0      0
sock_inode_cache      24     24    328   12    1 : tunables   32   16    0 : slabdata      2      2      0 : globalstat      26     24     2    0                                  0    0    0    0    0 : cpustat 293647      3 293639      0
skbuff_cb_store_cache      0      0     80   48    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat       0      0     0    0                              0    0    0    0    0 : cpustat      0      0      0      0
skbuff_fclone_cache     10     10    400   10    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat     310     10    28   27                                0    0    0    0    0 : cpustat    941     31    972      0
skbuff_head_cache    206    209    208   19    1 : tunables   32   16    0 : slabdata     11     11      0 : globalstat     417    206    19    1                                  0    2    0    0    0 : cpustat 156776     40 156603     13
file_lock_cache        0      0    120   32    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat      32     16     2    2                                  0    0    0    0    0 : cpustat    100      2    102      0
proc_inode_cache     106    108    312   12    1 : tunables   32   16    0 : slabdata      9      9      0 : globalstat     478    120    10    1                                  0    0    0    0    0 : cpustat    541     34    484      0
sigqueue               0      0    168   23    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat      96     16     6    6                                  0    0    0    0    0 : cpustat      9      6     15      0
radix_tree_node       29     36    312   12    1 : tunables   32   16    0 : slabdata      3      3      0 : globalstat      57     36     3    0                                  0    0    0    0    0 : cpustat     30      6      7      0
bdev_cache             3     10    392   10    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      18     10     1    0                                  0    0    0    0    0 : cpustat      1      2      0      0
sysfs_dir_cache      745    795     72   53    1 : tunables   32   16    0 : slabdata     15     15      0 : globalstat     758    758    15    0                                  0    0    0    0    0 : cpustat    700     57     12      0
mnt_cache             15     26    152   26    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      16     16     1    0                                  0    0    0    0    0 : cpustat     15      1      1      0
filp                  60    108    144   27    1 : tunables   32   16    0 : slabdata      4      4      0 : globalstat 1077100    108     4    0                                  0    1    0    0    0 : cpustat 3092944  67320 3092911  67315
inode_cache          247    247    288   13    1 : tunables   32   16    0 : slabdata     19     19      0 : globalstat     250    247    19    0                                  0    0    0    0    0 : cpustat 147129     20 146909      0
dentry               556    572    152   26    1 : tunables   32   16    0 : slabdata     22     22      0 : globalstat     620    572    22    0                                  0    0    0    0    0 : cpustat 588705     47 588214      4
names_cache            4      4   4096    1    1 : tunables   24   12    0 : slabdata      4      4      0 : globalstat       4      4     4    0                                  0    0    0    0    0 : cpustat 4116476      4 4116480      0
buffer_head            0      0     80   48    1 : tunables   32   16    0 : slabdata      0      0      0 : globalstat      16     16     1    1                                  0    0    0    0    0 : cpustat     95      1     96      0
vm_area_struct       101    210    112   35    1 : tunables   32   16    0 : slabdata      6      6      0 : globalstat 5213829    347    15    0                                  0    1    0    0    0 : cpustat 15764665 325873 15764582 325858
mm_struct             20     20    392   10    1 : tunables   32   16    0 : slabdata      2      2      0 : globalstat      20     20     2    0                                  0    0    0    0    0 : cpustat 661247      2 661244      0
fs_cache              32     67     56   67    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      32     32     1    0                                  0    0    0    0    0 : cpustat 367478      2 367476      0
files_cache           19     19    200   19    1 : tunables   32   16    0 : slabdata      1      1      0 : globalstat      19     19     1    0                                  0    0    0    0    0 : cpustat 367479      2 367476      0
signal_cache          32     32    504    8    1 : tunables   32   16    0 : slabdata      4      4      0 : globalstat      32     32     4    0                                  0    0    0    0    0 : cpustat 367496      4 367481      0
sighand_cache         31     31   3112    1    1 : tunables   24   12    0 : slabdata     31     31      0 : globalstat      31     31    31    0                                  0    0    0    0    0 : cpustat 367469     31 367481      0
task_struct           33     33   1176    3    1 : tunables   24   12    0 : slabdata     11     11      0 : globalstat      33     33    11    0                                  0    0    0    0    0 : cpustat 367489     11 367481      0
cred_jar              38     70    112   35    1 : tunables   32   16    0 : slabdata      2      2      0 : globalstat     102     86     3    1                                  0    1    0    0    0 : cpustat 882054      8 882031      4
anon_vma              65    226     32  113    1 : tunables   32   16    0 : slabdata      2      2      0 : globalstat 1077137    129     2    0                                  0    0    0    0    0 : cpustat 2577723  67322 2577683  67317
pid                73453  73458     72   53    1 : tunables   32   16    0 : slabdata   1386   1386      0 : globalstat   73453  73453  1386    0                                  0    0    0    0    0 : cpustat 361957   5543 294060      0
idr_layer_cache       83     88    176   22    1 : tunables   32   16    0 : slabdata      4      4      0 : globalstat     109     88     4    0                                  0    0    0    0    0 : cpustat    218     12    147      0
size-131072            0      0 131072    1   32 : tunables    8    4    0 : slabdata      0      0      0 : globalstat       0      0     0    0                                  0    0    0    0    0 : cpustat      0      0      0      0
size-65536             1      1  65536    1   16 : tunables    8    4    0 : slabdata      1      1      0 : globalstat       1      1     1    0                                  0    0    0    0    0 : cpustat      2      1      2      0
size-32768             0      0  32768    1    8 : tunables    8    4    0 : slabdata      0      0      0 : globalstat       0      0     0    0                                  0    0    0    0    0 : cpustat      0      0      0      0
size-16384             0      0  16384    1    4 : tunables    8    4    0 : slabdata      0      0      0 : globalstat       0      0     0    0                                  0    0    0    0    0 : cpustat      0      0      0      0
size-8192             23     23   8192    1    2 : tunables    8    4    0 : slabdata     23     23      0 : globalstat      31     31    31    8                                  0    4    0    0    0 : cpustat 367485     31 367495      2
size-4096             20     20   4096    1    1 : tunables   24   12    0 : slabdata     20     20      0 : globalstat      20     20    20    0                                  0    0    0    0    0 : cpustat 147789     20 147792      0
size-2048            210    210   2048    2    1 : tunables   24   12    0 : slabdata    105    105      0 : globalstat    2357    210   196   14                                  0   13    0    0    0 : cpustat   1499   1204   2481     15
size-1024             52     52   1024    4    1 : tunables   32   16    0 : slabdata     13     13      0 : globalstat      54     52    13    0                                  0    0    0    0    0 : cpustat  73935     14  73900      0
size-512              57     64    512    8    1 : tunables   32   16    0 : slabdata      8      8      0 : globalstat      80     80    10    2                                  0    1    0    0    0 : cpustat 302174     10 302139      1
size-256              30     30    256   15    1 : tunables   32   16    0 : slabdata      2      2      0 : globalstat      30     30     2    0                                  0    0    0    0    0 : cpustat 881852      2 881845      0
size-192             128    135    256   15    1 : tunables   32   16    0 : slabdata      9      9      0 : globalstat     168    135     9    0                                  0    0    0    0    0 : cpustat    147     17     36      0
size-128             136    150    128   30    1 : tunables   32   16    0 : slabdata      5      5      0 : globalstat     136    136     5    0                                  0    0    0    0    0 : cpustat 294035     11 293916      0
size-96               80     90    128   30    1 : tunables   32   16    0 : slabdata      3      3      0 : globalstat     198     90     3    0                                  0    0    0    0    0 : cpustat    425     16    361      0
size-64              450    450    128   30    1 : tunables   32   16    0 : slabdata     15     15      0 : globalstat    1109    450    16    1                                  0    0    0    0    0 : cpustat   1326     98    970     16
size-32              946    960    128   30    1 : tunables   32   16    0 : slabdata     32     32      0 : globalstat    1680    946    32    0                                  0    0    0    0    0 : cpustat 369950    125 369074     59
kmem_cache            80     96    160   24    1 : tunables   32   16    0 : slabdata      4      4      0 : globalstat      88     88     4    0                                  0    0    0    0    0 : cpustat     59     21      0      0

comment:7 Changed 8 years ago by acoul <alex at ozo.com>

looking into this bug with acinonyx we have located that on devices that they lack serail ports, inittab respawns /bin/ash --login for ttyS0.

it does that with the askfirst option that should wait for input before doing anything. for some reason inittab keeps respawning /bin/ash --login for ttyS0, pid jumps sky rocket and apparently ash does not properly clean memory upon exit, thereof the router dies at some point.

this bug reviled a bigger problem of ash not properly cleaning stuff upon exit.

comment:8 Changed 8 years ago by acoul <alex at ozo.com>

further digging revealed that udevtrigger creates /dev/ttyS0 & /dev/ttyS1 according to /sys/devices/platform/serial8250/tty/ttyS0 & /sys/devices/platform/serial8250/tty/ttyS1 even though the device does not have any UART ports.

somehow proper detection of lack of UART ports has to be implemented and ash memory leak issue has to be investigated.

comment:9 Changed 8 years ago by acoul <alex at ozo.com>

a quick workaround is to pass the 8250.nr_uarts=0 parameter on the kernel command line: CONFIG_CMDLINE on target/linux/brcm47xx/config-2.6.xx

we may even have this as an option on menuconfig. The question remains why the kernel thinks there is a uart on the system when there is not and why respawning ash through inittab and then dying as a zombie produces memory leak.

comment:10 Changed 8 years ago by acoul <alex at ozo.com>

8250.nr_uarts=0 is not enough to deal with this issue. had to remove:

ttyS0::askfirst:/bin/ash --login
tty1::askfirst:/bin/ash --login

from /etc/inittab. so far memory & pid looks ok

root@OpenWrt:/# uptime

 22:41:45 up 22:41, load average: 0.00, 0.00, 0.00
root@OpenWrt:/# free

              total         used         free       shared      buffers
  Mem:        13776         4376         9400            0            0
 Swap:            0            0            0
Total:        13776         4376         9400
root@OpenWrt:/# cat /proc/slabinfo  | grep pid

pid                   34     59     64   59    1 : tunables  120   60    0 : slabdata      1      1      0

comment:11 Changed 8 years ago by acoul <alex at ozo.com>

workaround works

BusyBox v1.15.3 (2010-01-07 22:29:48 EET) built-in shell (ash)
Enter 'help' for a list of built-in commands.

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 KAMIKAZE (bleeding edge, r19062) ------------------
  * 10 oz Vodka       Shake well with ice and strain
  * 10 oz Triple sec  mixture into 10 shot glasses.
  * 10 oz lime juice  Salute!
 ---------------------------------------------------
root@OpenWrt:/# uptime
 19:42:22 up 5 days, 19:42, load average: 0.00, 0.00, 0.00
root@OpenWrt:/# free
              total         used         free       shared      buffers
  Mem:        13776         4380         9396            0            0
 Swap:            0            0            0
Total:        13776         4380         9396

comment:12 Changed 8 years ago by thepeople

  • Owner changed from developers to hauke
  • Status changed from new to assigned

comment:13 Changed 7 years ago by hauke

Does this problem still occur in recent trunk or was it fixed some time ago?

comment:14 Changed 7 years ago by loswillios

  • Resolution set to fixed
  • Status changed from assigned to closed

I'm running r24149 and have not experienced this issue.

root@wgt:~# uptime
 11:22:12 up 47 days, 19:07, load average: 0.15, 0.03, 0.01

Please reopen if this still happens for you.

comment:15 Changed 4 years ago by jow

  • Milestone changed from Attitude Adjustment 12.09 to Barrier Breaker 14.07

Milestone Attitude Adjustment 12.09 deleted

Add Comment

Modify Ticket

Action
as closed .
The resolution will be deleted. Next status will be 'reopened'.
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.