Modify

Opened 7 years ago

Closed 5 years ago

#8433 closed defect (fixed)

Strange segfaults on Orion with latest Backfire WRT350Nv2

Reported by: Ernesto Owned by: developers
Priority: high Milestone:
Component: kernel Version: Trunk
Keywords: SEGFAULTS ORION WRT350NV2 Cc:

Description

I have on Orion WRT350Nv2 with latest Backfire this segfaults all 2-3 Minutes:

I have only one pppoe-session and use 1 LAN-Port, no wireless.
For me looks like problem with the klogd or syslogger that is the
trigger for the problems. The free memory is in the Range of 12 MB free after the crash and 3-4 MB before the next crash.

Dec 11 14:57:19 gateway2 user.warn kernel: swapper: page allocation failure. order:4, mode:0x20
Dec 11 14:57:19 gateway2 user.warn kernel: Backtrace:
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c0023490>] from [<c01f1074>]
Dec 11 14:57:19 gateway2 user.warn kernel:  r7:00000000 r6:c024a4f0 r5:00000020 r4:00000000
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c01f105c>] from [<c006392c>]
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c00633f8>] from [<c0083980>]
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c00836e8>] from [<c0083d0c>]
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c0083c74>] from [<c0172c38>]
Dec 11 14:57:19 gateway2 user.warn kernel:  r7:00000040 r6:c0256a10 r5:00008380 r4:c1af1c80
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c0172be4>] from [<c01e59f8>]
Dec 11 14:57:19 gateway2 user.warn kernel:  r9:c0256a30 r8:c195fc00 r7:c1af1c80 r6:c0256a10 r5:c195fec0
Dec 11 14:57:19 gateway2 user.warn kernel: r4:c1af1c80
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c01e5880>] from [<c017af58>]
Dec 11 14:57:19 gateway2 user.warn kernel:  r5:c0256a30 r4:c0256a28
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c017ad20>] from [<c017b4d0>]
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c017b160>] from [<bf015570>]
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<bf0153e0>] from [<bf0155a4>]
Dec 11 14:57:19 gateway2 user.warn kernel:  r8:c1af1c80 r7:00000021 r6:c1a936c0 r5:00000000 r4:c1ad4330
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<bf015590>] from [<bf00474c>]
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<bf0046d0>] from [<bf0055bc>]
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<bf005154>] from [<bf005820>]
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<bf005688>] from [<c017af58>]
Dec 11 14:57:19 gateway2 user.warn kernel:  r7:c1af1c80 r6:c0256a10 r5:c0256a30 r4:c0256a28
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c017ad20>] from [<c018b78c>]
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c018b738>] from [<c017b3ec>]
Dec 11 14:57:19 gateway2 user.warn kernel:  r9:00000000 r8:c1b9ba1c r7:00000000 r6:008afb78 r5:00000000
Dec 11 14:57:19 gateway2 user.warn kernel: r4:008afba0
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c017b160>] from [<c01a3670>]
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c01a3408>] from [<c01a3770>]
Dec 11 14:57:19 gateway2 user.warn kernel:  r9:00000000 r8:00000008 r7:c1af1ca0 r6:c1af1c80 r5:c1af1c80
Dec 11 14:57:19 gateway2 user.warn kernel: r4:c01a3408
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c01a36a8>] from [<c019f418>]
Dec 11 14:57:19 gateway2 user.warn kernel:  r5:c09b14a0 r4:c1af1c80
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c019f3d4>] from [<c019f6d8>]
Dec 11 14:57:19 gateway2 user.warn kernel:  r5:c09b14a0 r4:c019f3d4
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c019f420>] from [<c019df24>]
Dec 11 14:57:19 gateway2 user.warn kernel:  r7:c0580034 r6:c1af1c80 r5:c1af1c80 r4:c1af1c80
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c019db78>] from [<c019e214>]
Dec 11 14:57:19 gateway2 user.warn kernel:  r8:00000008 r7:c1ba0800 r6:c02567b4 r5:c1af1c80 r4:c1af1c80
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c019df48>] from [<c017a184>]
Dec 11 14:57:19 gateway2 user.warn kernel:  r7:c0256acc r6:c1af1c80 r5:c0256aac r4:c0256acc
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c0179cc8>] from [<c01e1648>]
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c01e14fc>] from [<c01e18b4>]
Dec 11 14:57:19 gateway2 user.warn kernel:  r8:c194d200 r7:00001d00 r6:00008895 r5:00000000 r4:c0580026
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c01e1694>] from [<c017a0a8>]
Dec 11 14:57:19 gateway2 user.warn kernel:  r8:00000001 r7:c1953400 r6:c1af1c80 r5:c024699c r4:c194d200
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c0179cc8>] from [<c01e5864>]
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c01e5628>] from [<c017a184>]
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c0179cc8>] from [<c015f39c>]
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c015ee68>] from [<c017a90c>]
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c017a8a4>] from [<c0034c38>]
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c0034bb8>] from [<c0034d0c>]
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c0034cc4>] from [<c001f074>]
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c001f000>] from [<c001fa94>]
Dec 11 14:57:19 gateway2 user.warn kernel: Exception stack(0xc0239f58 to 0xc0239fa0)
Dec 11 14:57:19 gateway2 user.warn kernel: 9f40:                                                       00000000 c023ad68
Dec 11 14:57:19 gateway2 user.warn kernel: 9f60: 00000000 60000013 c0238000 c024a7c4 c023ba18 c023ba10 0001c7bc 41069260
Dec 11 14:57:19 gateway2 user.warn kernel: 9f80: 0001c6ec c0239fbc c0239fa0 c0239fa0 c0020934 c0020e80 60000013 ffffffff
Dec 11 14:57:19 gateway2 user.warn kernel:  r7:c023ba10 r6:00200000 r5:fdd20200 r4:ffffffff
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c0020e20>] from [<c01ec9b4>]
Dec 11 14:57:19 gateway2 user.warn kernel:  r7:c023ba10 r6:c001de0c r5:c024a78c r4:c0251d14
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c01ec958>] from [<c00089ec>]
Dec 11 14:57:19 gateway2 user.warn kernel: Function entered at [<c000877c>] from [<00008034>]
Dec 11 14:57:19 gateway2 user.warn kernel:  r5:c024a868 r4:a0053175
Dec 11 14:57:19 gateway2 user.warn kernel: Mem-info:
Dec 11 14:57:19 gateway2 user.warn kernel: Normal per-cpu:
Dec 11 14:57:19 gateway2 user.warn kernel: CPU    0: hi:    0, btch:   1 usd:   0
Dec 11 14:57:19 gateway2 user.warn kernel: active_anon:266 inactive_anon:40 isolated_anon:0
Dec 11 14:57:19 gateway2 user.warn kernel:  active_file:692 inactive_file:1238 isolated_file:0
Dec 11 14:57:19 gateway2 user.warn kernel:  unevictable:0 dirty:0 writeback:0 unstable:0
Dec 11 14:57:19 gateway2 user.warn kernel:  free:616 slab_reclaimable:445 slab_unreclaimable:3593
Dec 11 14:57:19 gateway2 user.warn kernel:  mapped:306 shmem:22 pagetables:51 bounce:0
Dec 11 14:57:19 gateway2 user.warn kernel: Normal free:2464kB min:720kB low:900kB high:1080kB active_anon:1064kB inactive_anon:160kB active_file:2768kB inactive_file:4952kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:32512kB mlocked:0kB dirty:0kB write
Dec 11 14:57:19 gateway2 user.warn kernel: lowmem_reserve[]: 0 0
Dec 11 14:57:19 gateway2 user.warn kernel: Normal: 42*4kB 111*8kB 8*16kB 38*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2464kB
Dec 11 14:57:19 gateway2 user.warn kernel: 1952 total pagecache pages
Dec 11 14:57:19 gateway2 user.warn kernel: 0 pages in swap cache
Dec 11 14:57:19 gateway2 user.warn kernel: Swap cache stats: add 0, delete 0, find 0/0
Dec 11 14:57:19 gateway2 user.warn kernel: Free swap  = 0kB
Dec 11 14:57:19 gateway2 user.warn kernel: Total swap = 0kB
Dec 11 14:57:19 gateway2 user.warn kernel: 8192 pages of RAM
Dec 11 14:57:19 gateway2 user.warn kernel: 700 free pages
Dec 11 14:57:19 gateway2 user.warn kernel: 683 reserved pages
Dec 11 14:57:19 gateway2 user.warn kernel: 4038 slab pages
Dec 11 14:57:19 gateway2 user.warn kernel: 1676 pages shared
Dec 11 14:57:19 gateway2 user.warn kernel: 0 pages swap cached
Dec 11 14:59:13 gateway2 user.warn kernel: swapper: page allocation failure. order:4, mode:0x20
Dec 11 14:59:13 gateway2 user.warn kernel: Backtrace:
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c0023490>] from [<c01f1074>]
Dec 11 14:59:13 gateway2 user.warn kernel:  r7:00000000 r6:c024a4f0 r5:00000020 r4:00000000
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c01f105c>] from [<c006392c>]
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c00633f8>] from [<c0083980>]
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c00836e8>] from [<c0083d0c>]
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c0083c74>] from [<c0172c38>]
Dec 11 14:59:13 gateway2 user.warn kernel:  r7:00000040 r6:c0256a10 r5:00008180 r4:c0f15b00
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c0172be4>] from [<c01e59f8>]
Dec 11 14:59:13 gateway2 user.warn kernel:  r9:c0256a30 r8:c195fc00 r7:c0f15b00 r6:c0256a10 r5:c195fec0
Dec 11 14:59:13 gateway2 user.warn kernel: r4:c0f15b00
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c01e5880>] from [<c017af58>]
Dec 11 14:59:13 gateway2 user.warn kernel:  r5:c0256a30 r4:c0256a28
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c017ad20>] from [<c017b4d0>]
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c017b160>] from [<bf015570>]
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<bf0153e0>] from [<bf0155a4>]
Dec 11 14:59:13 gateway2 user.warn kernel:  r8:c0f15b00 r7:00000021 r6:c1a936c0 r5:00000000 r4:c1ad4330
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<bf015590>] from [<bf00474c>]
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<bf0046d0>] from [<bf0055bc>]
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<bf005154>] from [<bf005820>]
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<bf005688>] from [<c017af58>]
Dec 11 14:59:13 gateway2 user.warn kernel:  r7:c0f15b00 r6:c0256a10 r5:c0256a30 r4:c0256a28
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c017ad20>] from [<c018b78c>]
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c018b738>] from [<c017b3ec>]
Dec 11 14:59:13 gateway2 user.warn kernel:  r9:00000000 r8:c1b9ba1c r7:00000000 r6:00b3dc18 r5:00000000
Dec 11 14:59:13 gateway2 user.warn kernel: r4:00b3dc40
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c017b160>] from [<c01a3670>]
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c01a3408>] from [<c01a3770>]
Dec 11 14:59:13 gateway2 user.warn kernel:  r9:00000000 r8:00000008 r7:c0f15b20 r6:c0f15b00 r5:c0f15b00
Dec 11 14:59:13 gateway2 user.warn kernel: r4:c01a3408
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c01a36a8>] from [<c019f418>]
Dec 11 14:59:13 gateway2 user.warn kernel:  r5:c09d6ec0 r4:c0f15b00
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c019f3d4>] from [<c019f6d8>]
Dec 11 14:59:13 gateway2 user.warn kernel:  r5:c09d6ec0 r4:c019f3d4
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c019f420>] from [<c019df24>]
Dec 11 14:59:13 gateway2 user.warn kernel:  r7:c0790034 r6:c0f15b00 r5:c0f15b00 r4:c0f15b00
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c019db78>] from [<c019e214>]
Dec 11 14:59:13 gateway2 user.warn kernel:  r8:00000008 r7:c1ba0800 r6:c02567b4 r5:c0f15b00 r4:c0f15b00
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c019df48>] from [<c017a184>]
Dec 11 14:59:13 gateway2 user.warn kernel:  r7:c0256acc r6:c0f15b00 r5:c0256aac r4:c0256acc
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c0179cc8>] from [<c01e1648>]
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c01e14fc>] from [<c01e18b4>]
Dec 11 14:59:13 gateway2 user.warn kernel:  r8:c194d200 r7:00001d00 r6:00008895 r5:00000000 r4:c0790026
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c01e1694>] from [<c017a0a8>]
Dec 11 14:59:13 gateway2 user.warn kernel:  r8:00000001 r7:c1953400 r6:c0f15b00 r5:c024699c r4:c194d200
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c0179cc8>] from [<c01e5864>]
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c01e5628>] from [<c017a184>]
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c0179cc8>] from [<c015f39c>]
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c015ee68>] from [<c017a90c>]
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c017a8a4>] from [<c0034c38>]
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c0034bb8>] from [<c0034d0c>]
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c0034cc4>] from [<c001f074>]
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c001f000>] from [<c001fa94>]
Dec 11 14:59:13 gateway2 user.warn kernel: Exception stack(0xc0239f58 to 0xc0239fa0)
Dec 11 14:59:13 gateway2 user.warn kernel: 9f40:                                                       00000000 c023ad68
Dec 11 14:59:13 gateway2 user.warn kernel: 9f60: 00000000 60000013 c0238000 c024a7c4 c023ba18 c023ba10 0001c7bc 41069260
Dec 11 14:59:13 gateway2 user.warn kernel: 9f80: 0001c6ec c0239fbc c0239fa0 c0239fa0 c0020934 c0020e80 60000013 ffffffff
Dec 11 14:59:13 gateway2 user.warn kernel:  r7:c023ba10 r6:00200000 r5:fdd20200 r4:ffffffff
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c0020e20>] from [<c01ec9b4>]
Dec 11 14:59:13 gateway2 user.warn kernel:  r7:c023ba10 r6:c001de0c r5:c024a78c r4:c0251d14
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c01ec958>] from [<c00089ec>]
Dec 11 14:59:13 gateway2 user.warn kernel: Function entered at [<c000877c>] from [<00008034>]
Dec 11 14:59:13 gateway2 user.warn kernel:  r5:c024a868 r4:a0053175
Dec 11 14:59:13 gateway2 user.warn kernel: Mem-info:
Dec 11 14:59:13 gateway2 user.warn kernel: Normal per-cpu:
Dec 11 14:59:13 gateway2 user.warn kernel: CPU    0: hi:    0, btch:   1 usd:   0
Dec 11 14:59:13 gateway2 user.warn kernel: active_anon:235 inactive_anon:71 isolated_anon:0
Dec 11 14:59:13 gateway2 user.warn kernel:  active_file:692 inactive_file:1238 isolated_file:0
Dec 11 14:59:13 gateway2 user.warn kernel:  unevictable:0 dirty:0 writeback:0 unstable:0
Dec 11 14:59:13 gateway2 user.warn kernel:  free:696 slab_reclaimable:445 slab_unreclaimable:3513
Dec 11 14:59:13 gateway2 user.warn kernel:  mapped:316 shmem:22 pagetables:51 bounce:0
Dec 11 14:59:13 gateway2 user.warn kernel: Normal free:2784kB min:720kB low:900kB high:1080kB active_anon:940kB inactive_anon:284kB active_file:2768kB inactive_file:4952kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:32512kB mlocked:0kB dirty:0kB writeb
Dec 11 14:59:13 gateway2 user.warn kernel: lowmem_reserve[]: 0 0
Dec 11 14:59:13 gateway2 user.warn kernel: Normal: 50*4kB 109*8kB 19*16kB 42*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2784kB
Dec 11 14:59:13 gateway2 user.warn kernel: 1952 total pagecache pages
Dec 11 14:59:13 gateway2 user.warn kernel: 0 pages in swap cache
Dec 11 14:59:13 gateway2 user.warn kernel: Swap cache stats: add 0, delete 0, find 0/0
Dec 11 14:59:13 gateway2 user.warn kernel: Free swap  = 0kB
Dec 11 14:59:13 gateway2 user.warn kernel: Total swap = 0kB
Dec 11 14:59:13 gateway2 user.warn kernel: 8192 pages of RAM
Dec 11 14:59:13 gateway2 user.warn kernel: 780 free pages
Dec 11 14:59:13 gateway2 user.warn kernel: 683 reserved pages
Dec 11 14:59:13 gateway2 user.warn kernel: 3958 slab pages
Dec 11 14:59:13 gateway2 user.warn kernel: 1686 pages shared
Dec 11 14:59:13 gateway2 user.warn kernel: 0 pages swap cached

Here is my actual process-list:

top - 15:05:15 up 17 min,  0 users,  load average: 0.00, 0.00, 0.00
Tasks:  36 total,   1 running,  35 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.0%us,  0.0%sy,  0.0%ni, 70.7%id,  0.0%wa,  3.9%hi, 23.4%si,  0.0%st
Mem:     30152k total,    26152k used,     4000k free,     2376k buffers
Swap:        0k total,        0k used,        0k free,     5432k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 2607 root      20   0   956  504  364 R  2.4  1.7   0:00.27 top
 2598 root      20   0   876  484  356 S  0.5  1.6   0:00.61 dropbear
    1 root      20   0  1164  348  284 S  0.0  1.2   0:02.13 init
    3 root      20   0     0    0    0 S  0.0  0.0   0:00.01 ksoftirqd/0
    5 root      20   0     0    0    0 S  0.0  0.0   0:00.15 khelper
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.01 kthreadd
   42 root      20   0     0    0    0 S  0.0  0.0   0:00.01 sync_supers
    4 root      20   0     0    0    0 S  0.0  0.0   0:00.26 events/0
   46 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kblockd/0
    8 root      20   0     0    0    0 S  0.0  0.0   0:00.00 async/mgr
   73 root      20   0     0    0    0 S  0.0  0.0   0:00.00 aio/0
   74 root      20   0     0    0    0 S  0.0  0.0   0:00.00 crypto/0
  603 root      20   0     0    0    0 S  0.0  0.0   0:00.84 mtdblockd
  650 root      20   0     0    0    0 S  0.0  0.0   0:00.00 mv_crypto
 1309 root      30  10     0    0    0 S  0.0  0.0   0:00.04 jffs2_gcd_mtd2
 1321 root      20   0  1156  244  192 S  0.0  0.8   0:00.01 rcS
 1322 root      20   0  1164  204  140 S  0.0  0.7   0:00.00 init
 1324 root      20   0  1156  304  248 S  0.0  1.0   0:00.03 logger
 1337 root      20   0  1168  296  244 S  0.0  1.0   0:00.13 syslogd
 1339 root      20   0  1152  260  208 S  0.0  0.9   0:00.21 klogd
   44 root      20   0     0    0    0 S  0.0  0.0   0:00.01 bdi-default
 1598 root      20   0  1272  600  460 S  0.0  2.0   0:00.48 pppd
   72 root      20   0     0    0    0 S  0.0  0.0   0:00.01 kswapd0
 1823 root      20   0   560  272  212 S  0.0  0.9   0:00.01 ntpclient
 1909 root      20   0     0    0    0 S  0.0  0.0   0:00.03 phy0
 2244 root      20   0  1052  404  324 S  0.0  1.3   0:00.08 hostapd
 2529 root      20   0   820  288  228 S  0.0  1.0   0:00.01 dropbear
 2531 root      20   0   548  300  228 S  0.0  1.0   0:03.28 mini_snmpd
 2537 root      20   0   812  300  232 S  0.0  1.0   0:00.09 uhttpd
 2541 root      20   0   740  196  148 S  0.0  0.7   0:00.01 vsftpd
 2563 nobody    20   0   656  336  260 S  0.0  1.1   0:00.04 dnsmasq
 2572 root      20   0   876  484  356 S  0.0  1.6   0:00.95 dropbear
 2573 root      20   0  1164  408  340 S  0.0  1.4   0:00.12 ash
 1352 root      20   0   536  260  200 S  0.0  0.9   0:00.08 hotplug2
 2599 root      20   0  1164  408  340 S  0.0  1.4   0:00.04 ash
 1747 root      20   0     0    0    0 S  0.0  0.0   0:00.00 cfg80211

Attachments (0)

Change History (8)

comment:1 Changed 7 years ago by Ernesto

And now i verified the SAME Problems with actual Trunk.

The segfaults comes up ONLY when pppoe-traffic is active in the range of 11-15 mbit downstream.

On Trunk the faults comes up all 1-2 Minutes.

comment:2 Changed 7 years ago by Ernesto

and a few minutes later the next "fun":

Dec 11 16:30:42 gateway2 user.warn kernel: 0 pages swap cached
Dec 11 16:33:34 gateway2 user.err kernel: net eth0: received packet spanning multiple descriptors
Dec 11 16:33:34 gateway2 user.err kernel: net eth0: received packet spanning multiple descriptors
Dec 11 16:33:49 gateway2 user.err kernel: net eth0: received packet spanning multiple descriptors
Dec 11 16:33:49 gateway2 user.err kernel: net eth0: received packet spanning multiple descriptors
Dec 11 16:33:53 gateway2 user.err kernel: net eth0: received packet spanning multiple descriptors
Dec 11 16:33:53 gateway2 user.err kernel: net eth0: received packet spanning multiple descriptors
Dec 11 16:34:20 gateway2 user.err kernel: net eth0: received packet spanning multiple descriptors
Dec 11 16:34:20 gateway2 user.err kernel: net eth0: received packet spanning multiple descriptors
Dec 11 16:34:25 gateway2 user.err kernel: net eth0: received packet spanning multiple descriptors
Dec 11 16:34:25 gateway2 user.err kernel: net eth0: received packet spanning multiple descriptors
Dec 11 16:34:35 gateway2 user.err kernel: net eth0: received packet spanning multiple descriptors
Dec 11 16:34:35 gateway2 user.err kernel: net eth0: received packet spanning multiple descriptors
Dec 11 16:34:36 gateway2 user.err kernel: net eth0: received packet spanning multiple descriptors
Dec 11 16:34:36 gateway2 user.err kernel: net eth0: received packet spanning multiple descriptors
Dec 11 16:35:20 gateway2 user.err kernel: net eth0: received packet spanning multiple descriptors
Dec 11 16:35:20 gateway2 user.err kernel: net eth0: received packet spanning multiple descriptors
Dec 11 16:35:31 gateway2 user.err kernel: net eth0: received packet spanning multiple descriptors

comment:3 Changed 7 years ago by Ernesto

I disabled the xt_imq and ip_queue kernel modules for testing, this delays the segfaults a few minutes. So my first idea about memory leaks in the post/pre-hockers of IMQ was wrong.

Fact is, that all NAT data is cached until 3 MB real memory is reached and then the kernel crashes and free the memory. as more traffic i have on pppoe Session, as faster the next segfault comes up.

Can someone verify this Problem with actual trunk/backfire rc4 on a brcm47xx router by using pppoe + LAN Port and simple NAT and make some NNTP Traffic ?

here are my used sysctl.conf:

kernel.panic=3

net.ipv4.conf.default.arp_ignore=1
net.ipv4.conf.all.arp_ignore=1
net.ipv4.ip_forward=1
net.ipv4.icmp_echo_ignore_broadcasts=1
net.ipv4.icmp_ignore_bogus_error_responses=1
net.ipv4.tcp_ecn=0
net.ipv4.tcp_fin_timeout=30
net.ipv4.tcp_keepalive_time=180
net.ipv4.tcp_syncookies=0
net.ipv4.tcp_timestamps=0

net.core.netdev_max_backlog=30

net.netfilter.nf_conntrack_checksum=0
net.netfilter.nf_conntrack_max=4096
net.netfilter.nf_conntrack_tcp_timeout_established=180
net.netfilter.nf_conntrack_udp_timeout=60
net.netfilter.nf_conntrack_udp_timeout_stream=180

... so not really unusual things here for a basic router setup ?!

comment:4 Changed 7 years ago by Ernesto

I checked out a fresh trunk before 1 hour and switched to uclibc 0.9.32.

Except for xtables/IMQ, that is usualy not working these days, it looks a "Little"
more stable and not so much memleaks/segfaults on pppoe-session now.

But: after 15 minutes of checking: this comes up again:

root@gateway2:~# dmesg
net eth0: received packet spanning multiple descriptors
net eth0: received packet spanning multiple descriptors
swapper: page allocation failure. order:5, mode:0x20
Backtrace:
Function entered at [<c00253f8>] from [<c0219c8c>]
 r7:00000000 r6:c027806c r5:00000020 r4:00000000
Function entered at [<c0219c74>] from [<c0069834>]
Function entered at [<c00692f0>] from [<c008b7f0>]
Function entered at [<c008b53c>] from [<c008bb8c>]
Function entered at [<c008baf4>] from [<c018fd68>]
 r7:00000020 r6:c0284f84 r5:0000ff20 r4:c1ac9c20
Function entered at [<c018fd04>] from [<c020e0b0>]
 r9:c1964f20 r8:c1965000 r7:c1ac9c20 r6:c0284f84 r5:c1965320
r4:c1ac9c20
Function entered at [<c020df38>] from [<c019846c>]
 r5:c1965000 r4:c0284fa4
Function entered at [<c0198048>] from [<c01987d0>]
Function entered at [<c01985c0>] from [<bf0155a4>]
Function entered at [<bf015414>] from [<bf0155dc>]
 r8:c1ac9c20 r7:00000021 r6:c1bb1320 r5:00000000 r4:c1acff38
Function entered at [<bf0155c8>] from [<bf004714>]
Function entered at [<bf004698>] from [<bf00556c>]
Function entered at [<bf005104>] from [<bf0057d0>]
Function entered at [<bf005638>] from [<c019846c>]
 r7:c1ac9c20 r6:c0284f84 r5:00000000 r4:c0284fa4
Function entered at [<c0198048>] from [<c01ac55c>]
Function entered at [<c01ac508>] from [<c01986dc>]
 r9:00000016 r8:c1acfa5c r7:00000000 r6:0262184f r5:00000000
r4:02621877
Function entered at [<c01985c0>] from [<c01c4f10>]
Function entered at [<c01c4c7c>] from [<c01c5020>]
Function entered at [<c01c4f48>] from [<c01c0b44>]
 r7:c03c0034 r6:c1ac9c20 r5:00000000 r4:00000001
Function entered at [<c01c0af0>] from [<c01c0e20>]
 r7:c03c0034 r6:c09dd5c0 r5:00000000 r4:c01c0af0
Function entered at [<c01c0b4c>] from [<c01bf60c>]
Function entered at [<c01bf250>] from [<c01bf934>]
 r8:c1a4f000 r7:c0284d54 r6:c1ac9c20 r5:00000000 r4:c1ac9c20
Function entered at [<c01bf630>] from [<c01971bc>]
 r8:c0285044 r7:c02760ac r6:c1ac9c20 r5:c0285024 r4:c0285044
Function entered at [<c0196d4c>] from [<c01973b0>]
Function entered at [<c0197354>] from [<c02075f4>]
 r4:02abf6e7
Function entered at [<c02073d4>] from [<c0207868>]
 r8:00001d00 r7:00008895 r6:00000000 r5:c03c0026 r4:c1acf400
Function entered at [<c020761c>] from [<c01970a4>]
Function entered at [<c0196d4c>] from [<c01973b0>]
Function entered at [<c0197354>] from [<c020df1c>]
 r4:00000000
Function entered at [<c020dce0>] from [<c01971bc>]
Function entered at [<c0196d4c>] from [<c01973b0>]
Function entered at [<c0197354>] from [<c017b354>]
 r4:ffc004b0
Function entered at [<c017ade0>] from [<c0197578>]
Function entered at [<c0197510>] from [<c0036660>]
Function entered at [<c00365e0>] from [<c0036734>]
Function entered at [<c00366ec>] from [<c0021080>]
Function entered at [<c0021000>] from [<c0021a54>]
Exception stack(0xc0267f50 to 0xc0267f98)
7f40:                                     00000000 c1be6ce0 00000000 60000013
7f60: c0266000 c02781e8 c0269a28 c0269a20 0001ded0 41069260 0001de00 c0267fb4
7f80: c0267f98 c0267f98 c002290c c0022e98 60000013 ffffffff
 r7:c0269a20 r6:00200000 r5:fdd20200 r4:ffffffff
Function entered at [<c0022e38>] from [<c0215040>]
 r7:c0269a20 r6:c001f200 r5:c02781b0 r4:c027f84c
Function entered at [<c0214fe0>] from [<c0008a24>]
Function entered at [<c00087c0>] from [<00008034>]
 r6:c0020014 r5:c0278288 r4:a0053175
Mem-info:
Normal per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
active_anon:369 inactive_anon:8 isolated_anon:0
 active_file:847 inactive_file:1379 isolated_file:0
 unevictable:0 dirty:0 writeback:0 unstable:0
 free:416 slab_reclaimable:513 slab_unreclaimable:3100
 mapped:334 shmem:35 pagetables:79 bounce:0
Normal free:1664kB min:720kB low:900kB high:1080kB active_anon:1476kB inactive_anon:32kB active_file:3388kB inactive_file:5516kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:32512kB mlocked:0kB dirty:0kB writeback:0kB mapped:1336kB shmem:140kB slab_reclaimable:2052kB slab_unreclaimable:12400kB kernel_stack:360kB pagetables:316kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0
Normal: 42*4kB 43*8kB 42*16kB 9*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1664kB
2261 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap  = 0kB
Total swap = 0kB
8192 pages of RAM
515 free pages
693 reserved pages
3613 slab pages
1847 pages shared
0 pages swap cached
net eth0: received packet spanning multiple descriptors
net eth0: received packet spanning multiple descriptors
net eth0: received packet spanning multiple descriptors
net eth0: received packet spanning multiple descriptors
net eth0: received packet spanning multiple descriptors
net eth0: received packet spanning multiple descriptors
net eth0: received packet spanning multiple descriptors
net eth0: received packet spanning multiple descriptors
root@gateway2:~#

I found in google, that these is marvell GE-driver related: "mv643xx_eth":
"net eth0: received packet spanning multiple descriptors"

So for now, NO backfire rc3 or rc4 or actual trunk is working reliable on
the orion/wrt350nv2. I compared with an Asus WL500gpV1 with exact the same config
and type of NAT-traffic and here are the memory is NOT consumed/memleaked after few minutes.

comment:5 Changed 7 years ago by Ernesto

After Talk to Kaloz, i added here a logfile from starting of making pppoe traffic
and the time where the memory is out and the first ethernet-driver messages comes up in the logfile:

http://openwrt.pastebin.com/N7QYAsfi

comment:6 Changed 7 years ago by Ernesto

And here is an extra log from "slabinfo & top". Starting with no pppoe-Traffic up to the point where the next crash comes up

http://openwrt.pastebin.com/WsFg1X73

comment:7 Changed 7 years ago by Matthew Parslow

I Experience the same crashes - is there anything being done about this?

comment:8 Changed 5 years ago by florian

  • Resolution set to fixed
  • Status changed from new to closed

Add Comment

Modify Ticket

Action
as closed .
The resolution will be deleted. Next status will be 'reopened'.
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.