Modify

Opened 5 years ago

Closed 5 years ago

Last modified 4 years ago

#12436 closed defect (fixed)

hotplug2 leaks memory in AA12.09-beta2

Reported by: anonymous Owned by: developers
Priority: normal Milestone: Barrier Breaker 14.07
Component: base system Version: Attitude Adjustment 12.09 Beta
Keywords: Cc:

Description

I had a couple spare WRT54GS with 10.03.1 installed and default configuration.

I started with one of them, telnet in, set password, set lan IP address, disable dhcp server on lan, reboot. Next, login to webui and install 12.09beta2 leaving the Keep Settings box checked. After it upgraded and rebooted, I logged back into webui to confirm version and left that page open in the browser. I connected a ssh session and ran top to see the memory stats, then walked away.

I returned more than 12hours later and took a look at the situation:
Mem: 27392K used, 1876K free, 0K shrd, 400K buff, 1076K cached
CPU: 0% usr 99% sys 0% nic 0% idle 0% io 0% irq 0% sirq
Load average: 4.69 4.20 4.09 2/35 32555
Uptime 12h 55m 48s
Not good! Free ram dropped nearly 13MB in almost 13hours, so it appears something is leaking at a rate of slightly under 1MB/hour.

A quick look at ps reveals the culprit:
510 root 11832 R /sbin/hotplug2 --override --persistent --set-rules-file /etc/hotplug2.rules --set-coldplug-cmd /sbin/udev

Since I have two identical boxes, I did an experiment next. I took the second one and upgraded it it to 12.09-beta2 with the Keep Settings box UNchecked. After the upgrade, I went in with telnet, set the password, set the lan IP address, set dhcp server disabled on lan, reboot. I opened ssh to start top and also opened the webui. Resident size of hotplug2 just after boot: 844K.

I rebooted the first one that had been leaking so now both are running with uptime within 5min of each other. One has a config that was migrated and one has a fresh configuration created this version. In the time it has taken me to open this ticket, they have reached 20min uptime and both show hotplug2 with resident size over 1600. So, the config does not have any influence, hotplug2 just leaks!

Attachments (0)

Change History (7)

comment:1 Changed 5 years ago by chris hoogenboom <chris@…>

Add me to the list on this, just got OOM error and hotplug2 the culprit.

OpenWrt Attitude Adjustment 12.09-beta2 / LuCI 0.11 Branch (0.11+svn9364)

Dec 12 21:23:20 OpenWrt kern.warn kernel: [98516.090000] grep invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Dec 12 21:23:20 OpenWrt kern.warn kernel: [98516.170000] Call Trace:[<80072950>] 0x80072950
Dec 12 21:23:20 OpenWrt kern.warn kernel: [98516.220000] [<802644dc>] 0x802644dc
Dec 12 21:23:20 OpenWrt kern.warn kernel: [98516.260000] [<802644dc>] 0x802644dc
Dec 12 21:23:20 OpenWrt kern.warn kernel: [98516.310000] [<800ac980>] 0x800ac980
Dec 12 21:23:20 OpenWrt kern.warn kernel: [98516.350000] [<800acb98>] 0x800acb98
Dec 12 21:23:20 OpenWrt kern.warn kernel: [98516.390000] [<800acaf0>] 0x800acaf0
Dec 12 21:23:20 OpenWrt kern.warn kernel: [98516.430000] [<800ad150>] 0x800ad150
Dec 12 21:23:20 OpenWrt kern.warn kernel: [98516.470000] [<800aff94>] 0x800aff94
Dec 12 21:23:20 OpenWrt kern.warn kernel: [98516.510000] [<800abe6c>] 0x800abe6c
Dec 12 21:23:20 OpenWrt kern.warn kernel: [98516.560000] [<800cac50>] 0x800cac50
Dec 12 21:23:20 OpenWrt kern.warn kernel: [98517.020000]  unevictable:0 dirty:0 writeback:0 unstable:0
Dec 12 21:23:20 OpenWrt kern.info kernel: [98518.340000] [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Dec 12 21:23:20 OpenWrt kern.info kernel: [98518.430000] [  621]     0   621      379       20   0       0             0 syslogd
Dec 12 21:23:20 OpenWrt kern.info kernel: [98518.520000] [  623]     0   623      375       17   0       0             0 klogd
Dec 12 21:23:20 OpenWrt kern.info kernel: [98518.610000] [  625]     0   625     3856     3683   0       0             0 hotplug2
Dec 12 21:23:20 OpenWrt kern.info kernel: [98518.700000] [  631]     0   631      216       15   0       0             0 ubusd
Dec 12 21:23:20 OpenWrt kern.info kernel: [98518.790000] [  661]     0   661      379       47   0       0             0 netifd
Dec 12 21:23:20 OpenWrt kern.info kernel: [98518.880000] [ 1432]     0  1432      288       20   0       0             0 dropbear
Dec 12 21:23:20 OpenWrt kern.info kernel: [98518.980000] [ 1482]     0  1482      286       32   0       0             0 uhttpd
Dec 12 21:23:20 OpenWrt kern.info kernel: [98519.070000] [ 1497]     0  1497      377       18   0       0             0 watchdog
Dec 12 21:23:20 OpenWrt kern.info kernel: [98519.160000] [ 1567] 65534  1567      234       20   0       0             0 dnsmasq
Dec 12 21:23:20 OpenWrt kern.info kernel: [98519.250000] [ 1574]     0  1574      422      100   0       0             0 lua
Dec 12 21:23:20 OpenWrt kern.info kernel: [98519.340000] [ 1600]     0  1600      377       19   0       0             0 ntpd
Dec 12 21:23:20 OpenWrt kern.info kernel: [98519.430000] [13317]     0 13317      384       38   0       0             0 pppd
Dec 12 21:23:20 OpenWrt kern.info kernel: [98519.520000] [20826]     0 20826      305       39   0       0             0 dropbear
Dec 12 21:23:20 OpenWrt kern.info kernel: [98519.610000] [20923]     0 20923      378       21   0       0             0 ash
Dec 12 21:23:20 OpenWrt kern.info kernel: [98519.700000] [15156]     0 15156      774      285   0       0             0 luci
Dec 12 21:23:20 OpenWrt kern.info kernel: [98519.780000] [15167]     0 15167      377       31   0       0             0 sh
Dec 12 21:23:20 OpenWrt kern.info kernel: [98519.870000] [15168]     0 15168      227       37   0       0             0 dnsmasq
Dec 12 21:23:20 OpenWrt kern.info kernel: [98519.960000] [15169]     0 15169      374       43   0       0             0 grep
Dec 12 21:23:20 OpenWrt kern.err kernel: [98520.050000] Out of memory: Kill process 625 (hotplug2) score 472 or sacrifice child
Dec 12 21:23:20 OpenWrt kern.err kernel: [98520.140000] Killed process 625 (hotplug2) total-vm:15424kB, anon-rss:14692kB, file-rss:40kB

comment:2 Changed 5 years ago by chris hoogenboom <chris@…>

Noticed that logging in to LUCI web interface spawns a 2nd/duplicate hotplug process which seems well behaved but the original hotplug process's memory usage grow 4 KB every 4 seconds or so.

root@OpenWrt:~# date ; ps | grep hot
Thu Dec 13 15:48:34 GMT 2012
  625 root      1344 S    /sbin/hotplug2 --override --persistent --set-rules-file /etc/hotplug2.rules --set-coldplug-cmd /sbin/udevtrigger --max-children 1
11217 root      1092 S    /sbin/hotplug2 --override --persistent --set-rules-file /etc/hotplug2.rules --set-coldplug-cmd /sbin/udevtrigger --max-children 1
18751 root      1500 S    grep hot
root@OpenWrt:~# date ; ps | grep hot
Thu Dec 13 15:48:41 GMT 2012
  625 root      1352 S    /sbin/hotplug2 --override --persistent --set-rules-file /etc/hotplug2.rules --set-coldplug-cmd /sbin/udevtrigger --max-children 1
11217 root      1092 S    /sbin/hotplug2 --override --persistent --set-rules-file /etc/hotplug2.rules --set-coldplug-cmd /sbin/udevtrigger --max-children 1
18948 root      1500 S    grep hot
root@OpenWrt:~# 

comment:3 Changed 5 years ago by chris hoogenboom <chris@…>

More debuhg showed it had somethnig to do with lua call to iwinfo.txpower_offset triggering a hotplug event. It only happened when I had client mode ocnfigured on the wifi interface. Unfortunately when I only figured this out after I deleted and recreated the wifi interface after which the problem wnet away and I have not been able to reproduce it. So bottom line is its a bug thats in iwinfo.txpower_offset causing a hotplug event when the wifi interface is in a client configuration.

comment:4 Changed 5 years ago by patrik@…

Well I have AA RC1 on TP-Link TL-WR740N/ND v4 (4.23) and hotplug also have memory leak, pretty major one. The memory consumption increase seems to be dependent on data flowing through router / activity time.

The bug is fatal on my router, it automatically reboots every couple hours without logging anything.

I am now killing and starting hotplug whenever the memory consumption gets huge, but this bug needs fix asap...

comment:5 Changed 5 years ago by patrik@…

So I did some additional testing and hotplug is leaking memory only if you have web interface of router open in the browser. After it is closed, the memory is NOT released.

comment:6 Changed 5 years ago by nbd

  • Resolution set to fixed
  • Status changed from new to closed

fixed in r35857, r35859

comment:7 Changed 4 years ago by jow

  • Milestone changed from Attitude Adjustment 12.09 to Barrier Breaker 14.07

Milestone Attitude Adjustment 12.09 deleted

Add Comment

Modify Ticket

Action
as closed .
The resolution will be deleted. Next status will be 'reopened'.
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.