Opened 12 years ago

Closed 9 years ago

#607 closed defect (fixed)

ipkg does not handle running out of flash gracefully

Reported by: anonymous Owned by: florian
Priority: high Milestone: Bugs Paradise
Component: base system Version:
Keywords: ipkg space cleanup error-handling Cc:

Description (last modified by mbm)

I tried to install kismet on a 4meg WRT54G and it ran out of space partway through. It left things in a partially installed state, and didn't back out the package changes to a clean state or limit the failure to a single package.

Here's my shell log:

root@OpenWrt:~# ipkg install kismet
Installing kismet (2005-08-R1-1) to root...
Installing kismet-client (2005-08-R1-1) to root...
Installing uclibc++ (0.1.11-2) to root...
Installing libncurses (5.2-7) to root...
Installing kismet-server (2005-08-R1-1) to root...
ipkg: write: No space left on device
ipkg: unable to close `/etc/kismet/client_manuf-ipkg.backup': No space left on device
ipkg: unable to preserve times of `/etc/kismet/client_manuf-ipkg.backup': No space left on device
ipkg: unable to preserve ownership of `/etc/kismet/client_manuf-ipkg.backup': No space left on device
ipkg: unable to preserve permissions of `/etc/kismet/client_manuf-ipkg.backup': No space left on device
file_copy: ERROR: failed to copy /etc/kismet/client_manuf to /etc/kismet/client_manuf-ipkg.backup
Configuring kismet-client
Configuring libncurses
Configuring uclibc++
ipkg_conf_write_status_files: Can't open status file: //usr/lib/ipkg/status for writing: No space left on device
pkg_write_filelist pkg=kismet-client returned 28
pkg_write_filelist pkg=libncurses returned 28
pkg_write_filelist pkg=uclibc++ returned 28
An error ocurred, return value: -1.
Collected errors:
backup_make_backup: Failed to copy /etc/kismet/client_manuf to /etc/kismet/client_manuf-ipkg.backup
Could not open //usr/lib/ipkg/info/kismet-client.list for writing: No space left on device
Could not open //usr/lib/ipkg/info/libncurses.list for writing: No space left on device
Could not open //usr/lib/ipkg/info/uclibc++.list for writing: No space left on device

An ipkg list_installed shows that the damage includes /none/ of the packages being marked as installed:

root@OpenWrt:~# ipkg list_installed
base-files - 8 - OpenWrt filesystem structure and scripts
base-files-brcm - 2 - Board/architecture specific files
bridge - 1.0.6-1 - Ethernet bridging tools
busybox - 1.00-3 - Core utilities for embedded Linux systems
dnsmasq - 2.27-1 - A lightweight DNS and DHCP server
dropbear - 0.48.1-1 - a small SSH 2 server/client designed for small memory environments.
haserl - 0.8.0-1 - a CGI wrapper to embed shell scripts in HTML documents
ipkg - 0.99.149-2 - lightweight package management system
iptables - 1.3.3-2 - The netfilter firewalling software for IPv4
iwlib - 28.pre7-1 - Library for setting up WiFi cards using the Wireless Extension
kernel - 2.4.30-brcm-3 - 
kmod-brcm-wl - 2.4.30-brcm-3 - Proprietary driver for Broadcom Wireless chipsets
kmod-diag - 2.4.30-brcm-3 - Driver for Router LEDs and Buttons
kmod-ppp - 2.4.30-brcm-3 - PPP support
kmod-pppoe - 2.4.30-brcm-3 - PPP over Ethernet support
kmod-switch - 2.4.30-brcm-1 - switch driver for robo/admtek switch
kmod-tun - 2.4.30-brcm-3 - Kernel TUN/TAP extension
kmod-wlcompat - 2.4.30-brcm-3 - Compatibility module for using the Wireless Extension with broadcom's wl
liblzo - 1.08-1 - a real-time data compression library
libopenssl - 0.9.7i-2 - OpenSSL (Secure Socket Layer) libraries
libpcap - 0.9.4-1 - a low-level packet capture library
libpopt - 1.7-4 - a command line option parsing library
mtd - 4 - Tool for modifying the flash chip
nas - 3.90.37-16 - Proprietary Broadcom WPA Authenticator/Supplicant
ntpclient - 2003_194-2 - NTP client for setting system time from NTP servers.
nvram - 1 - NVRAM utility and libraries for Broadcom hardware
openvpn - 2.0.5-1 - Open Source VPN solution using SSL
ppp - 2.4.3-7 - a PPP (Point-to-Point Protocol) daemon (with MPPE/MPPC support)
ppp-mod-pppoe - 2.4.3-7 - a PPPoE (PPP over Ethernet) plugin for PPP
rsync - 2.6.5-0 - utility that provides fast incremental file transfer
tcpdump - 3.8.3-1 - A tool for network monitoring and data acquisition.
uclibc - 0.9.27-8 - Standard C library for embedded Linux systems
webif - 0.2-1 - A modular, extensible web interface for OpenWrt.
wificonf - 6 - Replacement utility for wlconf
wireless-tools - 28.pre7-1 - Tools for setting up WiFi cards using the Wireless Extension
wl - 3.90.37-1 - Proprietary Broadcom utility for setting wireless driver parameters
Successfully terminated.

I know there is no good way to predict if the filesystem will run out of space /before/ installation is attempted, but surely ipkg can be made to handle cleaning up after itself after-the-fact if it does happen, right?

Attachments (0)

Change History (14)

comment:1 Changed 11 years ago by anonymous

Oh, and ipkg is also able to brick the device, if actually busybox is beeing updated while running out of space. Because busybox was corrupted, I had to reinstall OpenWRT, because afterwards no command worked any more.

comment:2 Changed 11 years ago by nbd

  • Milestone changed from 0.9/rc6 to Kamikaze

comment:3 Changed 11 years ago by florian

  • Owner changed from developers to florian
  • Status changed from new to assigned

comment:4 Changed 11 years ago by florian

  • Component changed from packages to base system
  • Milestone changed from Kamikaze to Afterburner 7.09
  • Priority changed from normal to high

comment:5 follow-up: Changed 10 years ago by anonymous

Any workarounds for a corrupt busybox? That just happened to me - ran out of space while upgrading busybox, and now every command returns a bus error. Naturally I can't start a new ssh shell, either...

comment:6 in reply to: ↑ 5 Changed 10 years ago by Bartman007

Replying to anonymous:

Any workarounds for a corrupt busybox? That just happened to me - ran out of space while upgrading busybox, and now every command returns a bus error. Naturally I can't start a new ssh shell, either...

Booting into failsafe mode bypasses all changes made since the firmware was flashed. Once in failsafe mode you can reinitialize the JFFS2 partition, or mount it and attempt to fix the issue.

comment:7 Changed 10 years ago by mbm

  • Description modified (diff)

comment:8 Changed 10 years ago by mbm

The problem with this bug is the concept of compressed filesystems; we know how much space we physically have, but we don't know how much data we can compress into that space. Hence even if ipkg were to check the available space before installing, it wouldn't be sufficient.

The second problem is that the ipkg status file is a part of the filesystem, and once you run out of space you can't manipulate the status file, forcing you to manually delete the items until there's such space on the filesystem.

comment:9 Changed 10 years ago by phil@…

A more graceful failure is needed here.

One way to avoid trashing the ipkg.status file so *completely* would be to create the new file before deleting the old one. That is, write (using a unique name mechanism), then rename the old file, then rename the new file, then remove the old file.

''write into''
mv ipkg.status ipkg.status.old
mv ipkg.status
rm ipkg.status.old

Another feature that would help save the system is to stop installing dependent packages if one earlier package install fails. I realize this is not ideal, since some failures are tolerable. Maybe the fatal failure should occur only if a "no space on device" error is encountered.

In the specific case of installing new ipkg's, the ideal situation is to have a failed install "do the right thing" and properly remove all changes that it made. For starters, simply deleting new files would be great. Undoing changes to extant files would be nice, but possibly unreasonable to expect.

comment:10 Changed 10 years ago by spudz76@…

This happened to me during an 'ipkg upgrade'. Yes, it blows up most commands if busybox gets half-installed, or if a new uClibc gets half-installed. I came up with a way to make it work again, without using failsafe mode, except you must not reboot before fixing it or you will be forced to use failsafe or reflash. If you 'rm /bin/busybox' to kill the broken version, and then 'ln -s /rom/bin/busybox /bin/busybox', things should return to relative sanity (by using the same binary that failsafe would use). If that doesn't do it then check /lib/libuClibc*.so and try the same workaround (rm the JFFS copy and symbolic link the squashfs/'rom' version). Then once things act sane again, clear out some more space by deleting the largest files you can find (that live in JFFS land) and try to reinstall just busybox by itself. Once that works, you can either reinstall the packages you had to massacre to make space, or if they were unnecessary in the first place just 'ipkg remove' them (it won't care if some files of the package are already gone). Then install what you really needed, and hope there's enough space.

A suggestion for space prediction, could it at least have a package size that could be checked against free space, certainly if free space is already < total space requirement ipkg could abort the rest of the pending packages, and clean up or complete the configuration steps for things it already installed. For that matter, couldn't each package have a known compressed size based on the fact that each file is going to compress the same as it would any other day of the week with the same compressor? Like a known compression ratio for the package, or total compressed size along with the total uncompressed size?

comment:11 Changed 9 years ago by anonymous

Would it be possible for ipkg to do a trial decompress, then count the bytes? That way, it knows how much it will take up before writing it to the filesystem.

comment:12 Changed 9 years ago by nbd

it can't know the effective size before decompressing, because the filesystem is compressed.

comment:13 Changed 9 years ago by nbd

  • Milestone changed from Kamikaze 8.09 RC1 to Kamikaze Bugs Paradize

comment:14 Changed 9 years ago by jow

  • Resolution set to fixed
  • Status changed from assigned to closed

opkg now has a free space check

Add Comment

Modify Ticket

as closed .
The resolution will be deleted. Next status will be 'reopened'.

E-mail address and user name can be saved in the Preferences.

Note: See TracTickets for help on using tickets.