Modify

Opened 7 years ago

Closed 6 years ago

Last modified 4 years ago

#8781 closed defect (fixed)

Reclaim unused space in WNDR3700 image

Reported by: Mark Mentovai <mark@…> Owned by: developers
Priority: normal Milestone: Barrier Breaker 14.07
Component: base system Version: Trunk
Keywords: Cc: juhosg@…, maddes

Description

Currently, WNDR3700 and WNDR3700v2 images use a fixed flash layout:

wndr3700_mtdlayout=mtdparts=spi0.0:320k(u-boot)ro,128k(u-boot-env)ro,1024k(kernel),6656k(rootfs),64k(art)ro,7680k@0x70000(firmware)
wndr3700v2_mtdlayout=mtdparts=spi0.0:320k(u-boot)ro,128k(u-boot-env)ro,1024k(kernel),14848k(rootfs),64k(art)ro,15872k@0x70000(firmware)

This means that 1MB is reserved for the kernel uImage squashfs regardless of how large the kernel actually is. In my own builds, the kernel uImage squashfs is just over 800kB. If this were rounded up to a 64kB flash erase block size, 832kB could be reserved for the kernel, allowing the rootfs mtd partition to grow by 192kB.

Similarly, if a kernel uImage squashfs ever exceeds 1MB, the build process will silently produce a broken image.

The best way to handle this is to dynamically adjust the mtd partitions given the size of the kernel during the build. This is slightly complicated, because the kernel needs to know the flash layout. The flash layout is passed to the kernel on its command line (mtdparts), and the kernel command line is actually part of the kernel itself, set by patch-cmdline during the build process; the result is lzma-compressed and placed within a squashfs. The compression steps make it difficult to precisely determine how large an mtd partition is needed.

My approach to this problem is to first build an “estimate” kernel uImage squashfs with a command line describing the standard flash layout. The “estimate” kernel’s size is measured, and in order to account for the possibility of poorer compression of the final kernel uImage squashfs, the length of the command line is added to this size. This value is then rounded up to the nearest 64kB and used as the size of the kernel mtd partition. The new rootfs partition is permitted to be as large as possible. These values are placed in the final kernel’s command line with patch-cmdline, and are also used to properly align the kernel and rootfs when the image is produced.

Before

bash$ ls -l openwrt-ar71xx-generic-wndr3700-squashfs-sysupgrade.bin
-rw-r--r-- 1 mark mark 2949124 Jan 31 21:41 openwrt-ar71xx-generic-wndr3700-squashfs-sysupgrade.bin

root@OpenWrt:/# cat /proc/cmdline 
rootfstype=squashfs,jffs2 noinitrd console=ttyS0,115200 board=WNDR3700 mtdparts=spi0.0:320k(u-boot)ro,128k(u-boot-env)ro,1024k(kernel),6656k(rootfs),64k(art)ro,7680k@0x70000(firmware)
root@OpenWrt:/# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/root                 1792      1792         0 100% /rom
tmpfs                    31056        32     31024   0% /tmp
tmpfs                      512         0       512   0% /dev
/dev/mtdblock4            4864       284      4580   6% /overlay
mini_fo:/overlay          1792      1792         0 100% /

After

bash$ ls -l openwrt-ar71xx-generic-wndr3700-squashfs-sysupgrade.bin
-rw-r--r-- 1 mark mark 2752516 Jan 31 21:48 openwrt-ar71xx-generic-wndr3700-squashfs-sysupgrade.bin

root@OpenWrt:/# cat /proc/cmdline 
rootfstype=squashfs,jffs2 noinitrd console=ttyS0,115200 board=WNDR3700 mtdparts=spi0.0:320k(u-boot)ro,128k(u-boot-env)ro,832k(kernel),6848k(rootfs),64k(art)ro,7680k@0x70000(firmware)
root@OpenWrt:/# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/root                 1792      1792         0 100% /rom
tmpfs                    31056        32     31024   0% /tmp
tmpfs                      512         0       512   0% /dev
/dev/mtdblock4            5056       284      4772   6% /overlay
mini_fo:/overlay          1792      1792         0 100% /

On a device with 8MB of flash, every little bit counts, and freeing up 192kB can have a noticeable effect.

Attachments (7)

wndr3700_flash_size.1.patch (4.9 KB) - added by Mark Mentovai <mark@…> 7 years ago.
wndr3700_flash_size.2.patch (4.9 KB) - added by Mark Mentovai <mark@…> 7 years ago.
Update patch to apply after r26605
wndr3700_flash_size.3.patch (4.9 KB) - added by Mark Mentovai <mark@…> 7 years ago.
Update patch to apply after r27053
wndr3700_flash_size.4.patch (4.8 KB) - added by Mark Mentovai <mark@…> 7 years ago.
Update patch to apply after r27899
210-mtd_uimage_split_reworked.patch (9.1 KB) - added by maddes 6 years ago.
Reworked patch from Lantiq, that should work on all platforms
wndr3700_flash_size.5.patch (5.1 KB) - added by Mark Mentovai <mark@…> 6 years ago.
Update patch to apply after r28597
wndr3700_flash_size.6.patch (5.0 KB) - added by Mark Mentovai <mark@…> 6 years ago.
Update patch to apply after r29326

Download all attachments as: .zip

Change History (24)

Changed 7 years ago by Mark Mentovai <mark@…>

comment:1 Changed 7 years ago by anonymous

Great, thanks for the patch! Some 100k more is certainly nice to have at least with the wndr3700v1

comment:2 Changed 7 years ago by anonymous

Seems this could be applied not only to WNDR3700 board.

comment:3 Changed 7 years ago by maddes

  • Cc maddes added

comment:4 Changed 7 years ago by hnyman

Sounds great, but I keep wondering if the double kernel image build could be avoided?

Most of us build repeatedly the firwares with unchanged settings, so the kernel size should not vary much between builds.

Couldn't the 'estimate' step be avoided by storing the kernel size value between builds in a new config file.

The logic might be to

  • check the '.kernelsize' file for stored value from previous build
  • if there is no value, do the estimate step to get the value (or use 832 as default for the first build)
  • use that value (+ 1000 bytes as buffer, rounded to 64k) during the kernel image build process
  • do the build process
  • after the build completes, store the actual image size to .kernelsize for next time
  • compare image size to the reserved size, and if the real size exceeded the reserved size, fail the build and give warning to user (or automatically do a new build with increased size).

comment:5 Changed 7 years ago by Mark Mentovai <mark@…>

hnyman, I’m not too keen on that. The build should be deterministic and shouldn’t rely on artifacts from previous builds. If the kernel is very large (say 1MB) in one build, then you reconfigure things so that the kernel shrinks (say to 832kB), the original inflated kernel shouldn’t cause an extra 192kB to be wasted in the image.

Changed 7 years ago by Mark Mentovai <mark@…>

Update patch to apply after r26605

comment:6 Changed 7 years ago by Thomas

Hi,

maybe this patch from the lantiq target can be moved to the generic patches and used on the other targets:

https://dev.openwrt.org/browser/trunk/target/linux/lantiq/patches/220-mtd_uimage_split.patch

It was tested with nor and serial flashes.

Changed 7 years ago by Mark Mentovai <mark@…>

Update patch to apply after r27053

comment:7 Changed 7 years ago by arokh <trondah@…>

Patch needs an update again :)

Changed 7 years ago by Mark Mentovai <mark@…>

Update patch to apply after r27899

comment:8 Changed 6 years ago by arokh <trondah@…>

Works great, thanks :)

Why isn't this upstream yet?

comment:9 Changed 6 years ago by ddxx0n

+1 for "very useful"

comment:11 Changed 6 years ago by maddes

Replying to comment #6 by Thomas:

maybe this patch from the lantiq target can be moved to the generic patches and used on the other targets:

https://dev.openwrt.org/browser/trunk/target/linux/lantiq/patches-3.0/210-mtd_uimage_split.patch

It was tested with nor and serial flashes.

(link to patch was corrected)

WARNING! Do not use this patch as is - it can brick your device.
Reason is that it does not deal correctly with the Endianness of the uImage header.

In my case (Linksys WRT350N v2, platform Orion/ARM) the kernel and rootfs partition got so huge, that my U-Boot partition at the end of the flash was overwritten during boot.

Additionally:

  • it does not use functionality already available in OpenWrt
  • it uses static structures instead of allocating memory as done in split_squashfs()
  • it creates two splits instead of one
  • it doesn't link the splits to the parent mtd partition

I did a rework of that patch, which is attached to this ticket.
Also I will post it on the developer mailing list for review.

Tip: To test new code which can destroy your flash, boot a ramdisk build.
This way no writes are done to the flash.

Changed 6 years ago by maddes

Reworked patch from Lantiq, that should work on all platforms

comment:12 Changed 6 years ago by maddes

Otherwise I'm still interested in doing it just by scripting.

comment:13 Changed 6 years ago by hnyman <hannu.nyman@…>

I have been using this patch for my trunk builds for some months without problems, until yesterday.

Yesterday I surprisingly ended up with error, as the "rootfs" apparently exceeded the calculated maxsize by 4 bytes (6946820 > 6946816).

I am just wondering if the calculation logic is quite correct there, or if there are edge cases, where the rootfs may slightly exceed the size. Kernel size is rounded up to 64k boundary and that is then subtracted from the total to get rootfs max size. That logic may cause a slightly lower limit for rootfs than needed.

let 'kk = (((s + c) / (64 * 1024) + 1) * 64)'; \
let 'rk = tk - kk'; \ 

Should rk be also bumped up to the next 64k similarly as kk?

comment:14 Changed 6 years ago by Mark Mentovai <mark@…>

If you don’t have room in the image for your rootfs with this patch, you wouldn’t have had room in the image without this patch either.

The logic is as it is so that the beginning of kernel and rootfs are both at the beginning of an erase block.

What are the sizes of your kernel (bin/ar71xx/openwrt-ar71xx-generic-uImage-lzma.bin) and rootfs (bin/ar71xx/openwrt-ar71xx-generic-root.squashfs-64k or bin/ar71xx/openwrt-ar71xx-generic-root.jffs2-64k)? On the WNDR3700 (v1), you have a total of 7.5MB that can be split between the two. This patch allows you to split that 7.5MB up any way that’s possible, as long as each gets an even number of 64kB erase blocks. It sounds like your rootfs has just grown too large. In my builds, I wind up with 896kB for the kernel (the uImage is actually about 850kB) and 6784kB available for the rootfs (the squashfs is actually under 2MB). If the kernel grows into another erase block, there’s less space for rootfs; if the kernel shrinks to free an erase block, there’s more space for rootfs. Without this patch, the WNDR3700 (v1) uses a fixed 1MB for the kernel and 6656kB for the rootfs.

Changed 6 years ago by Mark Mentovai <mark@…>

Update patch to apply after r28597

Changed 6 years ago by Mark Mentovai <mark@…>

Update patch to apply after r29326

comment:15 Changed 6 years ago by Mark Mentovai <mark@…>

This patch was checked in at r29406 with a follow-up checkin at r29431.

comment:16 Changed 6 years ago by swalker

  • Resolution set to fixed
  • Status changed from new to closed

comment:17 Changed 4 years ago by jow

  • Milestone changed from Attitude Adjustment 12.09 to Barrier Breaker 14.07

Milestone Attitude Adjustment 12.09 deleted

Add Comment

Modify Ticket

Action
as closed .
The resolution will be deleted. Next status will be 'reopened'.
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.