Modify

Opened 7 years ago

Last modified 4 years ago

#7768 accepted defect

block-extroot on WRT54GL (v1.1) SD card (mmc) overlay double mount problem

Reported by: lance@… Owned by: cshore
Priority: response-needed Milestone: Barrier Breaker 14.07
Component: packages Version: Backfire 10.03.1 RC3
Keywords: block-extroot response-needed Cc:

Description

I have added a 1G SD card to the WRT54GL and want to use it to increase the volume of packages I can install by using it for /overlay. After installing (recipe to follow) when I boot I find that the internal flash and the SD card are both reported as mounted on /overlay and both report near 900M free space. If I install packages however I soon reach the point where the internal flash would have been full and the process stops with disk full messages. At this point I cannot write anything to either flash or SD and can only recover by fully reinstalling. If I mount the SD on /mnt, I can read and write it without any problems.

Install recipe:

First:
        Install 10.03.1-rc1 via tftp
        Set LAN address and root password then reboot.

Install:
        kmod-mmc-over-gpio
        kmod-fs-ext3
        block-mount
        block-hotplug
        block-extroot
        e2fsprogs
        fdisk

Add: echo 0x9c > /proc/diag/gpiomask to /etc/init.d/mmc_over_gpio just before device is started.

Set:     /etc/config/mmc_over_gpio
        config 'mmc_over_gpio'
                option 'name' 'default'
                option 'enabled' '1'
                option 'DI_pin' '2'
                option 'DO_pin' '4'
                option 'CLK_pin' '3'
                option 'CS_pin' '7'
                option 'mode' '0'

Run:
        /etc/init.d/mmc_over_gpio enable
        /etc/init.d/mmc_over_gpio start

Prep to mount as the overlay fs:
        Edit /etc/config/fstab
        config mount
                option target   /overlay
                option device   /dev/mmcblk0p1
                option fstype   ext3
                option options  rw,sync
                option enabled_fsck 1
                option is_rootfs 1
                option enabled  1

        Remove the swap entry as not required.

Prep SD:
        fdisk /dev/mmcblk0         (make partitions - probably just 1 large one)
        mke2fs -j /dev/mmcblk0p1   (this will take a long time to run - around 20 minutes for 1G)


Mount the mmc partition somewhere (/mnt) and copy all of /overlay into it using:
        mount /dev/mmcblk0p1 /mnt
        cp -ar /overlay/* /mnt/.
        umount /mnt

Mount:
        /etc/init.d/fstab enable

REBOOT

This results in the following in the serial console boot sequence (not all lines shown):

...
- preinit -
Press the [f] key and hit [enter] to enter failsafe mode
- regular preinit -
switching to jffs2
mini_fo: using base directory: /
mini_fo: using storage directory: /overlay
- init -
...
sysctl net.netfilter.nf_conntrack_acct=1 to enable it.
gpio-mmc: Failed to request mmc_spi module.
mmc_spi spi32766.0: ASSUMING 3.2-3.4 V slot power
mmc_spi spi32766.0: SD/MMC host mmc0, no DMA, no WP, no poweroff
gpio-mmc: MMC-Card "default" attached to GPIO pins di=2, do=4, clk=3, cs=7
mmc_spi spi32766.0: can't change chip-select polarity
mmc0: host does not support reading read-only switch. assuming write-enable.
mmc0: new SD card on SPI
mmcblk0: mmc0:0000 SD 973 MiB
 mmcblk0: p1
kjournald starting.  Commit interval 5 seconds
EXT3 FS on mmcblk0p1, internal journal
EXT3-fs: mounted filesystem with writeback data mode.

But the following commands show the strange double overlay mount:

root@OpenWrt:/# mount
rootfs on / type rootfs (rw)
/dev/root on /rom type squashfs (ro,relatime)
none on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime)
tmpfs on /dev type tmpfs (rw,relatime,size=512k)
devpts on /dev/pts type devpts (rw,relatime,mode=600)
/dev/mtdblock3 on /overlay type jffs2 (rw,relatime)
mini_fo:/overlay on / type mini_fo (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
configfs on /config type configfs (rw,relatime)
/dev/mmcblk0p1 on /overlay type ext3 (rw,sync,relatime,errors=continue,data=writeback)
root@OpenWrt:/# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 1.9M      1.9M         0 100% /rom
tmpfs                     6.5M     48.0K      6.5M   1% /tmp
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/mtdblock3          957.8M     18.8M    890.4M   2% /overlay
mini_fo:/overlay          1.9M      1.9M         0 100% /
/dev/mmcblk0p1          957.8M     18.8M    890.4M   2% /overlay

This happens for both 10.03 and 10.03.1-rc1

Provided that I do not install extra packages and fill (and so jam) the flash, I can edit the /etc/config/fstab to change the mount location away from /overlay to /mnt and reboot. Having done this the SD auto mounts onto /mnt. This suggests to me that the writes are ending up in the internal flash overlay. The changed mount location is certainly still showing in /etc/config/fstab after the reboot.

After the change of mount to /mnt I get:

root@OpenWrt:/# mount
rootfs on / type rootfs (rw)
/dev/root on /rom type squashfs (ro,relatime)
none on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime)
tmpfs on /dev type tmpfs (rw,relatime,size=512k)
devpts on /dev/pts type devpts (rw,relatime,mode=600)
/dev/mtdblock3 on /overlay type jffs2 (rw,relatime)
mini_fo:/overlay on / type mini_fo (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
configfs on /config type configfs (rw,relatime)
/dev/mmcblk0p1 on /mnt type ext3 (rw,sync,relatime,errors=continue,data=writeback)
root@OpenWrt:/# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 1.9M      1.9M         0 100% /rom
tmpfs                     6.5M     48.0K      6.5M   1% /tmp
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/mtdblock3            1.0M    776.0K    248.0K  76% /overlay
mini_fo:/overlay          1.9M      1.9M         0 100% /
/dev/mmcblk0p1          957.8M     18.8M    890.4M   2% /mnt

The files on /mnt do not show any of the changes made while mounted as /overlay since the original copy to the newly made partition.

Please let me know if you would like me to make any further tests or provide more information.
Thanks.

Attachments (1)

files.tgz (1.0 KB) - added by lance@… 7 years ago.

Download all attachments as: .zip

Change History (27)

comment:1 Changed 7 years ago by anonymous

Try with the installed packages included directly into the image. Not with opkg. Then it should work.

Changed 7 years ago by lance@…

comment:2 Changed 7 years ago by lance@…

I have built an image from the image builder kit:

http://backfire.openwrt.org/10.03/brcm47xx/OpenWrt-ImageBuilder-brcm47xx-for-Linux-i686.tar.bz2

using the packages I require to get it going and their dependencies:

make image PACKAGES="kmod-mmc-over-gpio kmod-mmc-spi kmod-mmc kmod-crc-itu-t kmod-crc7 kmod-spi-gpio-old kmod-spi-bitbang kmod-fs-ext3 kmod-fs-mbcache block-mount block-hotplug block-extroot e2fsprogs libblkid libuuid libext2fs libpthread librt fdisk kmod-gpio-dev gpioctl" FILES=../files/

The files directory contains the modified start up files as previously described:

../files/./etc/init.d/mmc_over_gpio
../files/./etc/config/mmc_over_gpio
../files/./etc/config/fstab

(I'll try to attach a tar of these)

The result is the same with the double mount and the quickly over filling /overlay

comment:3 Changed 7 years ago by nico

  • Owner changed from developers to cshore
  • Status changed from new to assigned

comment:4 Changed 7 years ago by anonymous

This is actually not a bug in preinit or block-extroot, but rather the files you are using that aren't yet part of OpenWRT. I will try to make time to look at them to see what's wrong with them.

comment:5 Changed 7 years ago by lance@…

Thanks. The extra 3 files are derived from similar files created by one of the standard OpenWRT images when I install the relevant packages for accessing the SD card, edited with settings required for the SD GPIO connection and mount. I assume the system will need these directly in the squashfs image in order to access the SD based overlay at boot time.

comment:6 Changed 7 years ago by anonymous

That's right. What packages do you have installed (relevant). E.g. block-extroot broadcom-mmc-gpio-sdhc or whatever it is, etc.

comment:7 Changed 7 years ago by lance@…

The version I am currently testing with is based on the image builder tar listed above with the following packages added before making the image (I call explicitly for all the dependants that an opkg install would pull in - not sure if the image builder does that automatically or not):

kmod-mmc-over-gpio
kmod-mmc-spi
kmod-mmc
kmod-crc-itu-t
kmod-crc7
kmod-spi-gpio-old
kmod-spi-bitbang
kmod-fs-ext3
kmod-fs-mbcache
block-mount
block-hotplug
block-extroot
e2fsprogs
libblkid
libuuid
libext2fs
libpthread
librt
fdisk
kmod-gpio-dev
gpioctl

One think I have noticed is that the start up script /lib/functions/extmount.sh greps the module info files in /etc/modules.d for any containing the line "# May be required for rootfs" and that some that seem to be needed for GPIO MMC access ("93-mmc-over-gpio" amongst others I think) do not have this. Following from that, assuming that the file needs this line adding, the 93-... file can specify params for the driver that are passed to insmod. I assume I will need to pass in the GPIO pin assignments at this point (is there an equivalent of modinfo to find what they are called ?) to bring the card up early enough in the boot sequence to use for the overlay, unless there is a clever switch from the flash overlay to the mmc later during booting when more things are up.

comment:8 Changed 7 years ago by lance@…

I've tried again, this time using the backfire 10.03 brcm-2.4 ImageBuilder and following the instructions in the following discussion:

https://forum.openwrt.org/viewtopic.php?pid=107264

This time I am using the build command:

make image PACKAGES="block-extroot block-mount block-hotplug kmod-fs-ext3 kmod-broadcom-sdhc e2fsprogs libblkid libuuid libext2fs libpthread librt fdisk" FILES=../files_brcm-2.4/

The files included are the following, all of which are as described in the referenced discussion:

        files_brcm-2.4/.
        files_brcm-2.4/./lib
        files_brcm-2.4/./lib/preinit
        files_brcm-2.4/./lib/preinit/50_determine_usb_root
        files_brcm-2.4/./etc
        files_brcm-2.4/./etc/sdcard.conf
        files_brcm-2.4/./etc/config
        files_brcm-2.4/./etc/config/fstab
        files_brcm-2.4/./etc/modules.d
        files_brcm-2.4/./etc/modules.d/33-sdhc

The results now are just as expected, with the sizes and mounts reported as follows:

Filesystem                Size      Used Available Use% Mounted on
/dev/root                 1.3M      1.3M         0 100% /rom
tmpfs                     7.0M      1.0M      6.0M  14% /tmp
/dev/mtdblock/4           1.9M    324.0K      1.6M  16% /tmp/overlay
/dev/sdcard/part1       957.8M     20.5M    888.7M   2% /overlay
mini_fo:/overlay          1.3M      1.3M         0 100% /
rootfs on / type rootfs (rw)
/dev/root on /rom type squashfs (ro)
none on /dev type devfs (rw)
none on /proc type proc (rw)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw)
/dev/mtdblock/4 on /tmp/overlay type jffs2 (rw)
/dev/sdcard/part1 on /overlay type ext3 (rw)
mini_fo:/overlay on / type mini_fo (rw)

I've also installed a number of packages on the unit, and now they all fit, so I assume they are now ending up on the SD rather then the internal flash.

The startup on the serial console now shows the SD initialising before the overlay is applied:

- preinit -
b44: eth0: Link is up at 100 Mbps, full duplex.
b44: eth0: Flow control is off for TX and off for RX.
Press the [f] key and hit [enter] to enter failsafe mode
- regular preinit -
b44: eth0: Link is up at 100 Mbps, full duplex.
b44: eth0: Flow control is off for TX and off for RX.
Journalled Block Device driver loaded
[INF] sdhc: Version: 2.0.2  Parms: major=0 din=2 dout=4 clk=3 cs=7 maxsec=32 rahead=2
[INF] sdhc: SD Card detected & initialized
[INF] sdhc: Assigned dynamic major number 254
Partition check:
 /dev/sdcard: p1
[INF] sdhc: Module loaded
kjournald starting.  Commit interval 5 seconds
EXT3-fs warning: checktime reached, running e2fsck is recommended
EXT3 FS 2.4-0.9.19, 19 August 2002 on sdcard(254,1), internal journal
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
switching to external rootfs
mini_fo: using base directory: /
mini_fo: using storage directory: /overlay
b44: eth0: Link is up at 100 Mbps, full duplex.
b44: eth0: Flow control is off for TX and off for RX.
- init -

Is the problem is specific to the 2.6 kernel based image builds. Perhaps if I had supplied the SD card driver with the correct pin assignments for my card that would have worked also, but didn't know how to do that.

comment:9 Changed 7 years ago by jow

  • Resolution set to invalid
  • Status changed from assigned to closed

Config issue

comment:10 Changed 7 years ago by anonymous

  • Resolution invalid deleted
  • Status changed from closed to reopened

Resolved my ass. I did the same thing on WHRG54s and either I get sd card mounted to target or not at all. If I put /overlay as target I get this situation. I've tried everything I can think of to get it to work and it won't. I think sd card modules aren't getting executed early enough.

comment:11 Changed 7 years ago by cshore

  • Resolution set to invalid
  • Status changed from reopened to closed

Prior a recent trunk commit, using target /overlay was the incorrect thing to do. Set is_rootfs to to 1, and make sure that the target is *not* /overlay (as the wiki says). This is because if the rootfs fails to mount during preinit, there is already something mounted on /overlay (the jffs2), but then fstab mounts the failed rootfs as /overlay as well (because that was the mount target).

What you are seeing is a failure of the overlay to mount during preinit, and instead mounting during normal boot.

It is a config issue. Therefore this bug is invalid.

Not only that but in recent trunk it is no longer even an easily made config issue, as using /overlay as the target will become the preferred option, and is_rootfs is not necessary (failed mounts will be on /tmp/overlay-disabled).

comment:12 Changed 7 years ago by cshore

  • Resolution invalid deleted
  • Status changed from closed to reopened

comment:13 Changed 7 years ago by cshore

  • Keywords response-needed added

This bug is about the double mounted overlay. If the problem you are having is that SD (mmc) is not mounting during preinit, but only during regular boot, then you need to file a proper report of that bug, without doing a config issue like using target /overlay in backfire.

comment:14 Changed 7 years ago by cshore

  • Resolution set to invalid
  • Status changed from reopened to closed

comment:15 Changed 7 years ago by anonymous

Collecting logs, will post bug report. Further research shows something IS wrong with preinit scripts.

comment:16 Changed 7 years ago by anonymous

  • Resolution invalid deleted
  • Status changed from closed to reopened

I faced the same issue and could now identify why extroot is not beeing used at preinit state.
But I don't know whether there is a problem in the code or in my config.

I used the last rev r26741. I was always wondering why the extroot was not used and I always recieved the message:

sdcard - Card detected and initialized
switching to jffs2
mini_fo: using base directory: /
mini_fo: using storage directory: /overlay
- init -

With the result that my SD card was mounted later on to /overlay.

After I figured out that the file "50_determine_usb_root" from the package "block-extroot" would need set the var "pi_extroot_mount_success" to "true" in order to successfully mount a device other than the jffs, I added some debug lines to the file.

I noticed that during the mount process of the file "50_determine_usb_root" the ext2 module was still not loaded which ended up with jffs is beeing used.

sdcard - Card detected and initialized
**** /etc/config/fstab before load
***
Module                  Size  Used by    Not tainted
sdhc                   20576  0 
switch-robo             5724  0 
switch-core             5104  0 switch-robo
diag                   57968  0 
***
*** pi_extroot_mount_success= ***
switching to jffs2
mini_fo: using base directory: /

mini_fo: using storage directory: /overlay

- init -

So simply "insmod ext2" fixed my problem, but don't think this is on purpose. I could not figured out where the modules for preinit will be loaded.

comment:17 Changed 7 years ago by cshore

  • Milestone changed from Backfire 10.03.1 to Attitude Adjustment (trunk)
  • Priority changed from normal to response-needed
  • Status changed from reopened to accepted
  • Version changed from Backfire 10.03.1 RC1 to Backfire 10.03.1 RC3

ext2 module loading during preinit has been fixed AFAICT, so if you could test whether this is still any issue?

comment:18 Changed 7 years ago by windchine@…

I'm trying to test this problem with 10.03.1-rc5 but cannot get the SD to work even as a normal mount with brcm47xx (it works correctly as a normal mount with brcm-2.4). I've posted full details on #9582 which refers to a similar problem. If I can get past this problem I'll give extroot a try.

comment:19 Changed 7 years ago by windchine@…

Well I can mount the SD ok as a normal mount, but only on one occasion did the SD mount on /overlay and then only directly after a tftp install. After a reboot it was back on /mnt. I think a good part of the problem is that I don't have a decent starting point to work from with the configuration. I appreciate the superb work that is going into openwrt, but with the wiki being restructured and tracking the trunk and forums discussing evolving configurations, the methods that should work with the individual releases are really hard to fathom out. It is a shame that this makes using parts like extroot on SD (which have huge potential) a nightmare to get working or even to get a foot hold on helping to debug. The Linux kernel team deal with this by including a snapshot of the documentation text files with each release. A similar directory along side the target directories for each main and -rcX release would be invaluable provided it was taken from a master that is maintained along with the source. Sorry this is a bit off subject - but I think it would help everyone get the best out of this otherwise excellent embedded OS. Thanks for all your hard work.

comment:20 Changed 7 years ago by jow

Well the truth is that there is no starting point. SD-Card mods are an unsupported hardware-modification, using techniques that where never meant to be used as such. The fact that the driver for the mmc card stuff requires early boot configuration complicates it further.

You're part of a tiny minority here - not one single active developer I know has an sd-card mod to even test it - you might overestimate the huge potential here.

comment:21 Changed 6 years ago by anonymous

jow you suck

comment:22 Changed 6 years ago by David Campbell <david@…>

I'm seeing this same double-mounting problem in OpenWrt attitude adjustment with extroot onto a usb stick, with both "New external overlay variant" and "Whole external root" as documented in http://wiki.openwrt.org/doc/howto/extroot

basically it doesn't work if you follow the documentation

comment:23 Changed 6 years ago by David Campbell <david@…>

Just following up on my previous post, I'm seeing the problem in r31296

comment:24 Changed 6 years ago by David Campbell <david@…>

Here's what I see:

root@OpenWrt:~# df
Filesystem 1K-blocks Used Available Use% Mounted on
rootfs 1664 1580 84 95% /
/dev/root 1280 1280 0 100% /rom
tmpfs 15048 116 14932 1% /tmp
tmpfs 512 0 512 0% /dev
/dev/mtdblock5 1664 1580 84 95% /overlay
overlayfs:/overlay 1664 1580 84 95% /
/dev/sda1 1664 1580 84 95% /

root@OpenWrt:~# uname -a
Linux OpenWrt 3.2.14 #1 Tue Apr 17 00:54:55 EST 2012 mips GNU/Linux

root@OpenWrt:~# grep "r[0-9]" /etc/banner

ATTITUDE ADJUSTMENT (Bleeding Edge, r31296)

comment:25 Changed 6 years ago by David Campbell <david@…>

root@OpenWrt:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
rootfs                    1664      1580        84  95% /
/dev/root                 1280      1280         0 100% /rom
tmpfs                    15048       116     14932   1% /tmp
tmpfs                      512         0       512   0% /dev
/dev/mtdblock5            1664      1580        84  95% /overlay
overlayfs:/overlay        1664      1580        84  95% /
/dev/sda1                 1664      1580        84  95% /

root@OpenWrt:~# uname -a
Linux OpenWrt 3.2.14 #1 Tue Apr 17 00:54:55 EST 2012 mips GNU/Linux

root@OpenWrt:~# grep "r[0-9]" /etc/banner
 ATTITUDE ADJUSTMENT (Bleeding Edge, r31296)

comment:26 Changed 4 years ago by jow

  • Milestone changed from Attitude Adjustment 12.09 to Barrier Breaker 14.07

Milestone Attitude Adjustment 12.09 deleted

Add Comment

Modify Ticket

Action
as accepted .
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.