Modify

Opened 3 years ago

Last modified 19 months ago

#19586 new defect

procd / zram-ramdisc / ext4 / kernel crash / r45588

Reported by: bittorf@… Owned by: developers
Priority: normal Milestone:
Component: packages Version: Trunk
Keywords: Cc:

Description

when compiling r45588 with,

CONFIG_BUSYBOX_CONFIG_SWAPONOFF=y
CONFIG_BUSYBOX_CONFIG_FEATURE_SWAPON_PRI=y
CONFIG_PACKAGE_zram-swap=y
CONFIG_PROCD_ZRAM_TMPFS=y
CONFIG_PACKAGE_kmod-fs-ext4=y
CONFIG_PACKAGE_e2fsprogs=y

i see these crashes on ar71xx

<6>[   24.400000] br-lan: port 1(eth0.1) entered forwarding state
<4>[   25.840000] Kernel bug detected[#1]:
<4>[   25.840000] CPU: 0 PID: 1372 Comm: sed Not tainted 3.18.11 #9
<4>[   25.840000] task: 86ab8000 ti: 86af8000 task.ti: 86af8000
<4>[   25.840000] $ 0   : 00000000 00000001 00000001 87a03c00
<4>[   25.840000] $ 4   : 87a03c00 00000001 8754f794 879b3000
<4>[   25.840000] $ 8   : 00080000 00000000 00000001 00000001
<4>[   25.840000] $12   : 00000009 3b000000 00000000 fffffffe
<4>[   25.840000] $16   : 8745a068 86af9920 86af9a68 879b3000
<4>[   25.840000] $20   : 87a03c00 87a03c00 00000001 86af9a38
<4>[   25.840000] $24   : 00000000 87b0bd50                  
<4>[   25.840000] $28   : 86af8000 86af98d0 00000000 87b32520
<4>[   25.840000] Hi    : 00000000
<4>[   25.840000] Lo    : 00000001
<4>[   25.840000] epc   : 87b31ae8 ext4_set_bits+0x289c/0x3964 [ext4]
<4>[   25.840000]     Not tainted
<4>[   25.840000] ra    : 87b32520 ext4_set_bits+0x32d4/0x3964 [ext4]
<4>[   25.840000] Status: 1100dc03    KERNEL EXL IE 
<4>[   25.840000] Cause : 00800034
<4>[   25.840000] PrId  : 0001974c (MIPS 74Kc)
<4>[   25.840000] Modules linked in: ath9k ath9k_common pppoe ppp_async iptable_nat ath9k_hw ath pppox ppp_generic nf_nat_ipv4 nf_conntrack_ipv6 nf_conntrack_
ipv4 mac80211 ipt_REJECT ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_id
 xt_hl xt_ecn xt_dscp xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_NETMAP xt_LOG xt_HL xt_DSCP xt_CT xt_CLASSIFY slhc nf_reject_ipv4 nf_nat_masquerade_ipv
4 nf_nat_ftp nf_nat nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack_ftp nf_conntrack macvlan iptable_raw iptable_mangle iptable_fi
lter ipt_ECN ip_tables crc_ccitt compat sch_teql sch_tbf sch_sfq sch_red sch_prio sch_htb sch_gred sch_dsmark sch_codel em_text em_nbyte em_meta em_cmp cls_ba
sic act_police act_ipt act_skbedit act_mirred em_u32 cls_u32 cls_tcindex cls_flow cls_route cls_fw sch_hfsc sch_ingress ledtrig_usbdev ip6t_REJECT nf_reject_i
pv6 nf_log_ipv6 nf_log_common ip6table_raw ip6table_mangle ip6table_filter ip6_tables x_tables ifb tun ipv6 arc4 crypto_blkcipher ehci_platform ehci_hcd gpio_
button_hotplug usbcore nls_base usb_common ext4 mbcache jbd2 crypto_hash crc16 zram zsmalloc lzo_decompress lzo_compress lz4_decompress lz4_compress
<4>[   25.840000] Process sed (pid: 1372, threadinfo=86af8000, task=86ab8000, tls=7732d440)
<4>[   25.840000] Stack : 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<4>[   25.840000]       00000000 00000000 00000000 8745a068 86af99d0 86af9a68 879b3000 87b32520
<4>[   25.840000]       8745a068 86af99d0 879b3000 87b2e710 00000000 00000000 00000000 00000001
<4>[   25.840000]       86af993c 86af9938 00000000 00000001 8745a068 86af99d0 86af9a68 879b3000
<4>[   25.840000]       00000000 87a03c00 87b50000 86af9a38 00000000 87b33ae8 00000000 00000000
<4>[   25.840000]       ...
<4>[   25.840000] Call Trace:
<4>[   25.840000] [<87b31ae8>] ext4_set_bits+0x289c/0x3964 [ext4]
<4>[   25.840000] [<87b32520>] ext4_set_bits+0x32d4/0x3964 [ext4]
<4>[   25.840000] 
<4>[   25.840000] 
<4>[   25.840000] Code: 8e620020  00a2102b  38420001 <00020336> 8e630080  8e620140  00009021  00651806  00031880 
<4>[   26.120000] ---[ end trace c1653cd6a9b4fe7d ]---

Attachments (0)

Change History (10)

comment:1 Changed 3 years ago by bittorf@…

culprit: the script /etc/init.d/zram is too naive and uses (already used) zram0.
working on a fix...

comment:2 Changed 3 years ago by bittorf@…

difficult: i see no way to add more zram-discs instead of
'insmod zram num_devices=4', but this needs unloading it first
which destroys the compressed tmpfs... 8-( seems zram must
be handled by procd too...

for now we should let fail /etc/init.d/zram like this:

zram_applicable()
{
  ...

  grep -sq ^"$zram_dev " /proc/mounts && {
    logger -s -t zram_applicable -p daemon.notice "[OK] '$zram_dev' already mounted"
    return 1
}

so no other user will run into that prob...

comment:3 Changed 3 years ago by bittorf@…

at least one thing comes to my mind: can you initialize the zram-module with cpu_cores + 1 (for tmpfs-zram). so the zram init-script can just use the discs...

comment:4 Changed 3 years ago by bittorf@…

ping: CC: John Crispin <blogic@…>

comment:5 Changed 2 years ago by openwrt@…

I just hit this exact issue on a CC 15.05 build (linux 3.18.21). Once I rebuilt with the lines above in the zram.init, the system booted and is running fine. Removing the swapon/swapoff binaries also fixes the issue, which is obvious once you look at zram_applicable() method in zram.init.

Is there any movement on this? Is there a better solution than returning 1 from zram_applicable()? I have not tried a build without ext4, but the first kernel crash always seems to mention ext4_set_bits.

comment:6 Changed 2 years ago by bittorf@…

i will send a patch for this, please give me 2 weeks.

comment:7 Changed 2 years ago by Nilfred

So, you want zram0 for /tmp and for swapon.
Better than make cpu_cores + 1 zrams: Why not 2 partitions to the current zram0 block?
BTW: No swap is working in MIPS. No one.
Does the procd dd the control structures of ext4 instead of mkfs.ext4?

comment:8 Changed 2 years ago by bittorf@…

@Nilfred: interesting idea to use 1 block for 2 partitions, i will try/check that. i know that there are issues with MIPS, but we *use* zram-swap on a lot of routers and it *is* working - problem only arise (it seems) if the is a lot of pressure...see #21705

here a mips/ar71xx router with r48717 / kernel 4.1.16:

root@box:~ free; dd if=/dev/urandom of=/tmp/test.bin bs=4k count=2000; free
             total       used       free     shared    buffers     cached
Mem:         28208      24292       3916       1076       1684       7312
-/+ buffers/cache:      15296      12912
Swap:        13308          0      13308

2000+0 records in
2000+0 records out

             total       used       free     shared    buffers     cached
Mem:         28208      27220        988       9052        556      10468
-/+ buffers/cache:      16196      12012
Swap:        13308        140      13168

comment:9 Changed 20 months ago by hojuruku

Hi,

I had a fail to boot by enabling
CONFIG_PROCD_ZRAM_TMPFS=y
with:
ERROR("Can't exec /sbin/mkfs.ext4\n");
from:
https://git.openwrt.org/?p=project/procd.git;a=blob_plain;f=initd/zram.c;hb=HEAD

The problem is the makemenuconfig defines didn't remind me that the ext4 module& userspace tools/zram module needs to be be built into the image.

An earlier proposed patch used mkfs.ext2 which is supported in busybox that never made it to git. Is there any reason we need to use ext4 here? Ext2 has less overhead and I don't see ext2 module in the makemenuconfig, though I enabled it in make kernel_menuconfig.
https://lists.openwrt.org/pipermail/openwrt-devel/2014-December/029587.html

Seeing this feature is aimed at "memory constrained systems", and therefore likely rom constrained systems (4mb flash), can we tweak this whole patch to use ext2/busybox instead. Put an ext2 module into menuconfig and make the menuconfig Config.in files force enable the required busybox options. That's much better. Procd these days is in every standard rom image, so it's not something people can enable in a standard build, so there is no harm done making this change.

I'm going disable this option and or patch procd to do this. Just my two cents. Any thoughts?

I think ndb at openwrt.org & tjwasiak at gmail.com & blogic (John Chrispin) should be notified of this suggestion.

Busybox mount options:
https://dev.openwrt.org/browser/trunk/package/utils/busybox/config/util-linux/Config.in
Needs:
BUSYBOX_CONFIG_MOUNT
BUSYBOX_CONFIG_FEATURE_MOUNT_VERBOSE
BUSYBOX_CONFIG_FEATURE_MOUNT_FLAGS
BUSYBOX_CONFIG_UMOUNT ??

Also the zram-swap script could alternatively use:
https://dev.openwrt.org/browser/trunk/package/system/zram-swap/files/zram.init

BUSYBOX_CONFIG_MKSWAP
BUSYBOX_CONFIG_SWAPONOFF
BUSYBOX_CONFIG_FEATURE_SWAPON_DISCARD ??

And then of course there is an alternative to zram on /tmp - zswap or does it require a standard swap device as well?
https://wiki.archlinux.org/index.php/Zswap

comment:10 Changed 19 months ago by hojuruku

For the love of god. You can hot add zram devices. read zram.txt that comes with the kernel.
Don't bother patching the kernel to enable two by default.

Add Comment

Modify Ticket

Action
as new .
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.