Modify

Opened 3 years ago

Closed 3 years ago

#18248 closed defect (fixed)

Open vSwitch module could not be loaded

Reported by: anonymous Owned by: developers
Priority: normal Milestone:
Component: packages Version: Trunk
Keywords: open vswitch, openvswitch Cc: ardeleanalex@…

Description

I built the Open vSwitch package (including the kernel module). Although I managed to make it run on a device (Netgear WNDR3700v2), I noticed that it does not load the module by default. I manually tried to insert the module, but it failed (Segmentation Fault on the console after "insmod openvswitch").

Here is part of dmesg:

[ 489.290000] openvswitch: Open vSwitch switching datapath 2.1.2, built Oct 28 2014 20:31:18
[ 489.300000] CPU 0 Unable to handle kernel paging request at virtual address 65000100, epc == 82f1a698, ra == 82f1a668
[ 489.310000] Oops#1:
[ 489.310000] CPU: 0 PID: 1470 Comm: insmod Not tainted 3.10.49 #9
[ 489.310000] task: 82e90000 ti: 82eea000 task.ti: 82eea000
[ 489.310000] $ 0 : 00000000 00000000 65000000 ffff7fff
[ 489.310000] $ 4 : 82ee9e00 82f20000 83b4d000 80394b50
[ 489.310000] $ 8 : 00000000 80064910 82ed5400 82cf88c0
[ 489.310000] $12 : 00000014 00000001 00000000 00000001
[ 489.310000] $16 : 82ee9e00 00000006 80323cd8 00000000
[ 489.310000] $20 : 8032a4f4 80323d40 82f1e060 00000004
[ 489.310000] $24 : 00000000 800f9b44
[ 489.310000] $28 : 82eea000 82eebd40 00000124 82f1a668
[ 489.310000] Hi : 00000007
[ 489.310000] Lo : 00000000
[ 489.310000] epc : 82f1a698 ovs_netdev_detach_dev+0x48/0x90 [openvswitch]
[ 489.310000] Not tainted
[ 489.310000] ra : 82f1a668 ovs_netdev_detach_dev+0x18/0x90 [openvswitch]
[ 489.310000] Status: 1000fc03 KERNEL EXL IE
[ 489.310000] Cause : 00800008
[ 489.310000] BadVA : 65000100
[ 489.310000] PrId : 00019374 (MIPS 24Kc)
[ 489.310000] Modules linked in: openvswitch(+) ath9k ath9k_common pppoe ppp_async iptable_nat ath9k_hw ath pppox ppp_generic nf_nat_ipv4 nf_conntrack_ipv6 nf_conntrack_ipv4 mac80211 ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_id xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_CT slhc nf_nat_irc nf_nat_ftp nf_nat nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_irc nf_conntrack_ftp nf_conntrack iptable_raw iptable_mangle iptable_filter ipt_REJECT ip_tables crc_ccitt compat ledtrig_usbdev ip6t_REJECT ip6table_raw ip6table_mangle ip6table_filter ip6_tables x_tables ip_gre ip_tunnel tun libcrc32c gre ipv6 arc4 crypto_blkcipher ohci_hcd ehci_platform ehci_hcd gpio_button_hotplug usbcore nls_base usb_common crc32c crypto_hash
[ 489.310000] Process insmod (pid: 1470, threadinfo=82eea000, task=82e90000, tls=77029440)
[ 489.310000] Stack : 82f1de68 00000004 82f1ddc8 82f1cd90 82f1e060 83b4d000 00000006 82f135a8
[ 489.310000] 00000000 80390000 ffff7fff 80202d14 00000000 82f1e020 83b4d000 80285bac
[ 489.310000] 82f1ddc8 8008f0bc 00000000 82f20000 80390000 ffffffef 00000000 00000000
[ 489.310000] 82f20000 82f20000 82f1cd90 82ee0198 8108217c 82f1d49c 82f1e060 ffffffff
[ 489.310000] 00000000 82ee0068 82eebee0 80390000 00000000 80330000 80330000 00000000
[ 489.310000] ...
[ 489.310000] Call Trace:
[ 489.310000] [<82f1a698>] ovs_netdev_detach_dev+0x48/0x90 [openvswitch]
[ 489.310000] [<82f135a8>] cleanup_module+0xc4/0x12c [openvswitch]
[ 489.310000]
[ 489.310000]
[ 489.310000] Code: 8e020098 3c03ffff 24637fff <8c440100> 00831824 ac430100 0c06dd1e 8e040098 8e110098
[ 489.550000] ---[ end trace e14ff4cb50a82de9 ]---

I should note that I am using Open vSwitch version 2.1.2, but it does not seems to relate to that. I tried to run klogd and syslogd, but could not get any further information out of those logs.

By looking at dmesg, it seems that the problem is with function "ovs_netdev_detach_dev" in datapath/linux/vport-netdev.c. This function is probably called from module clean up.

Attachments (0)

Change History (12)

comment:1 Changed 3 years ago by ardeleanalex@…

So, with the packages move to Github, OpenVSwitch is officially supported as a package in OpenWRT.
The only thing is, it's version 2.3.0.

You can check it out here : https://github.com/openwrt/packages/tree/master/net/openvswitch

I don't have any ideas about your failure.
But, would it be an option for you to try the official one ?

Thanks

comment:2 Changed 3 years ago by cyrus

  • Resolution set to invalid
  • Status changed from new to closed

comment:3 Changed 3 years ago by anonymous

  • Resolution invalid deleted
  • Status changed from closed to reopened

I also tested version 2.3.0, and it gave the same error (it does not seem to be related to the version).
Here is dmesg for version 2.3.0:

[ 59.100000] openvswitch: Open vSwitch switching datapath 2.3.0, built Oct 29 2014 00:23:01
[ 59.110000] CPU 0 Unable to handle kernel paging request at virtual address 00000100, epc == 82edaf08, ra == 82edaed8
[ 59.120000] Oops#1:
[ 59.120000] CPU: 0 PID: 1453 Comm: insmod Not tainted 3.10.49 #11
[ 59.120000] task: 82e9d4f0 ti: 82e98000 task.ti: 82e98000
[ 59.120000] $ 0 : 00000000 00000000 00000000 ffff7fff
[ 59.120000] $ 4 : 82ea3980 82ee0000 83b4d000 80394b50
[ 59.120000] $ 8 : 00000000 80064910 00000010 82ceab00
[ 59.120000] $12 : 00000014 00000001 00000000 00000001
[ 59.120000] $16 : 82ea3980 00000006 80323cd8 00000000
[ 59.120000] $20 : 8032a4f4 80323d40 82ee0000 82edd740
[ 59.120000] $24 : 00000010 800f9b44
[ 59.120000] $28 : 82e98000 82e99d40 00000124 82edaed8
[ 59.120000] Hi : 00000007
[ 59.120000] Lo : 00000000
[ 59.120000] epc : 82edaf08 ovs_netdev_detach_dev+0x48/0x90 [openvswitch]
[ 59.120000] Not tainted
[ 59.120000] ra : 82edaed8 ovs_netdev_detach_dev+0x18/0x90 [openvswitch]
[ 59.120000] Status: 1000fc03 KERNEL EXL IE
[ 59.120000] Cause : 00800008
[ 59.120000] BadVA : 00000100
[ 59.120000] PrId : 00019374 (MIPS 24Kc)
[ 59.120000] Modules linked in: openvswitch(+) ath9k ath9k_common pppoe ppp_async iptable_nat ath9k_hw ath pppox ppp_generic nf_nat_ipv4 nf_conntrack_ipv6 nf_conntrack_ipv4 mac80211 ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_id xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_CT slhc nf_nat_irc nf_nat_ftp nf_nat nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_irc nf_conntrack_ftp nf_conntrack iptable_raw iptable_mangle iptable_filter ipt_REJECT ip_tables crc_ccitt compat ledtrig_usbdev ip6t_REJECT ip6table_raw ip6table_mangle ip6table_filter ip6_tables x_tables ip_gre ip_tunnel tun libcrc32c gre ipv6 arc4 crypto_blkcipher ohci_hcd ehci_platform ehci_hcd gpio_button_hotplug usbcore nls_base usb_common crc32c crypto_hash
[ 59.120000] Process insmod (pid: 1453, threadinfo=82e98000, task=82e9d4f0, tls=77b8b440)
[ 59.120000] Stack : 00000124 8008f06c 82edec30 80323d10 8032a4ec 83b4d000 00000006 82ed3be8
[ 59.120000] 82edea54 00000004 82ede8c8 8008f0bc 80330000 82edec30 83b4d000 80285bac
[ 59.120000] 82edea54 00000000 82edd740 00000004 82edec74 ffffffef 00000000 00000000
[ 59.120000] 00000004 82edec74 82ee0000 82ebc10c 8108217c 82edde2c 82edec74 ffffffff
[ 59.120000] 8105d500 82ebc068 82e99ee0 80390000 00000000 80330000 80330000 00000000
[ 59.120000] ...
[ 59.120000] Call Trace:
[ 59.120000] [<82edaf08>] ovs_netdev_detach_dev+0x48/0x90 [openvswitch]
[ 59.120000] [<82ed3be8>] cleanup_module+0xb8/0x120 [openvswitch]
[ 59.120000]
[ 59.120000]
[ 59.120000] Code: 8e020098 3c03ffff 24637fff <8c440100> 00831824 ac430100 0c06dd1e 8e040098 8e110098
[ 59.360000] ---[ end trace c43e9c8c4aba188c ]---

comment:4 Changed 3 years ago by ardeleanalex@…

Seems you're using kernel version 3.10.49
OpenWRT's 3.10 version is at 3.10.58

I'll try to reproduce this with 3.10.58
I'm assuming you're also using the OpenWRT trunk to test ?

Are you using any custom kernel patches ?

Thanks
Alex

comment:5 Changed 3 years ago by anonymous

No, I am not using any specific kernel patch or use all original kernel parameters.
I am also using trunk.

comment:6 Changed 3 years ago by ardeleanalex@…

For me it works with kernel 3.10.58
I have it running on a MIPS target, but the build there is not too close to trunk.
So, it's not 100% accurate.

I tried trunk on VM x86_64.
I had to apply this patch: https://github.com/openwrt/packages/pull/486
Also have to wait for this pull request to apply: https://github.com/openwrt/packages/pull/485

In the meantime I found that on kernel 3.14 it fails to load too.
Some other error.

I'll come back with a request for you to test after the 2 pull requests on Github get merged.

Thanks
Alex

comment:7 Changed 3 years ago by ardeleanalex@…

For me it works with kernel 3.10.58
I have it running on a MIPS target, but the build there is not too close to trunk.
So, it's not 100% accurate.

I tried trunk on VM x86_64.
I downgraded the kernel (default is 3.14 on x86_64) to 3.10.58.
I had to apply this patch: https://github.com/openwrt/packages/pull/486
Also have to wait for this pull request to apply: https://github.com/openwrt/packages/pull/485

In the meantime I found that on kernel 3.14 it fails to load too.
Some other error.

I'll come back with a request for you to test after the 2 pull requests on Github get merged.

Thanks
Alex

comment:8 Changed 3 years ago by ardeleanalex@…

Both pull requests got merged.

Mind to check again with the latest OpenWRT trunk ?
You should have kernel version 3.10.58.

Thanks
Alex

comment:9 Changed 3 years ago by anonymous

Thanks. I cleaned my build directory, update to trunk and rebuild. Now the module is now loading correctly.

However, it seems that the kernel module does not provide good performance throughput. I connect to machine to the switch and setup vlans so that traffic among them should be passed via openvswitch running on the openwrt. I used both simple hub and l2 learning switch implementation of PoX, and the throughput (tested using scp a file from one machine to other one) was not good. It was around 330 KB/s, which is close to the number for the case that module was not loading and the userspace module was handling the traffic (although a bit higher).

Does anyone know how I can measure to see whether module actually perform well (similar to how it could handle even gigabit speeds on PC kernels)?

comment:10 Changed 3 years ago by ardeleanalex@…

Hmmm,

I think this question would be suited also for the OpenVSwitch mailing list.
What are the HW specs you're running OVS on ?

Maybe we would also need to check/add HW offloading if supported.
OVS is usually targeted at powerful HW.

comment:11 Changed 3 years ago by anonymous

I tested the switch with another machine and it seems the problem was due to the first machine. Now I am getting around 90 Mbps which is good.

Thanks again for the patch. I think we can now close this ticket.

comment:12 Changed 3 years ago by florian

  • Resolution set to fixed
  • Status changed from reopened to closed

Add Comment

Modify Ticket

Action
as closed .
The resolution will be deleted. Next status will be 'reopened'.
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.