Modify

Opened 5 years ago

Closed 5 years ago

#13729 closed defect (fixed)

6relayd keeps deleting the connected route

Reported by: berni@… Owned by: cyrus
Priority: high Milestone: Chaos Calmer 15.05
Component: base system Version: Trunk
Keywords: Cc: cyrus@…

Description

6relayd keeps removing (and adding? Might be the kernel that readds it) the connected routes from the IPv6 routing table in irregular intervals. The IP address is still on the interface, but the connected route is gone. With one ISP/prefix

root@router:~# ip monitor | while read a; do printf "%s " `date`; echo $a; done | grep 2001:4c50:12f:ef01::/64
Thu Jun 20 22:45:37 CEST 2013 Deleted 2001:4c50:12f:ef01::/64 dev br-lan proto kernel metric 256 expires 28521sec
Thu Jun 20 22:45:48 CEST 2013 2001:4c50:12f:ef01::/64 dev br-lan proto kernel metric 256 expires 28510sec
Thu Jun 20 22:45:59 CEST 2013 Deleted 2001:4c50:12f:ef01::/64 dev br-lan proto kernel metric 256 expires 28799sec
Thu Jun 20 22:46:00 CEST 2013 2001:4c50:12f:ef01::/64 dev br-lan proto kernel metric 256 expires 28799sec
Thu Jun 20 22:46:01 CEST 2013 Deleted 2001:4c50:12f:ef01::/64 dev br-lan proto kernel metric 256 expires 28798sec
Thu Jun 20 22:46:03 CEST 2013 2001:4c50:12f:ef01::/64 dev br-lan proto kernel metric 256 expires 28796sec
Thu Jun 20 22:46:17 CEST 2013 Deleted 2001:4c50:12f:ef01::/64 dev br-lan proto kernel metric 256 expires 28781sec
Thu Jun 20 22:46:24 CEST 2013 2001:4c50:12f:ef01::/64 dev br-lan proto kernel metric 256 expires 28774sec
Thu Jun 20 22:46:49 CEST 2013 Deleted 2001:4c50:12f:ef01::/64 dev br-lan proto kernel metric 256 expires 28750sec
Thu Jun 20 22:46:50 CEST 2013 2001:4c50:12f:ef01::/64 dev br-lan proto kernel metric 256 expires 28748sec
Thu Jun 20 22:46:51 CEST 2013 Deleted 2001:4c50:12f:ef01::/64 dev br-lan proto kernel metric 256 expires 28747sec
Thu Jun 20 22:46:53 CEST 2013 2001:4c50:12f:ef01::/64 dev br-lan proto kernel metric 256 expires 28745sec
Thu Jun 20 22:47:07 CEST 2013 Deleted 2001:4c50:12f:ef01::/64 dev br-lan proto kernel metric 256 expires 28732sec
Thu Jun 20 22:47:09 CEST 2013 2001:4c50:12f:ef01::/64 dev br-lan proto kernel metric 256 expires 28730sec

Full ip monitor cycle

Thu Jun 20 22:55:52 CEST 2013 8: eth0.100@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
Thu Jun 20 22:55:52 CEST 2013 link/ether 4c:e6:76:c4:56:7e
Thu Jun 20 22:55:52 CEST 2013 8: eth0.100@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
Thu Jun 20 22:55:52 CEST 2013 link/ether 4c:e6:76:c4:56:7e
Thu Jun 20 22:55:52 CEST 2013 1: lo inet6 2001:4c50:12f:ef00::1/64 scope global dynamic
Thu Jun 20 22:55:52 CEST 2013 valid_lft 28507sec preferred_lft 28507sec
Thu Jun 20 22:55:52 CEST 2013 6: br-lan inet6 2001:4c50:12f:ef01::1/64 scope global dynamic
Thu Jun 20 22:55:52 CEST 2013 valid_lft 28507sec preferred_lft 28507sec
Thu Jun 20 22:55:52 CEST 2013 12: br-wlan inet6 2001:4c50:12f:ef02::1/64 scope global dynamic
Thu Jun 20 22:55:52 CEST 2013 valid_lft 28507sec preferred_lft 28507sec
Thu Jun 20 22:55:57 CEST 2013 172.16.3.35 dev br-lan lladdr 00:20:4a:c5:5b:15 STALE
Thu Jun 20 22:56:00 CEST 2013 delete ff02::1:ff47:3c5 dev br-lan lladdr 33:33:ff:47:03:c5 NOARP
Thu Jun 20 22:56:00 CEST 2013 8: eth0.100@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
Thu Jun 20 22:56:00 CEST 2013 link/ether 4c:e6:76:c4:56:7e
Thu Jun 20 22:56:00 CEST 2013 8: eth0.100@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
Thu Jun 20 22:56:00 CEST 2013 link/ether 4c:e6:76:c4:56:7e
Thu Jun 20 22:56:00 CEST 2013 1: lo inet6 2001:4c50:12f:ef00::1/64 scope global dynamic
Thu Jun 20 22:56:00 CEST 2013 valid_lft 28499sec preferred_lft 28499sec
Thu Jun 20 22:56:00 CEST 2013 6: br-lan inet6 2001:4c50:12f:ef01::1/64 scope global dynamic
Thu Jun 20 22:56:00 CEST 2013 valid_lft 28499sec preferred_lft 28499sec
Thu Jun 20 22:56:00 CEST 2013 12: br-wlan inet6 2001:4c50:12f:ef02::1/64 scope global dynamic
Thu Jun 20 22:56:00 CEST 2013 valid_lft 28499sec preferred_lft 28499sec
Thu Jun 20 22:56:00 CEST 2013 Deleted 80000: from 2001:4c50:100:2:a0ad:8cb6:8759:6efb lookup 1008
Thu Jun 20 22:56:00 CEST 2013 Deleted 90000: from 2001:4c50:100:2:a0ad:8cb6:8759:6efb lookup 1008
Thu Jun 20 22:56:00 CEST 2013 Deleted 8: eth0.100 inet6 2001:4c50:100:2:a0ad:8cb6:8759:6efb/128 scope global dynamic
Thu Jun 20 22:56:00 CEST 2013 valid_lft 28499sec preferred_lft 28499sec
Thu Jun 20 22:56:00 CEST 2013 Deleted local 2001:4c50:100:2:a0ad:8cb6:8759:6efb via :: dev lo table local proto none metric 0
Thu Jun 20 22:56:00 CEST 2013 2001:4c50:100:2:a0ad:8cb6:8759:6efb dev eth0.100 proto kernel metric 256 expires 28800sec
Thu Jun 20 22:56:00 CEST 2013 80000: from 2001:4c50:100:2:a0ad:8cb6:8759:6efb lookup 1008
Thu Jun 20 22:56:00 CEST 2013 90000: from 2001:4c50:100:2:a0ad:8cb6:8759:6efb lookup 1008
Thu Jun 20 22:56:00 CEST 2013 Deleted 2001:4c50:100:2:a0ad:8cb6:8759:6efb dev eth0.100 proto kernel metric 256 expires 28800sec
Thu Jun 20 22:56:00 CEST 2013 1: lo inet6 2001:4c50:12f:ef00::1/64 scope global dynamic
Thu Jun 20 22:56:00 CEST 2013 valid_lft 28800sec preferred_lft 28800sec
Thu Jun 20 22:56:00 CEST 2013 6: br-lan inet6 2001:4c50:12f:ef01::1/64 scope global dynamic
Thu Jun 20 22:56:00 CEST 2013 valid_lft 28800sec preferred_lft 28800sec
Thu Jun 20 22:56:00 CEST 2013 12: br-wlan inet6 2001:4c50:12f:ef02::1/64 scope global dynamic
Thu Jun 20 22:56:00 CEST 2013 valid_lft 28800sec preferred_lft 28800sec
Thu Jun 20 22:56:01 CEST 2013 Deleted 2001:4c50:12f:ef01::/64 dev br-lan proto kernel metric 256 expires 28799sec
Thu Jun 20 22:56:01 CEST 2013 8: eth0.100 inet6 2001:4c50:100:2:a0ad:8cb6:8759:6efb/128 scope global dynamic
Thu Jun 20 22:56:01 CEST 2013 valid_lft 28800sec preferred_lft 28800sec
Thu Jun 20 22:56:01 CEST 2013 local 2001:4c50:100:2:a0ad:8cb6:8759:6efb via :: dev lo table local proto none metric 0
Thu Jun 20 22:56:04 CEST 2013 8: eth0.100@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
Thu Jun 20 22:56:04 CEST 2013 link/ether 4c:e6:76:c4:56:7e
Thu Jun 20 22:56:04 CEST 2013 8: eth0.100@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
Thu Jun 20 22:56:04 CEST 2013 link/ether 4c:e6:76:c4:56:7e
Thu Jun 20 22:56:04 CEST 2013 1: lo inet6 2001:4c50:12f:ef00::1/64 scope global dynamic
Thu Jun 20 22:56:04 CEST 2013 valid_lft 28796sec preferred_lft 28796sec
Thu Jun 20 22:56:04 CEST 2013 6: br-lan inet6 2001:4c50:12f:ef01::1/64 scope global dynamic
Thu Jun 20 22:56:04 CEST 2013 valid_lft 28796sec preferred_lft 28796sec
Thu Jun 20 22:56:04 CEST 2013 2001:4c50:12f:ef01::/64 dev br-lan proto kernel metric 256 expires 28796sec

It happens a LOT more often when I run my usual setup with two ISPs giving out three prefixes, to the point where it is almost impossible to use.

I did not find anything unusal on the uplink interface when it happens.

It looks like the READDING of the route is done by the kernel itself, the moment it happens is related to the moment 6relayd sends out its unsolicited RA on br-lan, so the router seems to learn from it's own advertisement.

This behaviour stops immediately when I stop 6relayd.

Attachments (0)

Change History (13)

comment:1 Changed 5 years ago by anonymous

Additional info:

  • Downgrading to 6relayd_2013-06-11-8f107ae49698740f374d8adafa90ff5da459d1ec_ar71xx.ipk seems to help. I am reasonably sure I was able to reproduce it with this version, but now I can't. I wanted to send you the -vvv output of 6relayd, so I started it in the foreground, waited for 15 minutes and it did not remove the route once. Upgrading to 2013-06-19 makes it reappear within a minute. So it might just be this last update.
  • There is nothing in syslog/logread/dmesg when it happens

comment:2 Changed 5 years ago by cyrus

Please post your /etc/config/network, your /etc/config/6relayd, your /etc/sysctl.conf and the command-line parameters used for 6relayd when it happens (ps | grep 6relayd).

There must be something wrong with your configuration.
In general the kernel shouldn't react on any Router Advertisements. 6relayd doesn't really add or remove this kind of routes, it just reacts on changes to local interface addresses and then sends an unsolicited RA (actually it does something similar to ip monitor internally).

But I cannot really tell you anything else without knowing your configuration and environment.

comment:3 Changed 5 years ago by cyrus

  • Cc cyrus@… added
  • Owner changed from developers to cyrus
  • Status changed from new to assigned

comment:4 Changed 5 years ago by martijn@…

I see the same behaviour on r37012 (branches/attitude_adjustment). My config files look like this:

/etc/config/network

config interface 'loopback'
	option ifname 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config interface 'lan'
	option ifname 'eth0.1'
	option type 'bridge'
	option proto 'static'
	option ipaddr '192.168.3.1'
	option netmask '255.255.255.0'
	option ip6assign '64'
	option ip6hint '1'

config interface 'wan'
	option ifname 'eth0.6'
	option proto 'pppoe'
	option ipv6 1
	option 'username' 'foobar'
	option 'password' 'foobar'

config interface 'wan6'
	option ifname @wan
	option proto dhcpv6

config switch
	option name 'eth0'
	option reset '1'
	option enable_vlan '1'

config switch_vlan
	option device 'eth0'
	option vlan '1'
	option ports '0t 2 3 4 5'

config switch_vlan
	option device 'eth0'
	option vlan '6'
	option ports '0t 1t'

config switch_vlan               
        option device 'eth0'     
        option vlan '4'
        option ports '0t 1t'

/etc/config/6relayd

config server default
	option master	wan6
	list network	lan
	option rd	server
	option dhcpv6	server
	option fallback_relay	'rd dhcpv6 ndp'
	option management_level 1
	option compat_ula 1


# Example #1: Create a relay for several networks in proxy mode
# This can be used to proxy Router Discovery, DHCPv6 and NDP.

#config server examplerelay
#	option master	'wan'
#	option network	'lan'
#	option rd	'relay'
#	option dhcpv6	'relay'
#	option ndp	'relay'


# Example #2: Provide Router Discovery and stateless DHCPv6 in server mode
# This can be used as a small radvd and stateless DHCPv6-server replacement.

#config server exampleserver
#	option network	'lan'
#	option rd	'server'
#	option dhcpv6	'server'

(commenting fallback_relay doesn't change the behaviour)

/etc/sysctl.conf:

kernel.panic=3
net.ipv4.conf.default.arp_ignore=1
net.ipv4.conf.all.arp_ignore=1
net.ipv4.ip_forward=1
net.ipv4.icmp_echo_ignore_broadcasts=1
net.ipv4.icmp_ignore_bogus_error_responses=1
net.ipv4.tcp_ecn=0
net.ipv4.tcp_fin_timeout=30
net.ipv4.tcp_keepalive_time=120
net.ipv4.tcp_syncookies=1
net.ipv4.tcp_timestamps=1
net.ipv4.tcp_sack=1
net.ipv4.tcp_dsack=1

net.ipv6.conf.all.forwarding=1

net.netfilter.nf_conntrack_acct=1
net.netfilter.nf_conntrack_checksum=0
net.netfilter.nf_conntrack_max=16384
net.netfilter.nf_conntrack_tcp_timeout_established=3600
net.netfilter.nf_conntrack_udp_timeout=60
net.netfilter.nf_conntrack_udp_timeout_stream=180

# disable bridge firewalling by default
net.bridge.bridge-nf-call-arptables=0
net.bridge.bridge-nf-call-ip6tables=0
net.bridge.bridge-nf-call-iptables=0

comment:5 Changed 5 years ago by cyrus

In your /etc/sysctl.conf please add a line

net.ipv6.conf.default.forwarding=1

above

net.ipv6.conf.all.forwarding=1

and see if that helps.

comment:6 Changed 5 years ago by martijn@…

Added that line, restarted the device (TP-Link WDR4300), no improvement:

Thu Jan 1 01:00:54 CET 1970 2001:980:ef09:1::/64 dev br-lan proto kernel metric 256 expires 7200sec
Thu Jan 1 01:00:56 CET 1970 Deleted 2001:980:ef09:1::/64 dev br-lan proto kernel metric 256 expires 7197sec
Thu Jan 1 01:01:07 CET 1970 2001:980:ef09:1::/64 dev br-lan proto kernel metric 256 expires 7186sec

comment:7 Changed 5 years ago by martijn@…

Just after the route is added, tcpdump shows a router advertisement is sent out:

15:43:08.219519 IP6 fe80::6670:2ff:fe3e:a022 > ff02::1: ICMP6, router advertisement, length 120

But then, randomly, the route gets deleted again:

Sun Jun 23 15:30:33 CEST 2013 Deleted 2001:980:ef09:1::/64 dev br-lan proto kernel metric 256 expires 6730sec
Sun Jun 23 15:33:10 CEST 2013 2001:980:ef09:1::/64 dev br-lan proto kernel metric 256 expires 6574sec
Sun Jun 23 15:33:10 CEST 2013 Deleted 2001:980:ef09:1::/64 dev br-lan proto kernel metric 256 expires 6573sec
Sun Jun 23 15:43:07 CEST 2013 2001:980:ef09:1::/64 dev br-lan proto kernel metric 256 expires 5977sec
Sun Jun 23 15:43:38 CEST 2013 Deleted 2001:980:ef09:1::/64 dev br-lan proto kernel metric 256 expires 5945sec

comment:8 Changed 5 years ago by anonymous

That is very strange and I am still unsure what causes this. However I'm certain that it cannot be 6relayd deleting the routes because there is only 2 occasions when 6relayd sets / deletes routes: when NDP-proxying is enabled and even than the routes have a prefix of /128 and in case DHCPv6-PD service is provided but for that to happen the prefix assigned to an interface would have to be greater than /64 and even then it only sets up and deletes routes for parts of the prefix.

Also that total randomness of the removal bothers me. And from your log it shows that only the on-link route but not the corresponsing address seems to be removed.

The immediate RA being sent is due to the address being added to the interface - 6relayd receiving that event and triggering an RA to update the downstream clients.

comment:9 Changed 5 years ago by martijn@…

How can I (start to) figure out who or what *is* removing the route?

comment:10 Changed 5 years ago by berni@…

/etc/config/network

config interface 'loopback'
	option ifname 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'
	option ip6assign '64'
	option ip6hint '0'

config globals 'globals'

config interface 'lan'
	option type 'bridge'
	option proto 'static'
	option ip6assign '64'
	option ip6hint '1'
	option _orig_ifname 'eth0 radio0.network1 radio1.network1'
	option _orig_bridge 'true'
	option ipaddr '172.16.3.1'
	option netmask '255.255.255.192'
	option ifname 'eth0.1'

config switch
	option name 'switch0'
	option reset '1'
	option enable_vlan '1'

config switch_vlan
	option device 'switch0'
	option vlan '1'
	option vid '1'
	option ports '0 2 3 4'

config interface 'mnet'
	option proto 'pppoe'
	option ipv6 '1'
	option username 'xxxx'
	option password 'xxxx'
	option ifname 'eth1'

config interface 'mnet6'
	option proto 'dhcpv6'
	option ifname '@mnet'
	option reqaddress 'try'
	option reqprefix 'auto'
	option auto '0'

config interface 'wlan'
	option _orig_ifname 'radio0.network1'
	option _orig_bridge 'false'
	option proto 'static'
	option ipaddr '172.16.3.65'
	option netmask '255.255.255.192'
	option ip6assign '64'
	option type 'bridge'
	option ip6hint '2'

config switch_vlan
	option device 'switch0'
	option vlan '2'
	option vid '2'
	option ports '0t 3t'

config interface 'voip'
	option proto 'static'
	option ifname 'eth0.2'
	option ipaddr '172.16.3.129'
	option netmask '255.255.255.248'
	option ip6addr '2001:a60:a000:f::1/64'

config switch_vlan
	option device 'switch0'
	option vlan '3'
	option ports '0t 1'
	option vid '100'

config interface 'cablesurf'
	option proto 'none'
	option ifname 'eth0.100'


config interface 'cablesurf6'
	option proto 'dhcpv6'
	option ifname '@cablesurf'
	option reqaddress 'try'
	option reqprefix 'auto'

/etc/config/6relayd

config server 'default'
	option rd 'server'
	option dhcpv6 'server'
	option compat_ula '1'
	list network 'lan'
	list network 'wlan'
	option fallback_relay 'rd dhcpv6'

/etc/sysctl.conf (not touched by me):

net.ipv4.conf.default.arp_ignore=1
net.ipv4.conf.all.arp_ignore=1
net.ipv4.ip_forward=1
net.ipv4.icmp_echo_ignore_broadcasts=1
net.ipv4.icmp_ignore_bogus_error_responses=1
net.ipv4.tcp_ecn=0
net.ipv4.tcp_fin_timeout=30
net.ipv4.tcp_keepalive_time=120
net.ipv4.tcp_syncookies=1
net.ipv4.tcp_timestamps=1
net.ipv4.tcp_sack=1
net.ipv4.tcp_dsack=1

net.ipv6.conf.default.forwarding=2
net.ipv6.conf.all.forwarding=2

net.netfilter.nf_conntrack_acct=1
net.netfilter.nf_conntrack_checksum=0
net.netfilter.nf_conntrack_max=16384
net.netfilter.nf_conntrack_tcp_timeout_established=3600
net.netfilter.nf_conntrack_udp_timeout=60
net.netfilter.nf_conntrack_udp_timeout_stream=180

# disable bridge firewalling by default
net.bridge.bridge-nf-call-arptables=0
net.bridge.bridge-nf-call-ip6tables=0
net.bridge.bridge-nf-call-iptables=0

6relayd arguments:

/usr/sbin/6relayd -l/tmp/hosts/6relayd,/usr/sbin/6relayd-update -c -Rserver -Dserver . br-lan br-wlan

comment:11 Changed 5 years ago by cyrus

@berni: Hmm unfortunately I still haven't got any clue. I just recommend setting the net.ipv6.conf.default.forwarding=2
net.ipv6.conf.all.forwarding=2
parameters to =1 if you think that RAs have something to do with it.

comment:12 Changed 5 years ago by anonymous

test 1, please ignore

comment:13 Changed 5 years ago by cyrus

  • Resolution set to fixed
  • Status changed from assigned to closed

apparently berni was right, 6relayd was indeed the problem but it only happened under certain conditions which made it hard to reproduce.

See r37018 / r37019 for fixes.
Please reopen if the problem still occurs to you.

Add Comment

Modify Ticket

Action
as closed .
The resolution will be deleted. Next status will be 'reopened'.
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.