Modify

Opened 4 years ago

Closed 4 years ago

Last modified 3 years ago

#17396 closed defect (fixed)

LAN clients unable to communicate via IPv6 while Internet connectivity works

Reported by: mhoran Owned by: developers
Priority: normal Milestone:
Component: base system Version: Barrier Breaker 14.07
Keywords: Cc: cyrus@…

Description

Since upgrading to Barrier Breaker rc2 I noticed that my IPv6 clients are unable to communicate with each other. However, they can get out to the Internet just fine, and external IPv6 hosts can communicate with them.

If I ping6 a host on the wired network from a wireless client, or vice-versa, I'm able to connect (e.g. ssh). However, the neighbor will inevitably become stale, and I cannot connect again with out a ping6.

I do not believe I saw this issue when previously running trunk or even in rc1. I am not aware of any configuration changes and did not install any additional packages. I do not believe this to be an OS level issue as I'm seeing it cross device (laptop, desktop, Android phone), though I could have upgraded something that caused this issue to manifest itself.

I do see neighbor solicitations and advertisements in wireshark. However, ip -6 n shows STALE for the host I'm trying to connect to, until I ping6 it. Just trying to ssh to the host will result in a fallback to IPv4, or if I force IPv6, a timeout.

Attachments (2)

network (943 bytes) - added by mhoran 4 years ago.
/etc/config/network
dhcp (865 bytes) - added by mhoran 4 years ago.
/etc/config/dhcp

Download all attachments as: .zip

Change History (12)

Changed 4 years ago by mhoran

/etc/config/network

Changed 4 years ago by mhoran

/etc/config/dhcp

comment:1 follow-up: Changed 4 years ago by cyrus

OK, that sounds a bit strange. First of all, there is no difference as to how ping6 and ssh do the neighbor lookup (in fact the kernel handles it for both).
Similarly a STALE entry is per-se not problematic since it will become active again automatically or at least should.

You mentioned fallback to IPv4 so I assume you are using some kind of DNS inside your home for the ssh. Since you have disabled ra_management IPv6 addresses aren't learnt through DHCPv6 so it's probably the manual stuff for "challenger" you added to dhcp?

Do you do the ping6 with the same hostname or using the IPv6 address to make it work again?

Does it make a difference if you try to SSH to the IPv6 address instead of the hostname?

Usually inter-device connectivity shouldn't be a concern of the router especially since you have a bridged lan-interface, however there is some multicast management involved in the bridge-layer.

Could you try to add:
option igmp_snooping 0

into the config interface lan section of /etc/config/network and do a reboot afterwards and see if it makes any difference?

comment:2 Changed 4 years ago by cyrus

  • Cc cyrus@… added

comment:3 in reply to: ↑ 1 Changed 4 years ago by mhoran

Replying to cyrus:

OK, that sounds a bit strange. First of all, there is no difference as to how ping6 and ssh do the neighbor lookup (in fact the kernel handles it for both).

Ah, interesting. Perhaps it's just that the ping is more "persistent", in that it tries to make a new connection every second. The first couple of pings fail, then succeed, in the following way:

[mhoran@mhoran-x1-carbon] ~% ping6 challenger
PING challenger(2604:2000:1280:a089:82ee:73ff:fe12:71cf) 56 data bytes
From 2604:2000:1280:a089::1 icmp_seq=1 Destination unreachable: Port unreachable
64 bytes from 2604:2000:1280:a089:82ee:73ff:fe12:71cf: icmp_seq=3 ttl=64 time=1.53 ms

SSH typically fails to connect immediately on the first try, if I force IPv6 (or use the address directly).

[mhoran@mhoran-x1-carbon] ~% ssh -vvv -6 challenger
OpenSSH_6.6.1, OpenSSL 1.0.1i-dev xx XXX xxxx
debug1: Reading configuration data /home/mhoran/.ssh/config
debug1: /home/mhoran/.ssh/config line 3: Applying options for challenger
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to challenger [2604:2000:1280:a089:82ee:73ff:fe12:71cf] port 22.
debug1: connect to address 2604:2000:1280:a089:82ee:73ff:fe12:71cf port 22: Connection refused
ssh: connect to host challenger port 22: Connection refused

Subsequent tries will typically hang, then time out.

[mhoran@mhoran-x1-carbon] ~% ssh -vvv 2604:2000:1280:a089:82ee:73ff:fe12:71cf
OpenSSH_6.6.1, OpenSSL 1.0.1i-dev xx XXX xxxx
debug1: Reading configuration data /home/mhoran/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 2604:2000:1280:a089:82ee:73ff:fe12:71cf [2604:2000:1280:a089:82ee:73ff:fe12:71cf] port 22.
debug1: connect to address 2604:2000:1280:a089:82ee:73ff:fe12:71cf port 22: Connection timed out
ssh: connect to host 2604:2000:1280:a089:82ee:73ff:fe12:71cf port 22: Connection timed out

You mentioned fallback to IPv4 so I assume you are using some kind of DNS inside your home for the ssh. Since you have disabled ra_management IPv6 addresses aren't learnt through DHCPv6 so it's probably the manual stuff for "challenger" you added to dhcp?

I'm using dnsmasq to provide a fallback A record, in addition to an AAAA record. I've configured this via /etc/config/dhcp.

I disabled ra_management as I discovered a bug in NetworkManager which would result in only the ULA subnet being configured on hosts. Seeing as auto configuration worked fine, I disabled ra_management so that only the auto-assigned addresses would be on my system.

Do you do the ping6 with the same hostname or using the IPv6 address to make it work again?

Hostname.

Does it make a difference if you try to SSH to the IPv6 address instead of the hostname?

No.

Usually inter-device connectivity shouldn't be a concern of the router especially since you have a bridged lan-interface, however there is some multicast management involved in the bridge-layer.

Could you try to add:
option igmp_snooping 0

into the config interface lan section of /etc/config/network and do a reboot afterwards and see if it makes any difference?

I see the same bug after adding igmp_snooping to /etc/config/network.

comment:4 Changed 4 years ago by cyrus

Thanks for the information. This actually looks more like a firewalling strangeness on the challenger system but nothing the router is involved with.

If it was a neighbor lookup issue like you suggested would get something like:
Destination unreachable: Address unreachable

However "Connection refused" and "Port unreachable" look like the challenger-host actively denied the packages. Why it changes that behaviour in such a strange way at somepoints is a mystery to me.

comment:5 Changed 4 years ago by mhoran

I double checked challenger and confirmed that there are no iptables rules (I don't think the module was even loaded until I typed the command). I also checked /etc/hosts.allow and /etc/hosts.deny and didn't see anything in there.

I ran tcpdump on the challenger system and found no SSH packets were received on the eth0 interface when I get connection refused.

The output is a bit different once I've gotten connection refused:

13:11:31.430470 IP6 2604:2000:1280:a089:5e51:4fff:fead:5246.50420 > challenger.ssh: Flags [S], seq 3610756443, win 28800, options [mss 1440,sackOK,TS val 236674824 ecr 0,nop,wscale 7], length 0
13:11:31.430520 IP6 challenger.ssh > 2604:2000:1280:a089:5e51:4fff:fead:5246.50420: Flags [S.], seq 3575874884, ack 3610756444, win 28560, options [mss 1440,sackOK,TS val 25167902 ecr 236674824,nop,wscale 7], length 0
13:11:31.430748 IP6 fe80::c43d:c7ff:fe9b:46fc > challenger: ICMP6, redirect, 2604:2000:1280:a089:5e51:4fff:fead:5246 to 2604:2000:1280:a089:5e51:4fff:fead:5246, length 136
13:11:31.430765 IP6 2604:2000:1280:a089:5e51:4fff:fead:5246.50420 > challenger.ssh: Flags [R], seq 3610756444, win 0, length 0
13:11:32.337759 IP challenger.60949 > brooklyn.matthoran.com.domain: 15418+ PTR? 6.4.2.5.d.a.e.f.f.f.f.4.1.5.e.5.9.8.0.a.0.8.2.1.0.0.0.2.4.0.6.2.ip6.arpa. (90)
13:11:32.429101 IP6 2604:2000:1280:a089:5e51:4fff:fead:5246.50420 > challenger.ssh: Flags [S], seq 3610756443, win 28800, options [mss 1440,sackOK,TS val 236675074 ecr 0,nop,wscale 7], length 0
13:11:32.429120 IP6 challenger.ssh > 2604:2000:1280:a089:5e51:4fff:fead:5246.50420: Flags [S.], seq 3591478356, ack 3610756444, win 28560, options [mss 1440,sackOK,TS val 25168152 ecr 236675074,nop,wscale 7], length 0
13:11:32.429353 IP6 fe80::c43d:c7ff:fe9b:46fc > challenger: ICMP6, redirect, 2604:2000:1280:a089:5e51:4fff:fead:5246 to 2604:2000:1280:a089:5e51:4fff:fead:5246, length 136
13:11:32.429369 IP6 2604:2000:1280:a089:5e51:4fff:fead:5246.50420 > challenger.ssh: Flags [R], seq 3610756444, win 0, length 0

Oddly enough this behavior happens in both directions. If I try to connect to my laptop via IPv6 from challenger, I see the same behavior. Again, I'm not aware of any firewall or SSH configuration that may be rejecting connections, as I'm able to connect fine from the outside world.

comment:6 follow-up: Changed 4 years ago by cyrus

OK judging from that dump, challenger has a problem with its neighbor-discovery and or routing. The return packets to 2604:2000:1280:a089:5e51:4fff:fead:5246 should be sent to the MAC-address ending with ...:ad:52:46 however challenger sents the return packets to ...:9b:46:fc. If the latter happens to be the OpenWrt router than challenger might be missing an on-link route for its prefix. You should probably check challenger's routing table if it has entries for 2604:2000:1280:a089::/64 and what they look like.

Add: You can see that by the ICMP6 redirects sent by :9b:46:fc that redirect 2604:2000:1280:a089:5e51:4fff:fead:5246 to 2604:2000:1280:a089:5e51:4fff:fead:5246 meaning challenger should directly send to 2604:2000:1280:a089:5e51:4fff:fead:5246 and not use the router.

Last edited 4 years ago by cyrus (previous) (diff)

comment:7 in reply to: ↑ 6 Changed 4 years ago by mhoran

Replying to cyrus:

OK judging from that dump, challenger has a problem with its neighbor-discovery and or routing. The return packets to 2604:2000:1280:a089:5e51:4fff:fead:5246 should be sent to the MAC-address ending with ...:ad:52:46 however challenger sents the return packets to ...:9b:46:fc. If the latter happens to be the OpenWrt router than challenger might be missing an on-link route for its prefix. You should probably check challenger's routing table if it has entries for 2604:2000:1280:a089::/64 and what they look like.

9b:46:fc is indeed the router (eth0.1).

Add: You can see that by the ICMP6 redirects sent by :9b:46:fc that redirect 2604:2000:1280:a089:5e51:4fff:fead:5246 to 2604:2000:1280:a089:5e51:4fff:fead:5246 meaning challenger should directly send to 2604:2000:1280:a089:5e51:4fff:fead:5246 and not use the router.

I thought the ICMP6 redirects were weird. Though I'm less familiar with IPv6 than I am with IPv4, I assumed a host would still try to communicate with its subnet directly. That seems not to be the case in my situation.

The eth0 interface on challenger shows the following:

          inet6 addr: 2604:2000:1280:a089:82ee:73ff:fe12:71cf/64 Scope:Global

The wlan0 interface on mhoran-x1-carbon shows the following:

          inet6 addr: 2604:2000:1280:a089:5e51:4fff:fead:5246/64 Scope:Global

I also tried changing ra_management back to 1 to enable DHCPv6 address assignment. Unfortunately that didn't solve the problem.

Here is the output of /sbin/ip -6 route

2604:2000:1280:a089::/64 via fe80::c43d:c7ff:fe9b:46fc dev wlan0  proto ra  metric 10 
fe80::/64 dev wlan0  proto kernel  metric 256 
default via fe80::c43d:c7ff:fe9b:46fc dev wlan0  proto static  metric 1024 

Is the router associated with the 2604:2000:1280:a089::/64 subnet the cause of the problem?

comment:8 Changed 4 years ago by mhoran

I added a ULA prefix to OpenWRT to test the above theory. It turns out that the ULA prefix does not have a a router associated with the link subnet.

2604:2000:1280:a089::/64 via fe80::c43d:c7ff:fe9b:46fc dev wlan0  proto ra  metric 10 
fd00:db80::/64 dev wlan0  proto ra  metric 10 
fd00:db80::/48 via fe80::c43d:c7ff:fe9b:46fc dev wlan0  proto ra  metric 10 
fe80::/64 dev wlan0  proto kernel  metric 256 
default via fe80::c43d:c7ff:fe9b:46fc dev wlan0  proto static  metric 1024 

This is what I would expect, given a host should communicate directly with its subnet as discussed above. So, for some reason, odhcpd seems to be sending a route to the prefix provided by my ISP pointing to itself as the router. I'm not sure why the ULA prefix would work fine but the ISP provided prefix would not.

All communication with challenger via the ULA prefix works just fine. I can ping6 without receiving "Port unreachable" as above, and SSH works on the first try.

Last edited 4 years ago by mhoran (previous) (diff)

comment:9 Changed 4 years ago by cyrus

  • Resolution set to fixed
  • Status changed from new to closed

OK I see your problem now. challenger receives both an on-link route and a route for the prefix to the router in the RA and seems to prefer the off-link route for whatever reason.
Not sure if its a bug in challenger's IPv6 implementation or not. Anyway I commited a workaround to OpenWrt that skips sending the off-link route if it has to same size as the assigned prefix (/64 here). Commited in r42024 should be in snapshots soon and in RC4 / final of BB (don't think it will make it for RC3).

comment:10 Changed 3 years ago by mhoran

Awesome! I was just setting up a buildroot to poke at odhcpd myself. I'm impressed with how simple and approachable the code is.

I just tested this out myself and all seems to be resolved. Thanks again for your help!

Add Comment

Modify Ticket

Action
as closed .
The resolution will be deleted. Next status will be 'reopened'.
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.