Hi Andrew,
On Wed, Jan 29, 2020 at 08:15:11PM -0800, Andrew Pullin wrote:
Uh-oh, I think I did a poor job of explaining it. Although you seem to have
surmised most of it.
The full rundown:
Host1:
- Micontroller device with no MAC or PHY
- running NetX IP stack
- Runs a PPP client, specifically PPPoS using TTL UART
- default route: ppp0
- ppp0 is the only network interface available
- ARP is supported and enabled
- IP 10.0.0.2 is assigned during PPP negotiation
There are no ARP in PPP, but whatever, this configuration is correct.
Host2:
- Micontroller device, ESP32
- has both WiFi and Ethernet PHY hardware (IP101GRI Ethernet PHY), used
mutually exclusively
- Running lwIP
- WiFi and Ethernet are independently verified to work to reach WAN from
this host
- PPPoS server
- ppp0 : 10.0.0.1/32
- IP for server and client are hard-coded in
- I believe it is /32, as the netmask is reported as 255.255.255.255
- When running w/ WiFi:
- wlan0 : ip 10.0.200.136/16 (IP and /16 from DHCP server)
- When running w/ Ethernet:
- eth0 : ip 10.0.200.113/16 (IP and /16 from DHCP server)
- Routing is unknown. lwIP built with IP_FORWARD enabled.
- default route: wlan0 or eth0 (assumed)
- ARP is enabled.
10.0.0.1/32 and 10.0.0.2/32 are within 10.0.0.0/16 subnet, that
shouldn't be a problem here, this is usually a bad practice and it
should be avoided. This is a problem for hosts on 10.0.0.0/16 that don't
know there is a hole in the subnet, since there are only 3 hosts here,
which is an expection rather than the rule, it is safe by luck.
WiFi AP's:
- AP only (no routing)
- Connected to Host3 by Ethernet
- MACs 0c:8d:db:6e:f0:03 or 0c:8d:db:6e:f0:88 (can't control
association)
Host3:
- Router (pfSense)
- DHCP server runs here
- Routes to WAN, does NAT, eth0 <-> eth1
- default route: set by DHCP, toward eth1
- no other route (as far as I know)
- Intranet WiFi AP's are connected via switch to eth0
- Intranet Ethernet devices are connected via switch to eth0
- eth0 : 10.0.1.1/16 , facing intranet, MAC 00:e0:67:18:54:95
- eth1 : IP from ISP, facing public IPv4 internet, MAC 00:e0:67:18:54:94
Here is the real problem, you need a route on Host3 telling that
10.0.0.2/32 is behind 10.0.200.136 (or 10.0.200.113). You can also add a
route for 10.0.0.1/32 but it's just a bonus.
Without this route, Host3 don't know how to route the packet and will
try to find an host on the 10.0.0.0/16 subnet using ARP. Then it's just
luck, and depends on what Host3 and Host2 are doing:
- If the packet is sent in a broadcast Ethernet frame by Host3 it might
works if Host2 catches the packet and forwards it to Host1.
- If the ARP request gets answered by Host2 when Host3 send their ARP
request to find who is 10.0.0.2 on the 10.0.0.0/16 subnet, then it also
works. This is known as ARP Proxy, lwIP does not support that.
It's just luck, using lazy features of IP stack to make it works even if
the configuration is broken should be avoided :-)
Layout:
Host1 <-----------------> Host2 <-------------------------> Host3
<---------> public internet
PPPoS Ethernet
or
Host1 <-----------------> Host2 <----------------> WiFi AP
<------------------> Host3 <-------> public internet
PPPoS WiFi Ethernet
For the case of using the WiFi connection, there are separate WiFi AP's in
the way.
Other than that, the two cases are host2 connecting to the local subnet via
wlan0 or connecting via eth0.
I guess the only reason it works with WiFi and not with Ethernet is
because WiFi is intrinsically broadcast-based while Ethernet is switched
(well, I hope so, unless you live in '90s :p). This is the "works
because Host2 catches the broadcast packet" case I've said before.
I am not manually adding any static routes in either Host1 or Host2.
I was worried that Host2 would not be able to accomplish the "connection
sharing" until I found the IP_FORWARDING option. That initially worked with
Host2 using WiFi without any manually added static routes.
This is not "connection sharing", this is IP routing, not NAT !,
"connection sharing" is usually used for hosts that do SNAT on their
Interface facing Internet.
IP routing needs a complete routing table on all hosts telling how to
reach (i.e. which next-hop to use) every host of the whole network.
Example with 2 subnets and 3 Hosts:
A-eth0 --------------------- eth0-B-eth1 --------------------- eth0-C
192.168.0.1/24 192.168.0.2/24 172.16.1.1/24 172.16.1.2/24
Host A routing table:
192.168.0.0/24 dev eth0
172.16.1.0/24 via 192.168.0.2
Host B routing table:
192.168.0.0/24 dev eth0
172.16.1.0/24 dev eth1
Host C routing table:
192.168.0.0/24 via 172.16.1.1
172.16.1.0/24 dev eth0
dev = directly connected
via = next-hop, gateway if you prefer
What you could see is that each host have a similar routing table, just
the next-hop changes. Default route is just a way to reduce the routing
table size by removing all entries that have an identical next-hop.
Unless I am missing something being changed in the config, that is the
config both both the case of Host2 using wlan0 and for using eth0. The only
change I can observe is that host2 IP is given a different IP by host3 DHCP.
Very interesting that you already see something wrong ... that means I am
likely missing something obvious. I certainly hope it is something simple to
tweak to make this work over host2's eth0!
Unfortunately, I have no ability to do debug on the ESP32, so I cannot check
how the stack is really behaving when checking if anything needs to be
forward.
Using the ETHARP_TRUST option does not appear to have an effect, either.
I did not enable Proxy ARP in either case, and it doesn't look like the
drivers around netif enable it, either.
As far as I know, NAT is only running in Host3, which is the router between
the whole subnet and the public internet.
I regenerated the logs with ARP logging and with relative timestamps, in
case that helps:
Host2 using wlan0, host1 is able to ping 8.8.8.8 and 10.0.1.1 :
https://pastebin.com/GJkSsPbb
Host2 using eth0, host1 is not able to ping 8.8.8.8 and 10.0.1.1 :
https://pastebin.com/riE345HN
The help is really appreciated.
Sylvain
_______________________________________________
lwip-users mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/lwip-users