ESP32 network discovery through LAN IP scanning and Ethernet ARP monitoring

Coming from Maintenance mode galore, this topic is about further investigating how we could manage to find out about the IP address when an ESP32-based Pycom device is connected to the same local network your workstation is running on.

The device hostname will be espressif .

Investigating ESP32 WiFi connectivity behavior

We will try to conduct some tests about how the FiPy connects to the WiFi network, especially regarding DHCP leasing as well as DNS- and ARP-table announcements and how this information can be inquired and processed appropriately.

We haven’t tried to change that name yet. Suggestions for doing so are welcome.

As our FiPy libero device we are using for conducting these tests doesn’t have any sensors attached, it will be the perfect target. Its alive-time and the connectivity window is so small the device doesn’t even manage to get into the network to be found by its name espressif appropriately – at least for me: socket.gethostbyname('espressif') just yields a cute "host not found", even when run in a tight loop to prevent eventual misses.

So, this is how my arp table looks after running the device for one hour on my local network, made up by a standard Fritz!Box 7660.

watch -n0.5 "arp -a | grep '80:7d:3a'"
? (192.168.178.113) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.115) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.117) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.119) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.121) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.123) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.125) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.127) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.130) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.132) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.135) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.137) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.139) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.141) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.143) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.145) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]

Another session right after the first one running for some minutes yielded

? (192.168.178.20)  at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.159) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.162) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.165) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.167) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.169) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]

Another bunch of different snapshots:

? (192.168.178.20) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
espressif.fritz.box (192.168.178.21) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.22) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.20) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.21) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.23) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.24) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.20) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.21) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.23) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
espressif.fritz.box (192.168.178.24) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.20) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
espressif.fritz.box (192.168.178.21) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.24) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.20) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.21) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.22) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.24) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]

When the device is on the network, it is visible for just a few seconds which might look like that.

ping 192.168.178.20

64 bytes from 192.168.178.20: icmp_seq=185 ttl=255 time=53.298 ms
64 bytes from 192.168.178.20: icmp_seq=186 ttl=255 time=3.768 ms
64 bytes from 192.168.178.20: icmp_seq=187 ttl=255 time=4.700 ms
64 bytes from 192.168.178.20: icmp_seq=188 ttl=255 time=17.620 ms

ping 192.168.178.21

64 bytes from 192.168.178.21: icmp_seq=93 ttl=255 time=53.213 ms
64 bytes from 192.168.178.21: icmp_seq=94 ttl=255 time=5.307 ms
64 bytes from 192.168.178.21: icmp_seq=95 ttl=255 time=6.069 ms
64 bytes from 192.168.178.21: icmp_seq=96 ttl=255 time=4.014 ms
64 bytes from 192.168.178.21: icmp_seq=97 ttl=255 time=3.092 ms

ping 192.168.178.22

64 bytes from 192.168.178.22: icmp_seq=177 ttl=255 time=4899.730 ms
64 bytes from 192.168.178.22: icmp_seq=178 ttl=255 time=3898.889 ms
64 bytes from 192.168.178.22: icmp_seq=179 ttl=255 time=2895.240 ms
64 bytes from 192.168.178.22: icmp_seq=180 ttl=255 time=1893.755 ms
64 bytes from 192.168.178.22: icmp_seq=181 ttl=255 time=889.207 ms
64 bytes from 192.168.178.22: icmp_seq=182 ttl=255 time=5.730 ms
64 bytes from 192.168.178.22: icmp_seq=183 ttl=255 time=6.098 ms
64 bytes from 192.168.178.22: icmp_seq=184 ttl=255 time=6.574 ms
64 bytes from 192.168.178.22: icmp_seq=185 ttl=255 time=3.800 ms
64 bytes from 192.168.178.22: icmp_seq=186 ttl=255 time=5.970 ms

ping 192.168.178.20

64 bytes from 192.168.178.20: icmp_seq=363 ttl=255 time=7.287 ms
64 bytes from 192.168.178.20: icmp_seq=364 ttl=255 time=3.836 ms
64 bytes from 192.168.178.20: icmp_seq=365 ttl=255 time=3.356 ms
64 bytes from 192.168.178.20: icmp_seq=366 ttl=255 time=7.973 ms
64 bytes from 192.168.178.20: icmp_seq=367 ttl=255 time=27.211 ms

ping 192.168.178.21

64 bytes from 192.168.178.21: icmp_seq=336 ttl=255 time=76.273 ms
64 bytes from 192.168.178.21: icmp_seq=337 ttl=255 time=3.750 ms
64 bytes from 192.168.178.21: icmp_seq=338 ttl=255 time=3.783 ms
64 bytes from 192.168.178.21: icmp_seq=339 ttl=255 time=5.420 ms
64 bytes from 192.168.178.21: icmp_seq=340 ttl=255 time=3.772 ms

ping 192.168.178.20

64 bytes from 192.168.178.20: icmp_seq=543 ttl=255 time=107.740 ms
64 bytes from 192.168.178.20: icmp_seq=544 ttl=255 time=3.901 ms
64 bytes from 192.168.178.20: icmp_seq=545 ttl=255 time=4.072 ms
64 bytes from 192.168.178.20: icmp_seq=546 ttl=255 time=4.066 ms

ping 192.168.178.20

64 bytes from 192.168.178.20: icmp_seq=632 ttl=255 time=56.037 ms
64 bytes from 192.168.178.20: icmp_seq=633 ttl=255 time=10.913 ms
64 bytes from 192.168.178.20: icmp_seq=634 ttl=255 time=3.716 ms
64 bytes from 192.168.178.20: icmp_seq=635 ttl=255 time=7.521 ms
64 bytes from 192.168.178.20: icmp_seq=636 ttl=255 time=5.889 ms

Connectivity seems to iterate between x.20, x.21 and x.22.

Some phone calls An hour later…

? (192.168.178.74) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.77) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.79) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.81) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.83) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.85) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.87) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.89) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.91) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.93) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.95) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.97) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.99) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.102) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.104) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.106) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.109) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]

During that session, my workstation was sitting on 192.168.178.101.

Maybe we can also catch it by scanning the network like

sudo nmap -sn 192.168.178.0/24

as this keeps scanning time reasonably low.

Nmap done: 256 IP addresses (3 hosts up) scanned in 1.99 seconds

Watching for open telnet ports like

nmap -Pn -p 23 192.168.178.0/24

will probably be too slow.

Nmap done: 256 IP addresses (256 hosts up) scanned in 6.20 seconds

However, we haven’t been able to verify that yet.

We just have been able to hit the device on a single random sampling without having run that in a tight loop yet. Lucky us.

$ nmap -p 23 192.168.178.0/24
Starting Nmap 7.70 ( https://nmap.org ) at 2019-07-04 15:18 CEST

Nmap scan report for fritz.box (192.168.178.1)
Host is up (0.0074s latency).
PORT   STATE  SERVICE
23/tcp closed telnet

Nmap scan report for fritz.repeater (192.168.178.35)
Host is up (0.0071s latency).
PORT   STATE  SERVICE
23/tcp closed telnet

Nmap scan report for 192.168.178.58
Host is up (0.0095s latency).
PORT   STATE SERVICE
23/tcp open  telnet

Nmap scan report for offgrid.fritz.box (192.168.178.101)
Host is up (0.00088s latency).
PORT   STATE  SERVICE
23/tcp closed telnet

Nmap done: 256 IP addresses (4 hosts up) scanned in 3.10 seconds

It’s this guy:

Nmap scan report for 192.168.178.58
Host is up (0.0095s latency).
PORT   STATE SERVICE
23/tcp open  telnet

By not skipping host discovery like

-Pn: Treat all hosts as online -- skip host discovery

scanning time seems to be reasonably fast:

$ nmap -p 23 192.168.178.0/24
[...]
Nmap done: 256 IP addresses (4 hosts up) scanned in 3.10 seconds

Recap

After power cycling Fritz and Libero [1], Libero starts obtaining addresses starting at x.20 again.

ARP table

? (192.168.178.20) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.22) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.24) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.26) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.29) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.31) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]
? (192.168.178.33) at 80:7d:3a:c2:de:44 on en1 ifscope [ethernet]

nmap

Looping manually

$ nmap -p 23 -oG - 192.168.178.0/24
# Nmap 7.70 scan initiated Thu Jul  4 15:39:08 2019 as: nmap -p 23 -oG - 192.168.178.0/24
Host: 192.168.178.1 (fritz.box)	Status: Up
Host: 192.168.178.1 (fritz.box)	Ports: 23/closed/tcp//telnet///
Host: 192.168.178.33 ()	Status: Up
Host: 192.168.178.33 ()	Ports: 23/open/tcp//telnet///
Host: 192.168.178.35 (fritz.repeater)	Status: Up
Host: 192.168.178.35 (fritz.repeater)	Ports: 23/closed/tcp//telnet///
Host: 192.168.178.101 (offgrid.fritz.box)	Status: Up
Host: 192.168.178.101 (offgrid.fritz.box)	Ports: 23/closed/tcp//telnet///
# Nmap done at Thu Jul  4 15:39:10 2019 -- 256 IP addresses (4 hosts up) scanned in 1.36 seconds

Hit on x.33 after ~1,5 scanning time:

$ nmap -p 23 -oG - 192.168.178.0/24
Host: 192.168.178.33 ()	Status: Up
Host: 192.168.178.33 ()	Ports: 23/open/tcp//telnet///
# Nmap done at Thu Jul  4 15:39:10 2019 -- 256 IP addresses (4 hosts up) scanned in 1.36 seconds

  1. … and clearing out the ARP table on Offgrid ↩︎

nmap performance options

Looking at you.

TIMING AND PERFORMANCE:
  Options which take <time> are in seconds, or append 'ms' (milliseconds),
  's' (seconds), 'm' (minutes), or 'h' (hours) to the value (e.g. 30m).
  -T<0-5>: Set timing template (higher is faster)
  --min-hostgroup/max-hostgroup <size>: Parallel host scan group sizes
  --min-parallelism/max-parallelism <numprobes>: Probe parallelization
  --min-rtt-timeout/max-rtt-timeout/initial-rtt-timeout <time>: Specifies
      probe round trip time.
  --max-retries <tries>: Caps number of port scan probe retransmissions.
  --host-timeout <time>: Give up on target after this long
  --scan-delay/--max-scan-delay <time>: Adjust delay between probes
  --min-rate <number>: Send packets no slower than <number> per second
  --max-rate <number>: Send packets no faster than <number> per second

Outcome

Host probing by ping scan

$ sudo nmap -sn --min-parallelism 128 -oG - 192.168.178.0/24
Warning: Your --min-parallelism option is pretty high!  This can hurt reliability.
# Nmap 7.70 scan initiated Thu Jul  4 15:51:00 2019 as: nmap -sn --min-parallelism 128 -oG - 192.168.178.0/24
Host: 192.168.178.1 (fritz.box)	Status: Up
Host: 192.168.178.34 (andromeda.fritz.box)	Status: Up
Host: 192.168.178.35 (fritz.repeater)	Status: Up
Host: 192.168.178.53 ()	Status: Up
Host: 192.168.178.101 (offgrid.fritz.box)	Status: Up
# Nmap done at Thu Jul  4 15:51:01 2019 -- 256 IP addresses (5 hosts up) scanned in 0.68 seconds

Port probing, with Host discovery

$ nmap -p 23 --min-parallelism 128 -oG - 192.168.178.0/24
Warning: Your --min-parallelism option is pretty high!  This can hurt reliability.
# Nmap 7.70 scan initiated Thu Jul  4 15:52:32 2019 as: nmap --min-parallelism 128 -p 23 -oG - 192.168.178.0/24
Host: 192.168.178.1 (fritz.box)	Status: Up
Host: 192.168.178.1 (fritz.box)	Ports: 23/closed/tcp//telnet///
Host: 192.168.178.35 (fritz.repeater)	Status: Up
Host: 192.168.178.35 (fritz.repeater)	Ports: 23/closed/tcp//telnet///
Host: 192.168.178.55 ()	Status: Up
Host: 192.168.178.55 ()	Ports: 23/open/tcp//telnet///
Host: 192.168.178.101 (offgrid.fritz.box)	Status: Up
Host: 192.168.178.101 (offgrid.fritz.box)	Ports: 23/closed/tcp//telnet///
# Nmap done at Thu Jul  4 15:52:33 2019 -- 256 IP addresses (4 hosts up) scanned in 1.27 seconds

Isn’t this a function of DHCP? We can recommend to assign always the same IP. This is possible in the Fritzbox settings, but of cause no general solution.

Looks like DHCP lease times are not honored at all.

That’s true. Static DHCP assignments – while tempting – is nothing we should put our feet on.

May I humbly ask @MKO, @clemens or @tonke to check whether this will work on a WSL2’ed Debian/Ubuntu and tell me about its outcome? I’m actually interested in success and speed. Thanks already!

P.S.: If this works and you feel adventurous, you might want to try speeding things up:

nmap -p 23 --min-parallelism 128 -oG - 192.168.178.0/24

I tested it several times, all with the same result. I think it does not work on WSL.

After that, i have install nmap for Windows

1 Like

Dear @MKO,

thanks for testing.

Sorry, I just recognized there might have slipped something through my guidelines. Did you also try adding sudo to the command like

sudo nmap -p 23 -oG - 192.168.178.0/24

?

However, as

seems to work at last, we know it could be an option to use.

Saying that, I’m in fact looking actually at Scapy. They are talking about that the most recent version would also work on Windows, see Download and Installation — Scapy 2.4.3rc2.dev16 documentation.

$ pip install scapy

$ sudo scapy

                     aSPY//YASa
             apyyyyCY//////////YCa       |
            sY//////YSpcs  scpCY//Pp     | Welcome to Scapy
 ayp ayyyyyyySCP//Pp           syY//C    | Version 2.4.0
 AYAsAYYYYYYYY///Ps              cY//S   |
         pCCCCY//p          cSSps y//Y   | https://github.com/secdev/scapy
         SPPPP///a          pP///AC//Y   |
              A//A            cyP////C   | Have fun!
              p///Ac            sC///a   |
              P////YCpc           A//A   | We are in France, we say Skappee.
       scccccp///pSP///p          p//Y   | OK? Merci.
      sY/////////y  caa           S//P   |             -- Sebastien Chabal
       cayCyayP//Ya              pY/Ya   |
        sY/PsY////YCc          aC//Yp
         sc  sccaCY//PCypaapyCP//YSs
                  spCPY//////YPSps
                       ccaacs

>>> arping('192.168.178.0/24')
Begin emission:
***.*.Finished sending 256 packets.
.
Received 7 packets, got 4 answers, remaining 252 packets
  bc:05:43:e0:15:4d 192.168.178.1
  04:d6:aa:84:d5:1f 192.168.178.34
  80:7d:3a:c2:de:44 192.168.178.154
  0e:96:d7:cf:cb:ea 192.168.178.35
(<ARPing: TCP:0 UDP:0 ICMP:0 Other:4>, <Unanswered: TCP:0 UDP:0 ICMP:0 Other:252>)

The Scapy blueprint for a Simplistic ARP Monitor looks promising.

Minimal ARP monitor in Python

Install

virtualenv .venv3
source .venv3/bin/pip
pip install scapy

arpmon.py

Put this into a file called "arpmon.py".

#! /usr/bin/env python
import sys
from scapy.all import *


def arp_monitor_callback(pkt):
    if ARP in pkt and pkt[ARP].op in (1,2): #who-has or is-at
        print(pkt.sprintf("%ARP.hwsrc% %ARP.psrc%"))
        sys.stdout.flush()


sniff(prn=arp_monitor_callback, filter="arp", store=0)

Invoke

sudo python arpmon.py | grep '80\:7d\:3a'

Result

80:7d:3a:c2:de:44 0.0.0.0
80:7d:3a:c2:de:44 0.0.0.0
80:7d:3a:c2:de:44 0.0.0.0
80:7d:3a:c2:de:44 192.168.178.174
80:7d:3a:c2:de:44 192.168.178.174
80:7d:3a:c2:de:44 192.168.178.174
80:7d:3a:c2:de:44 192.168.178.174
80:7d:3a:c2:de:44 192.168.178.174

on su, or with sudo it dosn´t work.


I have testet arp -a on windows comand line as admin, too.
After deleting the cache. I wait only 1 transmission
image

for /l %i in (1,1,255) DO @ping 192.168.178.%i -n 1 | find "Bytes=" 

works sometimes too, but very … verry … slowly.
tzzzzzz…

Arp on Win Comand line with 2 Fipy
arp -a | find “80-7d-3a”

image