ODROID-C1 networking problems?

[Havard Eidnes <he%NetBSD.org@localhost>]

Hi,

I have a couple of ODROID C1 boxes.  One of them appear to have
intermittent networking problems, in particular with receiving
packets.

droid# uname -a
NetBSD droid.urc.uninett.no 7.99.58 NetBSD 7.99.58 (ODROID-C1) #0: Thu Jan 12 10:12:54 CET 2017  he%mt.urc.uninett.no@localhost:/u/build/HEAD/obj/evbearmv7hf-el/sys/arch/evbarm/compile/ODROID-C1 evbarm
droid# dmesg | grep awge0
awge0 at amlogicio0: Gigabit Ethernet Controller
awge0: interrupting on irq 40
awge0: Ethernet address: 00:1e:06:20:1b:50
rgephy0 at awge0 phy 0: RTL8169S/8110S/8211 1000BASE-T media interface, rev. 6
rgephy1 at awge0 phy 1: RTL8169S/8110S/8211 1000BASE-T media interface, rev. 6
droid#

While pinging this box from a remote host (should give 1pps in),
and repeating

   vmstat -i | grep 'irq 40'

does not produce something which increases with 1 per second:

droid# vmstat -i | grep 'irq 40'
armgic (cpu0) irq 40                             49        0
droid# vmstat -i | grep 'irq 40'
armgic (cpu0) irq 40                             50        0
droid# vmstat -i | grep 'irq 40'
armgic (cpu0) irq 40                             51        0
droid# vmstat -i | grep 'irq 40'
armgic (cpu0) irq 40                             52        0
droid# 

(There were several seconds between each of thse.)

Tcpdump'ing on a neighboring host when it tries to ARP for this
neighbor on the same LAN shows that it sends ARP packets fine,
but appears to be "deaf" to the reception of the responses, and
the ARP requests are just retransmitted until the ODROID gives
up.

This same kernel worked fine for a while, I was midway into
checking out pkgsrc when it crashed (for other reasons), but now
I can't get it to be "live" on our network at all.  I've both
done warm and cold reboots.  Ifconfig says everything is fine:

droid# ifconfig awge0
awge0: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        ec_capabilities=1<VLAN_MTU>
        ec_enabled=0
        address: 00:1e:06:20:1b:50
        media: Ethernet autoselect (1000baseT full-duplex)
        status: active
        inet6 fe80::21e:6ff:fe20:1b50%awge0 prefixlen 64 scopeid 0x2
        inet 158.38.39.50 netmask 0xffffff00 broadcast 158.38.39.255
droid#

It has link etc.

droid# netstat -in
Name  Mtu   Network       Address              Ipkts Ierrs    Opkts Oerrs Colls
lo0   33180 <Link>                                14     0       14     0     0
lo0   33180 127/0         127.0.0.1               14     0       14     0     0
lo0   33180 default       ::1                     14     0       14     0     0
lo0   33180 default       fe80::1                 14     0       14     0     0
awge0 1500  <Link>        00:1e:06:20:1b:50       22     0       93     0     0
awge0 1500  default       fe80::21e:6ff:fe2       22     0       93     0     0
awge0 1500  158/0         158.38.39.50            22     0       93     0     0
droid#

The non-zero parts of "netstat -s" says:

droid# netstat -s | grep -v -w 0
...
udp:
        25 datagrams received
        25 delivered
        14 PCB hash misses
        67 datagrams output
ip:
        25 total packets received
        25 packets for this host
        67 packets sent from this host
ip6:
        11 total packets received
        11 packets for this host
        20 packets sent from this host
        Input packet histogram:
                UDP: 11
        Mbuf statistics:
                3 one mbufs
                two or more mbuf:
                        lo0 = 8
icmp6:
        Output packet histogram:
                multicast listener report: 8
                neighbor solicitation: 1
udp6:
        11 datagrams received
        11 delivered
        11 datagrams output
arp:
        42 packets sent
                42 request packets
        42 packets deferred pending ARP resolution
                42 dropped

and that's it.

My other ODROID which runs 7.99.47 is newly booted and which
works fine at the moment says

arp:
        3 packets sent
                3 request packets
        1775 packets received
                2 reply packets
                1773 valid request packets
                1773 broadcast/multicast packets
                3 packets with null source IP address
        2 packets deferred pending ARP resolution
                2 sent

I reverted to the same 7.99.47 kernel on the "problem" ODROID,
and on first boot it worked on the network, but I had a crash
(file system related), and after the auto-reboot it has again
fallen "off the net".

Has anyone seen anything like this before?

Does anyone have suggestions for what more debugging is needed to
narrow down the actual cause?

Regards,

- Håvard