Beware Children and HomePlug networks if you want to avoid Self-Looped problems!

As you might expect, I have a rather complicated home network. At the ‘core’ is a Cisco Catalyst switch which makes problem debugging very easy – normally.

Alongside the normal UTP connected devices across the house, I also use HomePlug to push Ethernet over my electrical ring mains. This connects a couple of wireless access points and things like a Microsoft Xbox 360. In fact there are two HomePlug networks connected together using UTP (as I have two different electrical supplies I am using different HomePlug network names to avoid Spanning Tree loops caused by the signal leaking out of the house and back in on the other supply – yes that does happen and yes that does cause a loop!).

Now two evenings ago my middle son moved the Xbox to a different place in the house. Yesterday my eldest son reported Internet access problems from a desktop PC that is connected to the HomePlug network (though it also has wireless). When I got round to looking at it, I noticed that the connection from the HomePlug network to the Catalyst switch was down. There was no green LED on the switch port.

(ignore times in these log snippets as some of this is recreated for the benefit of this article)

So I logged into the switch and looked at the log. It said I had a DTP flap:

Apr 23 10:20:47.501: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/12, changed state to down
Apr 23 10:20:52.573: %PM-4-ERR_DISABLE: dtp-flap error detected on Fa0/12, putting Fa0/12 in err-disable state
Apr 23 10:20:54.577: %LINK-3-UPDOWN: Interface FastEthernet0/12, changed state to down

and had disabled the port because of the error (err-disabled status):

Cat3550# sh int faste0/12
FastEthernet0/12 is down, line protocol is down (err-disabled)
Hardware is Fast Ethernet, address is 000b.465b.000c (bia 000b.465b.000c)
Description: Connection to EthernetOverPower network
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Auto-duplex, Auto-speed, media type is 10/100BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:13, output 00:00:14, output hang never
Last clearing of “show interface” counters 14:51:06
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 250000 bits/sec, 431 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
4566009 packets input, 813554637 bytes, 0 no buffer
Received 4566005 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 1872430 multicast, 0 pause input
0 input packets with dribble condition detected
8038 packets output, 634788 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

Note there was masses of input traffic but no output traffic.

I just assumed that this was some temporary blip and CLEARed the interface, but it kept coming back. So to save time I thought I would just put in automatic recovery from a DTP-Flap:

errdisable recovery cause dtp-flap
errdisable recovery interval 30

Sure enough recovery kicked in:

Cat3550#sh errdisable recovery
ErrDisable Reason            Timer Status
—————–            ————–
arp-inspection               Disabled
bpduguard                    Disabled
channel-misconfig            Disabled
dhcp-rate-limit              Disabled
dtp-flap                     Enabled
gbic-invalid                 Disabled
l2ptguard                    Disabled
link-flap                    Disabled
mac-limit                    Disabled
link-monitor-failure         Disabled
loopback                     Disabled
oam-remote-failure           Disabled
pagp-flap                    Disabled
port-mode-failure            Disabled
psecure-violation            Disabled
security-violation           Disabled
sfp-config-mismatch          Disabled
storm-control                Disabled
udld                         Disabled
unicast-flood                Disabled
vmps                         Disabled

Timer interval: 30 seconds

Interfaces that will be enabled at the next timeout:

Interface       Errdisable reason       Time left(sec)
———       —————–       ————–
Fa0/12                  loopback            14

Looking at the port config I realised that I had DTP trunking desirable:

interface FastEthernet0/12
description Connection to EthernetOverPower network
switchport mode dynamic desirable

Aha easily fixed, I would just change the port to an access port since it was not connecting to a DTP capable device – that HomePlug only has one port!

interface FastEthernet0/12
description Connection to EthernetOverPower network
switchport mode access

(portfast is switched off because this is connecting to a spanning-tree capable device)

but then I saw loopback errors.

Apr 23 10:32:41.202: %ETHCNTR-3-LOOP_BACK_DETECTED: Keepalive packet loop-back detected on FastEthernet0/12.
Apr 23 10:32:41.202: %PM-4-ERR_DISABLE: loopback error detected on Fa0/12, putting Fa0/12 in err-disable state
Apr 23 10:32:43.218: %LINK-3-UPDOWN: Interface FastEthernet0/12, changed state to down

This was becoming annoying. The Catalyst switch port was showing a flashing Orange/Amber LED. Masses of traffic was coming into the port but there was no output traffic.

Cat3550# sh int faste0/12
FastEthernet0/12 is up, line protocol is up (connected)
Hardware is Fast Ethernet, address is 000b.465b.000c (bia 000b.465b.000c)
Description: Connection to EthernetOverPower network
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 3/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:03, output hang never
Last clearing of “show interface” counters 15:04:59
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1546000 bits/sec, 2057 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
5581549 packets input, 908408095 bytes, 0 no buffer
Received 5581487 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 2887912 multicast, 0 pause input
0 input packets with dribble condition detected
8198 packets output, 648628 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

Switching on loopback recovery just brought the port back long enough for another loopback packet to disable it. No real traffic was passing. There was no impact on the rest of the ‘real’ network.

Plugging into a different switch port disabled the link because of BPDU protection:

Apr 22 20:03:53.808: %SPANTREE-2-BLOCK_BPDUGUARD: Received BPDU on port Fa0/10 with BPDU Guard enabled. Disabling port.
Apr 22 20:03:53.808: %PM-4-ERR_DISABLE: bpduguard error detected on Fa0/10, putting Fa0/10 in err-disable state
Apr 22 20:03:53.816: %SPANTREE-2-BLOCK_BPDUGUARD: Received BPDU on port Fa0/10 with BPDU Guard enabled. Disabling port.

This is expected:

interface FastEthernet0/10
switchport mode access
spanning-tree portfast
spanning-tree bpduguard enable

I looked at what Spanning Tree thought was going on:

Cat3550#sh span

VLAN0001
Spanning tree enabled protocol ieee
Root ID    Priority    32769
Address     000b.465b.0000
This bridge is the root
Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
Address     000b.465b.0000
Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
Aging Time 300

Interface           Role Sts Cost      Prio.Nbr Type
——————- —- — ——— ——– ——————————–
Fa0/12              Desg BLK 19        128.12   P2p self-looped
Fa0/18              Desg FWD 19        128.18   P2p Edge
Fa0/20              Desg FWD 19        128.20   P2p Edge
Fa0/22              Desg FWD 19        128.22   P2p
Fa0/24              Desg FWD 19        128.24   P2p
Gi0/1               Desg FWD 4         128.25   P2p

(Obviously I should get round to moving out of VLAN1…)

So the port is seen as self-looped and STP Blocked. Why?

I switched off all the other HomePlug devices. No difference.

At this point, I figured that I must have some sort of weird interference from outside given that on this HomePlug network I was using the default HomePlug network name (yes I know I should have changed this) and I knew that the signal can leak outside the house to the sub-station (another hair-pulling debug session from a few weeks ago I found when bridging two default named HomePlug networking using UTP).

I resolved to change the HomePlug network name on all the original HomePlug network devices. And that is when I found the issue. Whilst changing the first HomePlug, I noticed a remote HomePlug device and realised I had missed the multi-port HomePlug unit which connects to the Xbox. The Xbox had been removed and my middle son had (uncharacteristically) tidied up the end of the Ethernet cable plugged into the Xbox by putting it one of the other spare Ethernet ports on that device!

That was the cause of the problem and the reason why loopback packets were seen. Removing the cable fixed the issue – the Catalyst switch port LED went green, Spanning Tree saw the loop go away,

Cat3550#sh span

VLAN0001
Spanning tree enabled protocol ieee
Root ID    Priority    32769
Address     000b.465b.0000
This bridge is the root
Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
Address     000b.465b.0000
Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
Aging Time 300

Interface           Role Sts Cost      Prio.Nbr Type
——————- —- — ——— ——– ——————————–
Fa0/12              Desg LIS 19        128.12   P2p
Fa0/18              Desg FWD 19        128.18   P2p Edge
Fa0/20              Desg FWD 19        128.20   P2p Edge
Fa0/22              Desg FWD 19        128.22   P2p
Fa0/24              Desg FWD 19        128.24   P2p
Gi0/1               Desg FWD 4         128.25   P2p

and the interface started to pass packets properly. (LIS means listening, then the port goes to FWD if there are no STP issues).

No relevant causes for the ‘self-looped’ message appear on Google so perhaps this will help someone, and I will make sure I explain why looping back an Ethernet interface is bad to my son.

Easily done, not always easy to diagnose. At least I have a reason to change that default HomePlug network name now!

Update (Jan 2013):
I did change the HomePlug network name.

I also had a recurrence of a self-looped network seen by Spanning Tree on the port connected to the HomePlug network. This occurred after I replaced a Cisco Catalyst 2950 that was connecting my two (differently named) HomePlug networks with a Cisco Catalyst 3550. So the network now looked like

Cisco 3550_1 Fa0/12 >> HomePlug Network 1 >>> Cisco 3550_2 >> HomePlug Network 2

I also had a couple of VLANs on the intermediate 3550_2 and perhaps something went wobbly because I saw intermittent connectivity when looking at it from the intermediate 3550_2. Debugging the issue from 3550_1 showed the port was going up and down (because I have auto-recovery). I switched off all the HomePlug except the one connected to 3550_1 and the one connected to 3550_2. Using the HomePlug utility on a laptop connected directot to 3550_1’s HomePlug showed 3 remote devices. One was the HomePlug connected to 3550_2, one was something unknown (but it looked like a Cisco MAC address to me), and one was a device with ffffffff as the MAC address! Only HomePlug should be seen of course. Powering down the 3550_2 HomePlug cured the issue so it must have tickled a bug – possibly because the 3550_2 was sending Switchport Mode Dynamic Desirable packets (and thereby saying it was happy to become a trunk port)?

Advertisements

One thought on “Beware Children and HomePlug networks if you want to avoid Self-Looped problems!

  1. pdanderson May 3, 2012 at 4:30 pm Reply

    Never work with children, animals or Ethernet !

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: