Shrewsoft VPN on Windows, Cisco ASA access, and the curious ACL order problem

I was working with a Cisco ASA customer that wished to remain with classic IPSEC IKEv1 access from Windows 8 clients (rather than SSL or Anyconnect client access). Cisco no longer make a VPN client that will load onto Windows 8 (they have no 64 bit support), so I recommended they use the Shrewsoft VPN client.

All seemed to be going well, until the customer reported an issue with some of the v2.2.2 clients whereby they could not access privately addressed hosts over the VPN connection from the Shrewsoft client but using another login account, they could.

We collectively scratched our heads over this until it was realised that if the Split-Tunnel ACL had two or more lines AND the first line gave access to a single host (rather than a subnet), then the entire ACL failed to provide any access. If the Split-Tunnel ACL listed the entries with a subnet first, then the subsequent lines could be single hosts without any issue.

So this would fail because the host entry is listed first:

access-list example_fails extended permit ip host 192.0.0.225 172.16.200.0 255.255.255.0
access-list example_fails extended permit ip 192.9.1.0 255.255.255.0 172.16.200.0 255.255.255.0
access-list example_fails extended permit ip 192.9.215.0 255.255.255.0 172.16.200.0 255.255.255.0

but reorder it and it will work:

access-list example_works extended permit ip 192.9.1.0 255.255.255.0 172.16.200.0 255.255.255.0
access-list example_works extended permit ip 192.9.215.0 255.255.255.0 172.16.200.0 255.255.255.0
access-list example_works extended permit ip host 192.0.0.225 172.16.200.0 255.255.255.0

 

The same ACL in the order that would not work with the Shrewsoft client (example_fails), would work with a 32 bit Cisco IPSEC VPN client, and with a native OSX VPN client. So it was a bug.

Advertisements

Saving OSX Grab as JPEG instead of TIFF

Prior to OSX 10.9.2 I often needed to take screen shots for WordPress or documents, and therefore it is most useful to have them as JPG format.

By default the OSX utility Grab saves files in TIFF format, so you have to open them in Preview and then Export as JPG. Far too much work!

Opening a terminal window and typing

defaults write com.apple.screencapture type jpeg

sets the default save mode to be JPG. (jpeg and jpg are the same thing)

The file type options are:

  • defaults write com.apple.screencapture type png
  • defaults write com.apple.screencapture type pdf
  • defaults write com.apple.screencapture type jpg
  • defaults write com.apple.screencapture type jpeg
  • defaults write com.apple.screencapture type tif
  • defaults write com.apple.screencapture type psd

For the change to be effective you can then do a restart or use the command ‘killall SystemUIServer’ if you wish.

After 10.9.2 something changed to prevent this from working so the majority of my screenshots were done using Skitch, or latter just by using the Evernote Web Clipper extension in Chrome.

OSX Mavericks clock slow or wrong?

After my umpteenth late appearance for a conference call I finally decided that it was time to do something about my Mac Mini clock being slow. The only time (no pun intended) it seemed to be accurate was when I opened the Date and Time System Preference!

Image

As I am in the UK, OSX kindly chooses the European time servers. But clearly either they are running slow or we are just not checking enough to keep the clock accurate.

I could see only the Apple servers were being used by opening a Terminal Window and typing

cat /etc/ntp.conf

The only returned line was

server time.euro.apple.com.

There are NTP (Network Time Protocol) servers made available for use as part of the NTP Pool Project. They seemed like a good substitute for the Apple servers.

Now you can overtype the values Apple places in the Date and Time preferences:

Image

This uk.pool.ntp.org entry returns the address of one of the servers in the pool in a random order to spread the load. But you can actually have multiple values, separated by commas, in that NTP field, so I also added eu.pool.ntp.org:

Image

What does this look like at system level?

Checking again in ntp.conf shows

server uk.pool.ntp.org
server eu.pool.ntp.org

You can see some talk related to this in the stackexchange discussion.

Update: It seems on Mavericks that NTP no longer manages time, the program pacemaker does instead. I will report back to this posting whether my Date and Time preference changes did anything positive.

 

Beware Children and HomePlug networks if you want to avoid Self-Looped problems!

As you might expect, I have a rather complicated home network. At the ‘core’ is a Cisco Catalyst switch which makes problem debugging very easy – normally.

Alongside the normal UTP connected devices across the house, I also use HomePlug to push Ethernet over my electrical ring mains. This connects a couple of wireless access points and things like a Microsoft Xbox 360. In fact there are two HomePlug networks connected together using UTP (as I have two different electrical supplies I am using different HomePlug network names to avoid Spanning Tree loops caused by the signal leaking out of the house and back in on the other supply – yes that does happen and yes that does cause a loop!).

Now two evenings ago my middle son moved the Xbox to a different place in the house. Yesterday my eldest son reported Internet access problems from a desktop PC that is connected to the HomePlug network (though it also has wireless). When I got round to looking at it, I noticed that the connection from the HomePlug network to the Catalyst switch was down. There was no green LED on the switch port.

(ignore times in these log snippets as some of this is recreated for the benefit of this article)

So I logged into the switch and looked at the log. It said I had a DTP flap:

Apr 23 10:20:47.501: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/12, changed state to down
Apr 23 10:20:52.573: %PM-4-ERR_DISABLE: dtp-flap error detected on Fa0/12, putting Fa0/12 in err-disable state
Apr 23 10:20:54.577: %LINK-3-UPDOWN: Interface FastEthernet0/12, changed state to down

and had disabled the port because of the error (err-disabled status):

Cat3550# sh int faste0/12
FastEthernet0/12 is down, line protocol is down (err-disabled)
Hardware is Fast Ethernet, address is 000b.465b.000c (bia 000b.465b.000c)
Description: Connection to EthernetOverPower network
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Auto-duplex, Auto-speed, media type is 10/100BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:13, output 00:00:14, output hang never
Last clearing of “show interface” counters 14:51:06
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 250000 bits/sec, 431 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
4566009 packets input, 813554637 bytes, 0 no buffer
Received 4566005 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 1872430 multicast, 0 pause input
0 input packets with dribble condition detected
8038 packets output, 634788 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

Note there was masses of input traffic but no output traffic.

I just assumed that this was some temporary blip and CLEARed the interface, but it kept coming back. So to save time I thought I would just put in automatic recovery from a DTP-Flap:

errdisable recovery cause dtp-flap
errdisable recovery interval 30

Sure enough recovery kicked in:

Cat3550#sh errdisable recovery
ErrDisable Reason            Timer Status
—————–            ————–
arp-inspection               Disabled
bpduguard                    Disabled
channel-misconfig            Disabled
dhcp-rate-limit              Disabled
dtp-flap                     Enabled
gbic-invalid                 Disabled
l2ptguard                    Disabled
link-flap                    Disabled
mac-limit                    Disabled
link-monitor-failure         Disabled
loopback                     Disabled
oam-remote-failure           Disabled
pagp-flap                    Disabled
port-mode-failure            Disabled
psecure-violation            Disabled
security-violation           Disabled
sfp-config-mismatch          Disabled
storm-control                Disabled
udld                         Disabled
unicast-flood                Disabled
vmps                         Disabled

Timer interval: 30 seconds

Interfaces that will be enabled at the next timeout:

Interface       Errdisable reason       Time left(sec)
———       —————–       ————–
Fa0/12                  loopback            14

Looking at the port config I realised that I had DTP trunking desirable:

interface FastEthernet0/12
description Connection to EthernetOverPower network
switchport mode dynamic desirable

Aha easily fixed, I would just change the port to an access port since it was not connecting to a DTP capable device – that HomePlug only has one port!

interface FastEthernet0/12
description Connection to EthernetOverPower network
switchport mode access

(portfast is switched off because this is connecting to a spanning-tree capable device)

but then I saw loopback errors.

Apr 23 10:32:41.202: %ETHCNTR-3-LOOP_BACK_DETECTED: Keepalive packet loop-back detected on FastEthernet0/12.
Apr 23 10:32:41.202: %PM-4-ERR_DISABLE: loopback error detected on Fa0/12, putting Fa0/12 in err-disable state
Apr 23 10:32:43.218: %LINK-3-UPDOWN: Interface FastEthernet0/12, changed state to down

This was becoming annoying. The Catalyst switch port was showing a flashing Orange/Amber LED. Masses of traffic was coming into the port but there was no output traffic.

Cat3550# sh int faste0/12
FastEthernet0/12 is up, line protocol is up (connected)
Hardware is Fast Ethernet, address is 000b.465b.000c (bia 000b.465b.000c)
Description: Connection to EthernetOverPower network
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 3/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:03, output hang never
Last clearing of “show interface” counters 15:04:59
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1546000 bits/sec, 2057 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
5581549 packets input, 908408095 bytes, 0 no buffer
Received 5581487 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 2887912 multicast, 0 pause input
0 input packets with dribble condition detected
8198 packets output, 648628 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

Switching on loopback recovery just brought the port back long enough for another loopback packet to disable it. No real traffic was passing. There was no impact on the rest of the ‘real’ network.

Plugging into a different switch port disabled the link because of BPDU protection:

Apr 22 20:03:53.808: %SPANTREE-2-BLOCK_BPDUGUARD: Received BPDU on port Fa0/10 with BPDU Guard enabled. Disabling port.
Apr 22 20:03:53.808: %PM-4-ERR_DISABLE: bpduguard error detected on Fa0/10, putting Fa0/10 in err-disable state
Apr 22 20:03:53.816: %SPANTREE-2-BLOCK_BPDUGUARD: Received BPDU on port Fa0/10 with BPDU Guard enabled. Disabling port.

This is expected:

interface FastEthernet0/10
switchport mode access
spanning-tree portfast
spanning-tree bpduguard enable

I looked at what Spanning Tree thought was going on:

Cat3550#sh span

VLAN0001
Spanning tree enabled protocol ieee
Root ID    Priority    32769
Address     000b.465b.0000
This bridge is the root
Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
Address     000b.465b.0000
Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
Aging Time 300

Interface           Role Sts Cost      Prio.Nbr Type
——————- —- — ——— ——– ——————————–
Fa0/12              Desg BLK 19        128.12   P2p self-looped
Fa0/18              Desg FWD 19        128.18   P2p Edge
Fa0/20              Desg FWD 19        128.20   P2p Edge
Fa0/22              Desg FWD 19        128.22   P2p
Fa0/24              Desg FWD 19        128.24   P2p
Gi0/1               Desg FWD 4         128.25   P2p

(Obviously I should get round to moving out of VLAN1…)

So the port is seen as self-looped and STP Blocked. Why?

I switched off all the other HomePlug devices. No difference.

At this point, I figured that I must have some sort of weird interference from outside given that on this HomePlug network I was using the default HomePlug network name (yes I know I should have changed this) and I knew that the signal can leak outside the house to the sub-station (another hair-pulling debug session from a few weeks ago I found when bridging two default named HomePlug networking using UTP).

I resolved to change the HomePlug network name on all the original HomePlug network devices. And that is when I found the issue. Whilst changing the first HomePlug, I noticed a remote HomePlug device and realised I had missed the multi-port HomePlug unit which connects to the Xbox. The Xbox had been removed and my middle son had (uncharacteristically) tidied up the end of the Ethernet cable plugged into the Xbox by putting it one of the other spare Ethernet ports on that device!

That was the cause of the problem and the reason why loopback packets were seen. Removing the cable fixed the issue – the Catalyst switch port LED went green, Spanning Tree saw the loop go away,

Cat3550#sh span

VLAN0001
Spanning tree enabled protocol ieee
Root ID    Priority    32769
Address     000b.465b.0000
This bridge is the root
Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
Address     000b.465b.0000
Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
Aging Time 300

Interface           Role Sts Cost      Prio.Nbr Type
——————- —- — ——— ——– ——————————–
Fa0/12              Desg LIS 19        128.12   P2p
Fa0/18              Desg FWD 19        128.18   P2p Edge
Fa0/20              Desg FWD 19        128.20   P2p Edge
Fa0/22              Desg FWD 19        128.22   P2p
Fa0/24              Desg FWD 19        128.24   P2p
Gi0/1               Desg FWD 4         128.25   P2p

and the interface started to pass packets properly. (LIS means listening, then the port goes to FWD if there are no STP issues).

No relevant causes for the ‘self-looped’ message appear on Google so perhaps this will help someone, and I will make sure I explain why looping back an Ethernet interface is bad to my son.

Easily done, not always easy to diagnose. At least I have a reason to change that default HomePlug network name now!

Update (Jan 2013):
I did change the HomePlug network name.

I also had a recurrence of a self-looped network seen by Spanning Tree on the port connected to the HomePlug network. This occurred after I replaced a Cisco Catalyst 2950 that was connecting my two (differently named) HomePlug networks with a Cisco Catalyst 3550. So the network now looked like

Cisco 3550_1 Fa0/12 >> HomePlug Network 1 >>> Cisco 3550_2 >> HomePlug Network 2

I also had a couple of VLANs on the intermediate 3550_2 and perhaps something went wobbly because I saw intermittent connectivity when looking at it from the intermediate 3550_2. Debugging the issue from 3550_1 showed the port was going up and down (because I have auto-recovery). I switched off all the HomePlug except the one connected to 3550_1 and the one connected to 3550_2. Using the HomePlug utility on a laptop connected directot to 3550_1’s HomePlug showed 3 remote devices. One was the HomePlug connected to 3550_2, one was something unknown (but it looked like a Cisco MAC address to me), and one was a device with ffffffff as the MAC address! Only HomePlug should be seen of course. Powering down the 3550_2 HomePlug cured the issue so it must have tickled a bug – possibly because the 3550_2 was sending Switchport Mode Dynamic Desirable packets (and thereby saying it was happy to become a trunk port)?

Cisco Network Assistant and “Could not create Java machine”

The Cisco Network Assistant (CNA) is software that provides a more richly featured GUI interface to a low-medium range IOS Cisco switch. You can use the CLI of course, and there is a web interface (if you have not upgraded your switch and blown away the HTML files).

I was running version 5.4 though I had not used it for ages. When I had tried to start it I got a “Could not create a Java machine” error from my XP machine

Image

I figured I had installed so much software in between that I had broken something or it didn’t like the newer versions of Java, or just that I really shouldn’t still be running XP.

So I decided to upgrade it to 5.8.2 but afterwards I got the same error.

Poking around, I saw that the real issue was that I didn’t have the memory free that Java wanted when it started the application. It greedily expects to grab a Gig for itself! However you can change this quite easily using a text editor by modifying the entry in the properties file for CNA (normal risks apply when editing system startup files – only do this if you have a clue).

The file exists in C:\Program Files\Cisco Systems\Cisco Network Assistant\startup\startup.properties

Image

Change the value of
JVM_MAXIMUM_HEAP=1024m
to
JVM_MAXIMUM_HEAP=512m

as shown here:

Image

You should now be able to start the application. On my small network, I didn’t see an issue by reducing the startup memory size.

Debugging Mulberry Mail

There are times when you want to see what is happening to your mail at a protocol level, and if you use the Mulberry mail client then you are in luck because this excellent client will let you do that – if you can remember what the debug sequence is.

For the record on Windows it is ‘ALT F4 F4ALT F F’ which lets you select what you want to debug and the log files will then be written into the /logs sub folder in the Mulberry installation directory.

(Corrected the key sequence to ALT F F as of course ALT F4 closes Mulberry, sorry, very very sorry!)

On OSX it is ‘ALT Preferences’ and the log file is stored in the Mulberry application folder by default. You can get to this by opening the Applications folder (right click on the Applications folder icon on the dock bar), then selecting Mulberry and right clicking and selecting ‘Show Package Contents’.

 

There is information on the Mulberry wiki at http://trac.mulberrymail.com/mulberry/wiki/logging

Recovering files from a local Crashplan archive

Crashplan is an excellent cross-platform backup to the cloud utility. Here I use it to backup to a local NAS and to the Crashplan Central ‘in the cloud’ storage site.

Recently I had to recover (again) all the files for a laptop which had had a new hard disk fitted. It had the operating system and the crashplan software installed.

During recovery there seemed to be no way to recover the files from the local archive. This would have been faster than pulling everything back down from the Internet. I logged a support case with Crashplan and there is a way to do this (which I have suggested they make explicitly clear).

The archive can be attached and recovered from using the following procedure:

1. Open the CrashPlan desktop and go to Backup > Inbound.
(If you don’t see the Inbound section, go to Settings > General > Inbound backup from other computers > Configure. Click Accept Inbound Backups.)

2. Select Attach a backup archive
(If you already have another inbound device listed, click the triangle icon in the top-right of the window to open the menu.)

3. Navigate to and select the archive folder.
The archive folder will have a long series of numbers as the file name: e.g., 948212309528060501.

4. Click Ok.

5. Navigate back to the Restore tab and choose “This Computer” as the destination.

6. The archive on the NAS should be attached and you should be able to browse the files in that archive.

You can also use this procedure to restore files from the NAS archive belonging to a different computer then the one you are on (though of course you can also do that from the Crashplan software by accessing Crashplan Central, or from a web interface, or from a mobile interface).