14 August 2012

Cisco ASA VPN Troubleshooting Guide


Quick overview of IPSEC
It is important to understand how IPSEC works in order to understand how to troubleshoot a VPN connection. This is a quick overview of IPSEC and is by no means a complete detailed guide.

IPSEC is a suite of protocols, defined in RFC 2401, that is used to protect information as it travels from one private network to another private network over a public network.

IPSEC consists of Security Protocols (AH and ESP), Key Management (ISAKMP, IKE, and SKEME), and Algorithms (3DES, AES256, etc).

ISAKMP defines the procedures and packet formats used to establish, negotiate, and modify Security Associations. ISAKMP communicates over UDP 500.

Security Protocols consist of AH (Authentication Header) and ESP (Encapsulating Security Payload). AH communicates over IP 51 and provides data authentication, integrity, and replay protection (for man in the middle attacks), but does not provide confidentiality. It is important to understand that AH encapsulates the IP packet but does not encrypt it.
ESP communicates over IP 50 and provides the same service as AH in addition to providing data confidentiality by encrypting the original payload and encapsulating the packet.

SA’s (Security Associations):
In order to have an IPSEC conversation, you first need a security association. Each device must agree on the policies or rules of the conversation by negotiating these policies with their potential peers. The SA represents a unidirectional instance of a security policy for a given connection.

Main mode IPSEC packet exchange:
--Initiator--- ---Responder---
----------pk#1—Policy Proposal------>
<-------pk#2---Policy Accept/Reject-- ----------pk#3---DH Exchange-------->
<-------pk#4---DH Exchange---------- ----------pk#5---ID/Hash------------->
<------pk#6---ID/Hash--------------->

Packet handling order:

Step 1 Access lists applied to an interface and crypto map are used by Cisco IOS software to select interesting traffic to be encrypted.
Step 2 Cisco IOS software checks to see if IPSec SAs have been established.
Step 3 If the SA has already been established by manual configuration using the crypto ipsec transform-set and crypto map commands or has been previously set up by IKE, the packet is encrypted based on the policy specified in the crypto map and is transmitted out of the interface.
Step 4 If the SA has not been established, Cisco IOS software checks to see if an IKE SA has been configured and set up.
Step 5 If the IKE SA has been set up, the IKE SA governs negotiation of the IPSec SA as specified in the IKE policy configured by the crypto isakmp policy command, the packet is encrypted by IPSec, and it is transmitted.
Step 6 If the IKE SA has not been set up, Cisco IOS software checks to see if certification authority (CA) has been configured to establish an IKE policy.
Step 7 If CA authentication is configured with the various crypto ca commands, the router uses public and private keys previously configured, obtains the CA's public certificate, gets a certificate for its own public key, and then uses the key to negotiate an IKE SA, which in turn is used to establish an IPSec SA to encrypt and transmit the packet.


Configuring Phase 1:
The first 2 octets of IPs have been replaced with "y.y."
Phase I is not configured on a per connection basis. When a Phase I connection is being established, configured ISAKMP policies will be tried one at a time until a match is found.

Example of an ISAKMP policy:
#isakmp policy 20 authentication pre-share
#isakmp policy 20 encryption 3des
#isakmp policy 20 hash md5
#isakmp policy 20 group 2
#isakmp policy 20 lifetime 43200


Troubleshooting Phase I:

Check the syslogs

Show run isakmp 
This will show the isakmp policies for all VPN connections. To view a specific ISAKMP policy type show run isakmp | grep 
show vpn-sessiondb detail l2l

Show crypto isakmp sa detail – This command will display the state of Phase I of the IPSEC tunnel. A state of MM_Active indicates that Phase I was successfully completed. If Phase I does not complete, refer to the table below to find out exactly what state the Phase I connection is currently in. This will give you an indication of where the problem has occurred. More specific information can be found by running a debug(discussed later).

State Description
OAK_MM_No_STATE This is the initial state of Phase I. If you see Phase I
In this state for longer than a few seconds, this is an
indication that a failure of tunnel establishment for
Phase I has occurred.

OAK_MM_SA_SETUP The peers have agreed on parameters for the ISAKMP
SA. Phase I will be in this state after packet 1 and packet 2 exchange of the Main Mode negotiation (see above).


MM_WAIT_MSG The firewall is waiting on the remote end device to respond with DH and public key.


OAK_MM_KEY_EXCH The peers have exchanged DH public keys and have generated a shared secret.

OAK_MM_KEY_AUTH The ISAKMP SA has been authenticated.


The debug crypto isakmp 5 command will display real time information on every step of the Phase I connection. Debug level 5 should be sufficient for most troubleshooting however level 7 provides more detailed information if necessary.
Please note that you cannot limit the debug output to a specific tunnel.

IKMP_NO_ERROR_NO_TRANS indicates a matching transform set was not found

No Proposal Chosen=isakmp policy mismatch


syslog sample of a completed connection:
Mar 10 2008 18:47:05: %PIX-3-713119: Group = y.y.41.250, IP = y.y..41.250, PHASE 1 COMPLETED

Sample Debug output:
The following shows the initiation of the first packet for an IPSEC tunnel.
58534 02/27/2004 07:42:38.600 SEV=4 IKE/41 RPT=8619 y.y.11.49
IKE Initiator: New Phase 1, Intf 2, IKE Peer

The following indicates that the IKE Phase I policy was accepted by the remote gateway.
58534 02/27/2004 07:42:38.600 IP = y.y.11.49, Oakley proposal is acceptable

This indicates Phase I has completed.
58534 02/27/2004 07:42:38.600 Group= y.y.11.49, IP=y.y.11.49, Oakley begin quick mode

The following indicates that the remote gateway has indicated that none of the policies are acceptable.
5|Oct 02 2006 09:41:41|713904: IP = y.y.138.12, Received an un-encrypted NO_PROPOSAL_CHOSEN notify message, dropping

To clear the Security Associations related to Phase 1, use the clear crypto isakmp command. This will clear ALL of the SA’s currently built on this firewall.

To confirm that the IPSEC packets are reaching the firewall, a capture can be created for all UDP 500 traffic.
First create an access-list for the traffic you would like to capture.
Access-list capture1 permit udp any any eq 500

Next create a capture.
Capture cap1 access-list capture1 interface outside

Next display the results of the capture.
Show capture cap1 detail

ciscoasa#show capture cap1 detail
1: 13:04:06.284897 192.168.1.50 > 192.168.1.1: UDP:500

View capture on web
https://capture/pcap/cap1

View pre-shared keys:
more system:running-config


Configuring Phase 2:
A transform set combines encryption method and authentication method. During the IPSec security association negotiation with ISAKMP, the peers agree to use a particular transform set to protect a particular data flow. The transform set must be the same for both peers.
You can create multiple transform sets, and then specify one or more of these transform sets in a crypto map entry.
You can view previously created transform sets by typing the show crypto ipsec transform-set command. If the desired transform set has not been previously defined, the crypto ipsec transform-set command is used to create it.

Example:
#(config)crypto ipsec transform-set 3desmd5 esp-3des esp-md5-hmac

An access-list is used to define the “interesting traffic” or the traffic that should be encrypted and allowed through the VPN Tunnel. The access-list should always be defined from local to remote. The subnet sizes need to match on the remote gateway.

Example:
#(config) access-list tunnel1 extended permit ip y.y..191.0 255.255.255.0 host y.y..155.12

If port filtering is being used, and traffic is being initiated from the remote side, the destination port of the remote host must be the source port of the local matching acl.

A tunnel group is used to identify specific connection parameters and the definition of a group policy. The default tunnel groups are DefaultRAGroup (used for Remote Access tunnels) and DefaultL2Lgroup(used for IPSEc Lan-to-Lan tunnels).

Example:
#(config)tunnel-group y.y.155.1 type IPsec_l2l
#(config)tunnel-group y.y.155.1 ipsec-attributes
#(config-attributes) pre-shared-key abc123


The crypto map ties together several components that define the VPN tunnel. This is where the peer defined in the tunnel-group command is tied to the access-list and transform-set. The crypto map must be assigned a unique map id #. To view the previously used crypto map id numbers run the show ru crypto command.

Example:
#(config)crypto map mymap 10 match address tunnel1
#(config)crypto map mymap 10 set peer y.y,155.1
#(config)crypto map mymap 10 set transform-set 3desmd5

Nat considerations:
If a local address is going to be natted outbound, the crypto acl should use the outside ip address.

Troubleshooting Phase II:
Check syslogs

Show crypto ipsec sa- This command shows the output of the IPSEC SA’s. The SA will include the ip address of the local and remote endpoints, encryption domains (interesting traffic), transform set (what encryption and hash is being used), key lifetime, and # of packet encrypt/decrypts.
debug crypto engine—Displays the traffic that is encrypted.

Example of an IPSEC SA:
This shows the crypto map used for this connection.
Crypto map tag: vpn_map, seq num: 130, local addr: x.x.160.45

The following line shows the crypto acl that includes the traffic to be protected.
access-list VPN-CIDS704976 permit ip x.x.190.0 255.255.254.0 host 10.2 5.4.80
local ident (addr/mask/prot/port): (x.x.190.0/255.255.254.0/0/0)
remote ident (addr/mask/prot/port): (10.25.4.80/255.255.255.255/0/0)
current_peer: y.y.227.136

Encrypts indicate that this side is encrypting and sending traffic. Decrypts indicates that the other side is sending traffic.
#pkts encaps: 5, #pkts encrypt: 5, #pkts digest: 5
#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 5, #pkts comp failed: 0, #pkts decomp failed: 0
#send errors: 0, #recv errors: 0

This lists the local and remote endpoints.
local crypto endpt.: x.x.160.45, remote crypto endpt.: y.y.227.136

path mtu 1500, ipsec overhead 74, media mtu 1500
current outbound spi: 2AFEA5C7

There is a separate sa for inbound and outbound.
inbound esp sas:
spi: 0x9D111D2A (2635144490)
transform: esp-aes-256 esp-sha-hmac none
in use settings ={L2L, Tunnel, PFS Group 5, }
slot: 0, conn_id: 317225, crypto-map: vpn_map
sa timing: remaining key lifetime (kB/sec): (4275000/28789)
IV size: 16 bytes
replay detection support: Y
outbound esp sas:
spi: 0x2AFEA5C7 (721331655)
transform: esp-aes-256 esp-sha-hmac none
in use settings ={L2L, Tunnel, PFS Group 5, }
slot: 0, conn_id: 317225, crypto-map: vpn_map
sa timing: remaining key lifetime (kB/sec): (4274999/28789)
IV size: 16 bytes
replay detection support: Y

Clear crypto ipsec sa peer will clear the Phase 2 SA’s for a given peer.


debug crypto ipsec—Displays the IPSec negotiations of phase 2.

No Valid SA/ Identity mismatch – Transform set or crypto acl

Sample Debug output:
The following shows that the tunnel group configuration was found.
Oct 26 15:42:43 [IKEv1]: IP =y.y.205.92, Connection landed on tunnel_group y.y,.205.92

Sample syslog errors:

This shows interesting traffic ACL getting exchanged.
1754 11/29/2001 16:20:18.500 SEV=7 IKEDBG/0 RPT=546 y.y.205.92
Transmitting Proxy Id:
Remote host: 192.168.1.1 Protocol 0 Port 0
Local host: 10.64.10.9 Protocol 0 Port 0

Completion of Phase II.
1949 11/29/2001 16:20:18.540 SEV=4 IKE/49 RPT=3 y.y.205.92
Security negotiation complete
Responder, Inbound SPI = 0x11a56495, Outbound SPI = 0xb17718a5

Mar 10 2008 18:47:05: %PIX-5-713120: Group = y.y.41.250, IP = y.y.41.250, PHASE 2 COMPLETED (msgid=0f78e513)

Pre-shared key mismatch.
1754 11/29/2001 16:20:18.500 Group = 172.16.172.63, IP = 172.16.172.63, Received an un-encrypted PAYLOAD_MALFORMED notify message, dropping.

Pre-shared key mismatch reported by the report peer(receiving peer):

713903: Group = 172.1.12.1, IP = 172.1.12.1 ERROR. peer has indicated thatsomething is wrong with our message. This could indicate a pre-shared key mismatch.

Transform-set mismatch.
1754 11/29/2001 16:20:18.500 Group = 172.16.172.63, IP = 172.16.172.63, Received non-routine Notify message: No Proposal Chosen

Transform-set mismatch on remote peer(receiving peer):

713904” IP = 10.51.16.1, Received encrypted packet with no matching SA, dropping
713048: IP = 10.51.16.1 Error processing payload. Payload ID 1

The following indicates that the remote gateway is not finding matching interesting traffic.
1754 11/29/2001 16:20:18.500 Group = y.y.172.63, IP = y.y.172.63, Received non-routing Notify message: Invalid ID info (18)

The following indicates that the local gateway is not finding matching interesting traffic.
1754 11/29/2001 16:20:18.500 Group =y.y.172.63, IP = y.y.172.63, Static Crypto Map check, map = mymap, seq = 10, ACL does not match proxy IDs src:192.168.1.0 dst:192.168.2.0

PFS mismatch:

713068: Group – 172.1.12.1, Received non-rouing Notify message; No Proposal chosen (14)

PFS turned on on the remote peer. Local peer reports the following:
713902; Group = 10.51.16.1. QM FSM error (p2 struct &0x296fde8, mess id 0x518e80d)!
QM FSM is a generic message indicating that the phase II connection was rejected by the remote peer.

This indicated that the remote peer is natting:
%ASA-4-402116: IPSEC: Received an ESP packet (SPI= 0x72DEC2AA, sequence number= 0x41) from y.y.28.178 (user= y.y.28.178) to y.y.83.194. The decapsulated inner packet doesn't match the negotiated policy in the SA. The packet specifies its destination as y.y.83.194, its source as y.y.28.178, and its protocol as 1. The SA specifies its local proxy as y.y.10.16/255.255.255.240/0/0 and its remote_proxy as y.y.63.0/255.255.255.0/0/0.

When reverse route is turned on:


Jan 26 2009 18:15:07: %ASA-6-713211: Group =y.y43.160, IP = y.y.43.160, Adding static route for L2L peer coming in on a dynamic map. address: 192.168.8.5, mask: 255.255.255.255
Jan 26 2009 18:57:54: %ASA-6-713213: Group = y.y.43.160, IP =y.y43.160, Deleting static route for L2L peer that came in on a dynamic map. address: 192.168.8.5, mask: 255.255.255.255

35 comments:

  1. A great document. Thank you!

    ReplyDelete
  2. Agree with guys, it's very useful, thx.
    wish y all the best.

    Regards
    Toby, virtual data room comparison

    ReplyDelete
  3. Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work! pure vpn

    ReplyDelete
  4. Really appreciate this, thanks.

    ReplyDelete
  5. I high appreciate this post. It’s hard to find the good from the bad sometimes, but I think you’ve nailed it! would you mind updating your blog with more information? vpn services

    ReplyDelete
  6. This a good way to appreciate the teacher as they put their efforts to train students. UK dissertation Writers appreciates the teachers.  allerta privacy

    ReplyDelete
  7. I really like your blog. Great article. It's most evident, people should learn before they are able to privacidadenlared.es

    ReplyDelete
  8. I wanted to thank you for this great read!! I definitely enjoying every little bit of it I have you bookmarked to check out new stuff you post. https://privatnostonline.com

    ReplyDelete
  9.  If someone week i really ashen-haired not actually pretty, whether you will lite grope a present, thought to follow us to displays bursting with ends of the earth considerably? Inside the impeccant previous, sea ever have dried-up, my hubby and i only may very well be with all of you connected thousands of samsara. internetprivatsphare.ch

    ReplyDelete
  10. I know your expertise on this. I must say we should have an online discussion on this. Writing only comments will close the discussion straight away! And will restrict the benefits from this information. vpn

    ReplyDelete
  11. America Gun Ban Home, Syndicated by Countrymen Your News Source Worldwide. Taking the stink out of Gun Control and being a Patriot.  diebestenvpn.at

    ReplyDelete
  12. I think this is an informative post and it is very useful and knowledgeable. therefore, I would like to thank you for the efforts you have made in writing this article. https://privacyonline.fi/mika-on-vpn/

    ReplyDelete
  13. This comment has been removed by the author.

    ReplyDelete
  14. This article gives the light in which we can watch the truth. This is exceptionally decent one and gives indepth data. A debt of gratitude is in order for this decent article.  vpnveteran

    ReplyDelete
  15. I thought that was not real, but I tried high a safe online casino and won by what was pleasantly surprised

    ReplyDelete
  16. In expansive organizations with tremendous geological spread speaking with all representatives had truly presented impressive test to HR experts. gizlilikveguvenlik

    ReplyDelete
  17. The most inspiring stuff commonly is probably the most dull or boring concern. a single rooster may not be in the same woody plant strip very long, and also the pets within the pine is a kind of weight, always want to consult the next stars, the particular fowl because the plants are really hesitant to facial expression an equal beautiful places day by day. bezoek website

    ReplyDelete
  18. This comment has been removed by the author.

    ReplyDelete
  19. For example, some product firms have discovered that they can beaten the tight nearby market for programming engineers by sending undertakings to India or different countries where the wages are much lower. gizlilikveguvenlik

    ReplyDelete
  20. This is truly a decent and useful, containing all data furthermore greatly affects the new innovation. A debt of gratitude is in order for sharing it  internetetsecurite

    ReplyDelete
  21. VPN or Virtual Private Network is an innovation that interfaces two individual private systems to an open system, utilizing the web as a medium. The way toward setting up a Virtual Private Network is called Virtual Private Networking. vpn.surf/blog/buy-vpn-with-bitcoin-cash

    ReplyDelete
  22. Thanks you very much for sharing these links. Will definitely check this out.. nord vpn free trial

    ReplyDelete
  23. I definitely enjoying every little bit of it. It is a great website and nice share. I want to thank you. Good job! You guys do a great blog, and have some great contents. Keep up the good work. download super vpn for pc

    ReplyDelete
  24. Pretty good post. I have just stumbled upon your blog and enjoyed reading your blog posts very much. I am looking for new posts to get more precious info. Big thanks for the useful info. 13377x

    ReplyDelete