IPSec VPN How does it really work Yasushi

IPSec VPN: How does it really work? Yasushi Kono (Computer. Links Frankfurt)

Before the Agenda l My intention of this presentation: I know that many people do not have any clue what‘ s happening while establishing Security Associations prior to create VPN tunnels. This topic is quite complex. I want you to get started into this topic taking away a bit of its complexity.

Agenda l Introductory Information on IPSec VPN l Why Diffie-Hellman Algorithm? l IKE SA in Main Mode l IPSec SA in Quick Mode l Some Troubleshooting Tips

l Introductory Information on IPSec VPN

Before establishing a Site-to-Site VPN connection, both gateways must agree upon parameters for encrypting communication. This negotiation process is divided into two phases: Phase 1 and Phase 2. 2 In Phase 1 a IKE Security Association (IKE SA) will be created. Those parameters are therefore:

Before establishing a Site-to-Site VPN connection, both gateways must agree upon parameters for encrypting communication. This negotiation process is divided into two phases: Phase 1 and Phase 2. 2 In Phase 1 a IKE Security Association (IKE SA) will be created. Those parameters are therefore: 1. ) Encryption Algorithm (3 DES, AES-128, AES-256, …) to ensure privacy

Before establishing a Site-to-Site VPN connection, both gateways must agree upon parameters for encrypting communication. This negotiation process is divided into two phases: Phase 1 and Phase 2. 2 In Phase 1 a IKE Security Association (IKE SA) will be created. Those parameters are therefore: 1. ) Encryption Algorithm (3 DES, AES-128, AES-256, …) to ensure privacy 2. ) Hash Algorithm (SHA-1 or MD 5) to ensure data integrity

Before establishing a Site-to-Site VPN connection, both gateways must agree upon parameters for encrypting communication. This negotiation process is divided into two phases: Phase 1 and Phase 2. 2 In Phase 1 a IKE Security Association (IKE SA) will be created. Those parameters are therefore: 1. ) Encryption Algorithm (3 DES, AES-128, AES-256, …) to ensure privacy 2. ) Hash Algorithm (SHA-1 or MD 5) to ensure data integrity 3. ) Diffie-Hellman Group (1, 2, 5, or 14)

Before establishing a Site-to-Site VPN connection, both gateways must agree upon parameters for encrypting communication. This negotiation process is divided into two phases: Phase 1 and Phase 2. 2 In Phase 1 a IKE Security Association (IKE SA) will be created. Those parameters are therefore: 1. ) Encryption Algorithm (3 DES, AES-128, AES-256, …) to ensure privacy 2. ) Hash Algorithm (SHA-1 or MD 5) to ensure data integrity 3. ) Diffie-Hellman Group (1, 2, 5, or 14) 4. ) Method of mutual authentication (Preshared Key or Certificate)

You have the choice between two different modes in Phase 1: • Main Mode • Aggressive Mode • Main Mode consists of a 6 packet negotiation • In Aggressive Mode only three packets are exchanged between both VPN gateways.

Before beginning to analyze the information exchanged by the gateways, let‘s have a look at the Diffie. Hellman Algorithm: Why is Diffie-Hellman necessary?

Agenda l Introductory Information on IPSec VPN l Why Diffie-Hellman Algorithm? l IKE SA in Main Mode l IPSec SA in Quick Mode l Some Troubleshooting Tips

Usually, payloads are encrypted symmetrically by means of symmetric encryption algorithms, like 3 DES or AES. The problem with symmetric encryption is exchanging the keys over the Internet while preventing them from falling into the wrong hands. One answer is asymmetric encryption.

So, when asymmetric encryption is addressing problems with symmetric encryption, why is latter technology still necessary?

The answer is: Performance!

The Diffie-Hellman Algorithm is not an Encryption Algorithm rather than a Key Agreement Protocol.

To understand the Diffie-Hellman Algorithm, you have to have basic knowledge on the Modulo operation. The Modulo operation finds the remainder of division of one number by another. Given two numbers, a and b, n = a mod b is the remainder on division of a by b.

Examples: 30 mod 16 = 14 114 mod 100 = 14 8 mod 3 = 2 Isn‘t that simple?

Diffie-Hellman: Initiator takes a prime number p and an integer a with 1 < a < p and a secret integer x with: X = ax mod p X, a, p: public parameters, x: secret parameter.

Initiator sends X, a, and p to the Responder. The Responder takes a secret integer y and computes: Y = ay mod p The number Y, which is public will be sent back to the Initiator.

Initiator and Responder are computing their common encryption key: • Kx = Yx mod p • = (ay mod p)x mod p • = (ax mod p)y mod p • = Xy mod p = Ky • Hence Kx = Ky

Now, that we have the basic understanding of the Diffie. Hellman Algorithm, we can now fully understand IKE SA in Main Mode.

Agenda l Introductory Information on IPSec VPN l Why Diffie-Hellman Algorithm? l IKE SA in Main Mode l IPSec SA in Quick Mode l Some Troubleshooting Tips

As all of you might already know, IKE SA in Main Mode consists of 6 packets

First of all, the Initiator sends the IKE SA Parameters to be negotiated upon to the Responder: IKE SA: 3 DES or AES-128? SHA-1 or MD 5? DH-Group 5 or 14? Certificate or Preshared Key?

Initiator Packet Number 1: Responder

The Responder sends back the parameters to be used in common to the Initiator: IKE SA: AES-128! SHA-1! DH-Group 14! Preshared Key!

Initiator Packet Number 1: Packet Number 2: Responder

Then, the Initiator sends the public Diffie. Hellman Parameters and a random number, which is called „Nonce“:

Initiator Packet Number 3: Responder

Then, the Responder sends back its own public Diffie-Hellman Parameters and its own random number („Nonce“):

Initiator Packet Number 3: Packet Number 4: Responder

Both parties know from each other, which parameters and encryption key to use in common. The Initiator then • builds the hash of the Preshared Key • encrypts the Nonce of the Responder and the Hash of the Preshared Key

Initiator Packet Number 5: Packet Number 6: Responder

What happens next? Each of the Gateways are receiving their own Nonces encrypted by other parties. The next step is to decrypt the encrypted Nonces to verify the identity of the communicating gateways.

After receiving the sixth packet, you will get the following message in Smart. View Tracker: IKE SA: Main Mode completion

Agenda l Introductory Information on IPSec VPN l Why Diffie-Hellman Algorithm? l IKE SA in Main Mode l IPSec SA in Quick Mode l Some Troubleshooting Tips

Why is Quick Mode necessary? To establish the IPSec SA!

Quick Mode to establish an IPSec SA consists of 3 packets.

You have to negotiate upon: • • The Encryption algorithm The Hash Algorithm The IPSec Protocol (ESP, AH) If PFS is to be used or not (if yes, the DH-Group is to be determined again)

One word regarding IPSec Protocols: Authentication Header (AH) provides: • Data Integrity Checking • Replay Protection

Encapsulating Security Payload (ESP) provides: • • • Payload Encryption Data Integrity Checking Replay Protection

Why is it necessary to agree upon the • Encryption Algorithm • Hash Algorithm • Eventually DH-Group again?

Because the parameters negotiated there is for applying to the payload of the packets and not to the identity of the other gateway!

The 3 Packets in Quick Mode

The Message you will get after succesful IPSec SA Negotiation is: IKE SA: Quick Mode completion

Agenda l Introductory Information on IPSec VPN l Why Diffie-Hellman Algorithm? l IKE SA in Main Mode l IPSec SA in Quick Mode l Some Troubleshooting Tips

l Some Troubleshooting Methods: Mostly, you will get the right information with Smart. View Tracker There, the most frequent error messages are: • • • IKE SA: No proposal chosen Encryption failure: No valid SA INVALID_ID_INFORMATION

l Some Troubleshooting Methods: Received Notification from Peer: Malformed Payload Any solution?

l One standard method for Debugging IKE/IPSec is vpn debug ikeon to generate the output file ike. elg.

l But, without any knowledge of theory of IPSec, is it useful to analyze the ike. elg file?

For more troubleshooting IKE/IPSec in a Check Point Environment, attend the next presentation: 305: Troubleshooting in the Check Point Environment – Part II By Tobias Lachmann

Any Questions?

Thanks a lot for your attention! Should you have questions: yk@computerlinks. de