Open Source IPSec

It took me many years to secure my time to write down a blog. Recently, one of my friends asked me to help him resolve an IPsec issue with Fortigate Device. In fact, I do not have much experience with this so I reluctant to help. I just want to list down all knowledge I learned when investigating this issue.


RFC standards

  • Group of standard
  • Relationship


  • Packet format
  • How to parse packets
  • IKEv1 protocol and message exchange
  • IKEv2 protocol and message exchange (T.B.D)

Strongswan Implementation

  • Strongswan architecture
  • Strongswan implementation for IKEv1
  • Strongswan implementation for IKEv2
  • Talking with Linux Kernel IPsec Framework XFRM
  • How to debug

RFC standards

In general, IP security is one of the complicated protocol suites and went through many updated version. What I focused on is that site-to-site IPsec VPN. In this mode, ESP tunneling is selected so that I will focus on these standards:

  • RFC4301 provides a Security Architecture for Internet Protocol
  • RFC2407
  • RFC2408 Internet Security Association and Key Management Protocol (ISAKMP)
  • RFC4306 The Internet Key Exchange v2 (Obsolete rfc2409 IKEv1). In fact, the newest one is RFC7296 published in October 2014
  • RFC4303 Encapsulation Security Protocol
  • RFC3948 UDP Encapsulation of IPsec ESP Packets.

In general, the site to site IPsec VPN require these protocol

  • The internet key exchange (version 1 or version 2): which help to set up the security associations (SA) between parties such as negotiate the security parameters (DH group), authenticate peers (pre-shared key), auto rekeying …
  • Encryption protocol (ESP or AH)

The following picture described how normal IP packets are encoded to IPsec packets.

This assumed that UDP encapsulation (RFC3948) is used so that we have UDP Header field between New IP Hdr and ESP Hdr. This is applied whenever any sites of IPsec tunnel connection is behind the NAT which is mostly the case. If it is not the case, UDP Hdr is stripped off.


  • We must differentiate between IKE SA vs IPsec SA.
  • In general, IKE SA is created by IKEv1 or IKEv2. Based on that, IPsec SAs are created using the secure connection created by IKE.
  • In Strongswan, it called these IPsec SAs as Child SA.


ISAKMP is a protocol to allow IPsec peers to exchange and negotiate the security parameters. ISAKMP is split into two phases: phase 1 and phase 2. At phase 1, two ISAKMP peers establish a secure, authenticated channel to communicate which is called ISAKMP SA. At phase 2, IPsec SA is negotiated and established.

Here is the format of ISAKMP frame. The header has the following format:


Each payload is composed as follows

Phase 1

Main Mode


Aggressive Mode


Phase 2

When a security association (SA) is initially established, one side assumes the role of initiator and the other the role of the responder. Once the SA is established, both the original initiator and responder can initiate a phase 2 negotiation with the peer entity. Thus, ISAKMP SAs are bidirectional in nature.


TODO: What is ESP Header, ESP Auth?

TODO: How new IP Header is formed?

Both IKE packets and IPsec packets use ESP protocol to encode the data. The difference is that IKE packets have ESP marker equal to 00 00 00 00.


In general, Strongwan is written in C language with OOP in mind. The public header files will have a structure including public APIs only. The private structure will include the public structure which declares the method and hides all internal data member in the private structure. For example, the public identification_t structure is a public interface with public methods.

struct identification_t {
* Get the encoding of this id, to send over
* the network.
* Result points to internal data, do not free.
* @return a chunk containing the encoded bytes
chunk_t (*get_encoding) (identification_t *this);

* Get the type of this identification.
* @return id_type_t
id_type_t (*get_type) (identification_t *this);

The private implementation is as following

struct private_identification_t {
* Public interface.
identification_t public;

* Encoded representation of this ID.
chunk_t encoded;

* Type of this ID.
id_type_t type;

The following diagram depicts some small parts of Strongswan.

TODO1: How the egress packets are processed

TODO2: How the ingress packets are processed



This entry was posted in Uncategorized and tagged . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.