xTom - What Is IPSec, and How Does It Secure Your Network Traffic?

At its core, the Internet relies on IP (Internet Protocol) addresses for routing data between countless connected devices.

However, the data itself—whether it’s sensitive corporate information, personal login credentials, or even your League of Legends chat logs (everyone knows you don't want those to get out)—does not come with inherent protection.

This is where IPSec comes in handy.

How IPSec works

IPSec isn’t a single protocol, but rather a suite that includes several key processes and components before any data is securely transmitted:

Key exchange: Communicating parties negotiate cryptographic keys and algorithms. Keys act like secret codes, ensuring that only authorized endpoints can encrypt and decrypt the transmitted data.
Packet headers and trailers: Data travels across the Internet in packets. IPSec adds headers (and sometimes trailers) that contain authentication and encryption details. Two primary protocols, AH (Authentication Header) and ESP (Encapsulating Security Payload), handle these tasks. AH ensures the packets come from a trusted source and haven’t been tampered with, while ESP provides encryption so that only approved recipients can read the payload.
Authentication and encryption: IPSec uses the agreed-upon keys to authenticate each packet’s origin and verify that it’s unchanged. At the same time, it encrypts the payload so that if anyone intercepts the traffic, it appears as random, unreadable data.
Transmission over UDP: IPSec often uses UDP as a transport protocol to help encrypted packets pass through firewalls and NAT devices. This differs from standard IP traffic, which commonly uses TCP. After the packets are sent, the receiving end decrypts and authenticates them before delivering the data to the intended application.

IPSec can operate in two distinct modes depending on the use case:

Transport Mode: In transport mode, IPSec encrypts only the packet’s payload, leaving the original IP header intact. This mode is typically used for end-to-end communication between two hosts, such as a client and a server. Because the original header remains visible, intermediary devices can still route the packet normally.
Tunnel Mode: Tunnel mode encapsulates the entire original IP packet—including its header—inside a new IPSec packet. Both the original header and the payload are protected. This creates a secure “tunnel” between two gateways (e.g., routers or firewalls), making it an ideal choice for site-to-site VPNs. Intermediary routers only see the encrypted outer packet and can’t discern the internal addresses or details.

Between these two mechanisms, IPSec makes sure that while data may pass through multiple untrusted networks en route to its destination, it stays confidential, authentic, and free from tampering.

IPv4, IPv6, and IPSec

IPSec was originally designed as an integral part of IPv6, with built-in support through extension headers—something not possible with IPv4's structure. However, as IPv4 networks continued to dominate the internet, IPSec evolved to work effectively with both protocols.

In IPv4 networks, IPSec operates as a payload within IP packets, while in IPv6, it can function either through extension headers or as a payload.

Although IPv6 includes IPSec support in its core specifications, implementation isn't mandatory.

While IPv6 networks can use IPSec's native integration, both IPv4 and IPv6 networks benefit equally from IPSec's security features.

Setting up an IPSec VPN

One of the most common use cases of IPSec is as a VPN to encrypt traffic between remote locations, servers, team members, or beyond. This doesn't have to be complicated, either.

For example, a popular option is the hwdsl2/setup-ipsec-vpn script from GitHub, which works on many Linux distributions.

After obtaining a Linux server (you can get a dedicated server from us, but one of our affordable NVMe KVM VPS will work great as well), updating your server (and making sure you have sudo privileges as well as wget), simply run the following command:

wget https://get.vpnsetup.net -O vpn.sh && sudo sh vpn.sh

The script will install and configure an IPSec VPN server with IPsec/L2TP, Cisco IPsec, and IKEv2. It uses Libreswan for IPSec and xl2tpd as the L2TP provider.

Once the installation completes, it’ll display randomly generated VPN credentials, and you’ll be ready to connect supported clients—including Windows, macOS, iOS, Android, Linux, and more—to your new, secured IPSec-based VPN.

Do you need an IPSec VPN if you're on an IPSec network?

Understanding the context

First, it's essential to differentiate between an IPSec-secured network and an IPSec VPN:

IPSec-Secured network: This refers to a network environment where IPSec protocols are used to secure internal communications. For example, within a corporate network, IPSec might be employed to protect data exchanged between different departments or between servers and internal devices.
IPSec VPN: An IPSec VPN extends this security beyond the internal network, enabling secure connections over untrusted external networks like the Internet. It creates encrypted tunnels between endpoints, such as remote users or branch offices and the main corporate network.

Do you need both?

Whether you need an IPSec VPN in addition to an IPSec-secured internal network depends on your specific requirements:

Internal security without remote access

Scenario: Your organization operates entirely within a single physical location with no need for remote access.
Need for IPSec VPN: Not necessary. If all communications remain within a controlled, secure environment, the existing IPSec-secured network may suffice.

Remote access or distributed offices

Scenario: Employees need to access the network remotely from various locations, or you have multiple branch offices that need to communicate securely over the Internet.
Need for IPSec VPN: Essential. An IPSec VPN ensures that data transmitted over public or untrusted networks remains secure, maintaining the same level of protection as your internal IPSec-secured communications.

Hybrid environments

Scenario: Your infrastructure includes both on-premises systems and cloud-based services, with some data traversing the Internet.
Need for IPSec VPN: Highly recommended. An IPSec VPN can secure data exchanges between different environments, ensuring consistent security policies across all connections.

Benefits of using an IPSec VPN on an IPSec network

Even if your internal network already utilizes IPSec, incorporating an IPSec VPN offers additional advantages:

Extended security: While your internal IPSec setup protects intra-network traffic, an IPSec VPN secures data as it travels between your network and external endpoints, such as remote users or branch offices.
Unified security policies: Using IPSec for both internal and VPN communications allows for consistent security configurations and management.
Scalability: An IPSec VPN provides a scalable solution for adding remote users or new branch locations without compromising security.

When it might be redundant

In highly specialized or tightly controlled environments where all devices are securely managed, and there is absolutely no need for external access, adding an IPSec VPN might be unnecessary. However, such scenarios are rare, especially with the increasing prevalence of remote work and distributed networks.

Conclusion

By encrypting packets at the network layer, authenticating their sources, and verifying their integrity, IPSec transforms the open, unprotected environment of the public Internet into a safer place.

Whether you’re adopting IPv6 and looking for simpler end-to-end encryption, maintaining legacy IPv4 systems, or mixing both protocols, IPSec is here for you if you need it!

By the way, pairing IPSec with premium hosting is always a win/win scenario ;-) -- that said, xTom offers services such as dedicated servers, colocation, IP transit, or scalable NVMe KVM VPS. We would love to provide you with the reliable digital infrastructure your project or brand needs. Don't be afraid to reach out!