How to Configure IPSec on Windows

There are many applications that authenticate and encrypt network traffic using a separate third-party program.

However, Microsoft’s operating system can also implement this natively through IPSEC configuration. In this article, we’ll look at what IPSEC is and a simple implementation example.

What is IPSEC?

Internet Protocol Security or IPSEC is a protocol used to authenticate and encrypt IP communications. This is achieved through mutual authentication between agents, as well as exchange of cryptographic keys at the beginning of the session.

IPSEC will also allow IP restrictions and TCP / UDP layer encryption to be added for applications that might not otherwise support it. IPSEC uses IP 50 (ESP), IP 51 (AH) and UDP port 500.

IPSEC implementation

In this example, we will configure IPSEC to encrypt communication between two Windows computers. The first machine, a Windows 2012 server, will act as a VPN server.

The second machine, the Windows 10 client, will act as the VPN client. LT2P IPSEC VPN can exchange a preshared key or certificate. In this example, we will exchange the shared key.

Set up a VPN server

On a Windows 2012 machine, we will need to install the routing and remote access functionality. To do this, go to Server Manager and add roles and features. Select a role-based or feature-based installation. Select a local server. Choose to install the following server roles.

Network Policy and Access Services

Network Policy Server

Remote access

Direct Access & VPN (RAS)

Routing

After installing these new features, you need a snap-in to manage them. Open mmc.exe as administrator. Go to file | Add / remove snap-in. Add the Routing and Remote Access snap-in.

This snap-in allows you to configure LAN-to-LAN, LAN-to-WAN, virtual private network (VPN), and network address translation (NAT) multi-protocol routing services.

In the mmc console, right click on Routing and Remote Access and select Add Server. Select your local machine. Then right-click on the newly created computer and select “Configure and Enable Routing and Remote Access”. Select remote access (remote access or VPN).

Then check the VPN option. To do this, you must have at least two network cards. One of them can be looped. Specify the address range for the incoming connection. Make sure they do not conflict with other addresses allocated on your existing network. In this example, we will not be using a RADIUS server.

Then try starting the Routing and Remote Access service. You may need to delete the following registry key to start the service.

HKEY_LOCAL_MACHINE SYSTEM CurrentControlSet Services RemoteAccess RouterManagers Ipv6

In the mmc.exe console, right-click the computer name and select Properties. Change these properties on the security tab.

Select authentication methods as shown below.

Select the checkbox to enable custom IPSEC policy for L2TP / IKEv2 connection. Add a pre-shared key.

Finally, you will need to change the user to access the VPN. Open compmgmt.msc, go to the Local Users and Groups section and select the properties of the user you want to use for VPN.

Go to the Dial Up tab. Select Allow Access and click Apply. A restart is required on your computer. After rebooting, you will be ready to test your first client.

Windows 10 device setup

On your Windows 10 PC, open Network & Internet Settings. Select VPN in the left pane and add a VPN connection. Edit the advanced options.

Place the IP address of your VPN server under the server name or address. Select L2TP / IPSEC with pre-shared key option under VPN type. Add the pre-shared key, username and password.

The security properties for the VPN will need to be changed on the network adapter. On the VPN adapter, select properties and go to the Security tab. Select the EAP radio button and select Microsoft: Secure Password (EAP-MSCHAPv2) (encryption enabled).

Finally, right-click the adapter again to connect. Congratulations! You have created an IPSEC VPN tunnel.