Today I tried to implement IPsec for certain protocols (in my example for TCP port 80 from Windows 10 client to Windows server 2016 running IIS and ICMP just to show it is possible to enable IPSec on per-protocol basis).

In my environment I have setup a simple domain with 2 servers, 1 DC and 1 member server with IIS, 1 Windows 10 domain joined client and one Windows 10 with Wireshark just to sniff the traffic (by using Hyper-V port mirroring).

You can check a 6 minutes video tutorial here.

So steps to enable IPSec by using Windows Firewall with Advanced Security (introduced in Windows Vista) are the following:

1. First thing you need to do is to create a security group where you put servers and clients you want to have IPSec policies enabled.

2. Then you need to create a group policy object on top of computers / servers OU (in my case I have created GPO on a domain level – in a production environment I suggest that you put it somewhere lower on top of your computers and servers OUs.
Remove authenticated users from GPO security filtering and insert your IPSec security group

3. Then you need to edit the GPO on the location: Computer configuration / Policies / Windows settings / Security settings / Windows Firewall with Advanced Security / Windows Firewall with Advanced Security / Connection Security Rules
Here you create a new rule:
– I choose Custom rule option
– I left Any IP address selected on both Endpoints (so it will work for all IP addresses)
– I configured the second option on next screen so – Require auth for inbound (so all inbound connections will require authentication – Warning! No access from workgroup or not domain joined computers) and reqeust authentication for outbound (www.google.com does not care about your IPSec policy :))
– On the next screen I used Computer (Kerberos V5) authentication method
– On protocols and ports – in my first example I used TCP 80 on Endpoint1 (so “server side” will require authentication for everyone who would like to access web server on port 80)
– I applied policy on all profiles and I gave a name to my policy.
After creation of the policy you should run: gpupdate /force on both – server and client – I did a reboot just to be sure it will do it – but most of the time gpupdate /force will be enough. You can see your policy on server and client if you open Windows Firewall with Advanced Security and you click on Connection Security Rules

I used another Windows 10 machine with Wireshark software just to monitor the functioning of IPSec – I used Hyper-V port mirroring to send copy of all traffic from my domain joined Windows 10 so you can see from captured traffic that policy was applied correctly and that traffic (opening the http://srv1 and later on pinging srv1) is encrypted.