topic Re: Inspection of Inter-Subnet traffic in AWS VPC using CloudGuard in Cloud Firewall

Inspection of Inter-Subnet traffic in AWS VPC using CloudGuard

Vladimir — Thu, 15 Feb 2018 17:46:56 GMT

I've been asked an interesting and, seemingly, trivial question: "How would you protect the hosts in AWS VPC located in a different subnets by inspecting traffic between them?"

I was also assured that presently, AWS did not have a solution to this problem, as every routing table you create will contain "local" route, all traffic from all subnets within one VPC will be routed through it.

To work on this puzzle, this lab environment was provisioned:

...and answer to this dilemma is to use static routes in the instances pointing to the interfaces of the vSEC or cluster, as well as security groups as Sources fro the traffic to the Private Subnets:

[root@ip-10-255-255-200 ec2-user]# route

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

default 10.255.255.193 0.0.0.0 UG 0 0 0 eth0

10.255.255.128 10.255.255.201 255.255.255.192 UG 0 0 0 eth0

10.255.255.192 * 255.255.255.192 U 0 0 0 eth0

169.254.169.254 * 255.255.255.255 UH 0 0 0 eth0

[root@ip-10-255-255-200 ec2-user]#

[root@ip-10-255-255-150 ec2-user]# route

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

default 10.255.255.129 0.0.0.0 UG 0 0 0 eth0

10.255.255.128 * 255.255.255.192 U 0 0 0 eth0

10.255.255.192 10.255.255.140 255.255.255.192 UG 0 0 0 eth0

169.254.169.254 * 255.255.255.255 UH 0 0 0 eth0

[root@ip-10-255-255-150 ec2-user]#

With Firewall Access rules set:

With NAT rules set to:

And was able to see the packet traversing firewall (10.255.255.201 and 10.255.255.140 are its interfaces):

[root@ip-10-255-255-150 ec2-user]# ssh ec2-user@10.255.255.200

Permission denied (publickey).

[root@ip-10-255-255-150 ec2-user]#

And here is the tcpdump from the target host:

[root@ip-10-255-255-200 ec2-user]# tcpdump src 10.255.255.150

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

21:03:53.440273 IP 10.255.255.150.60118 > 10.255.255.200.ssh: Flags [S], seq 2098326363, win 26883, options [mss 1460,sackOK,TS val 843716 ecr 0,nop,wscale 7], length 0

...

With this Security group assigned to both hosts in my demo, the 10.255.255.150 and 10.255.255.200:

Where sg-e2264391 is the:

[ec2-user@ip-10-255-255-150 ~]$ date; ssh 10.255.255.200
Fri Feb 16 13:29:42 UTC 2018
Permission denied (publickey).
[ec2-user@ip-10-255-255-150 ~]$ curl http://169.254.169.254/latest/meta-data/security-groups
TempSG1[ec2-user@ip-10-255-255-150 ~]$

---

[ec2-user@ip-10-255-255-200 ~]$ date; ssh 10.255.255.150
Fri Feb 16 13:30:04 UTC 2018
Permission denied (publickey).

[ec2-user@ip-10-255-255-200 ~]$ curl http://169.254.169.254/latest/meta-data/security-groups
TempSG1[ec2-user@ip-10-255-255-200 ~]$

And f you really want to be sure that the traffic in question was traversing the firewall and NOT a default VPC router:

and

[root@ip-10-255-255-200 ec2-user]# ifconfig | grep eth0

eth0 Link encap:Ethernet HWaddr 02:70:96:B0:44:80

[root@ip-10-255-255-200 ec2-user]#

----------------

[root@ip-10-255-255-200 ec2-user]# tcpdump -tttt -ne host 10.255.255.150

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

2018-02-15 16:01:28.245759 02:af:87:e2:04:c6 > 02:70:96:b0:44:80, ethertype IPv4 (0x0800), length 74: 10.255.255.150.39480 > 10.255.255.200.ssh: Flags [S], seq 3739857756, win 26883, options [mss 1460,sackOK,TS val 331468 ecr 0,nop,wscale 7], length 0

2018-02-15 16:01:28.245898 02:70:96:b0:44:80 > 02:af:87:e2:04:c6, ethertype IPv4 (0x0800), length 74: 10.255.255.200.ssh > 10.255.255.150.39480: Flags [S.], seq 3645387522, ack 3739857757, win 26847, options [mss 8961,sackOK,TS val 324911 ecr 331468,nop,wscale 7], length 0

2018-02-15 16:01:28.246290 02:af:87:e2:04:c6 > 02:70:96:b0:44:80, ethertype IPv4 (0x0800), length 66: 10.255.255.150.39480 > 10.255.255.200.ssh: Flags [.], ack 1, win 211, options [nop,nop,TS val 331469 ecr 324911], length 0

2018-02-15 16:01:28.246441 02:af:87:e2:04:c6 > 02:70:96:b0:44:80, ethertype IPv4 (0x0800), length 87: 10.255.255.150.39480 > 10.255.255.200.ssh: Flags [P.], seq 1:22, ack 1, win 211, options [nop,nop,TS val 331469 ecr 324911], length 21

2018-02-15 16:01:28.246450 02:70:96:b0:44:80 > 02:af:87:e2:04:c6, ethertype IPv4 (0x0800), length 66: 10.255.255.200.ssh > 10.255.255.150.39480: Flags [.], ack 22, win 210, options [nop,nop,TS val 324912 ecr 331469], length 0

The addition of the static routes could be either bootstrapped or included in AMIs, depending on your situation.

To verify that the instances residing in different subnets will remain isolated in the absence of the static routes, those were removed and we can see that the SSH connection attempt is timing out:

[ec2-user@ip-10-255-255-150 ~]$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 10.255.255.129 0.0.0.0 UG 0 0 0 eth0
10.255.255.128 * 255.255.255.192 U 0 0 0 eth0
169.254.169.254 * 255.255.255.255 UH 0 0 0 eth0
[ec2-user@ip-10-255-255-150 ~]$ ssh ec2-user@10.255.255.200
ssh: connect to host 10.255.255.200 port 22: Connection timed out
[ec2-user@ip-10-255-255-150 ~]$

----

[ec2-user@ip-10-255-255-200 ~]$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 10.255.255.193 0.0.0.0 UG 0 0 0 eth0
10.255.255.192 * 255.255.255.192 U 0 0 0 eth0
169.254.169.254 * 255.255.255.255 UH 0 0 0 eth0
[ec2-user@ip-10-255-255-200 ~]$ ssh ec2-user@10.255.255.150
ssh: connect to host 10.255.255.150 port 22: Connection timed out
[ec2-user@ip-10-255-255-200 ~]$

And reinstatement of the static routes results in:

[root@ip-10-255-255-150 ec2-user]# nano /etc/sysconfig/network-scripts/route-eth0
[root@ip-10-255-255-150 ec2-user]# reboot
[root@ip-10-255-255-150 ec2-user]#
Broadcast message from ec2-user@ip-10-255-255-150
(/dev/pts/0) at 16:54 ...

The system is going down for reboot NOW!
Using username "ec2-user".
Authenticating with public key "imported-openssh-key"
Last login: Fri Feb 16 16:42:28 2018 from xx.xx.xxx.98

__| __|_ )
_| ( / Amazon Linux AMI
___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2017.09-release-notes/
[ec2-user@ip-10-255-255-150 ~]$ ssh ec2-user@10.255.255.200
Permission denied (publickey).
[ec2-user@ip-10-255-255-150 ~]$

and:

root@ip-10-255-255-200 ec2-user]# nano /etc/sysconfig/network-scripts/route-eth0
[root@ip-10-255-255-200 ec2-user]# reboot
[root@ip-10-255-255-200 ec2-user]#
Broadcast message from ec2-user@ip-10-255-255-200
(/dev/pts/0) at 16:55 ...

The system is going down for reboot NOW!
Using username "ec2-user".
Authenticating with public key "imported-openssh-key"
Last login: Fri Feb 16 16:42:10 2018 from xx.xx.xxx.98

__| __|_ )
_| ( / Amazon Linux AMI
___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2017.09-release-notes/
[ec2-user@ip-10-255-255-200 ~]$ ssh ec2-user@10.255.255.150
Permission denied (publickey).
[ec2-user@ip-10-255-255-200 ~]$

This is the Gaia config for the vSEC used in this lab:

vSEC01> show configuration
#
# Configuration of vSEC01
# Language version: 13.1v1
#
# Exported by admin on Thu Feb 15 13:47:33 2018
#
set installer policy check-for-updates-period 3
set installer policy periodically-self-update on
set installer policy send-cpuse-data off
set installer policy self-test install-policy off
set installer policy self-test network-link-up off
set installer policy self-test start-processes on
set arp table cache-size 4096
set arp table validity-timeout 60
set arp announce 2
set message banner on

set message motd on

set message caption off
set core-dump enable
set core-dump total 1000
set core-dump per_process 2
set clienv debug 0
set clienv echo-cmd off
set clienv output pretty
set clienv prompt "%M"
set clienv rows 24
set clienv syntax-check off
set dns primary 10.255.255.2
set dns secondary 8.8.8.8
set edition 64-bit
set expert-password-hash $blablabla
set format date dd-mmm-yyyy
set format time 24-hour
set format netmask Dotted
set hostname vSEC01
add allowed-client host any-host
set web table-refresh-rate 15
set web session-timeout 30
set web ssl-port 443
set web ssl3-enabled off
set web daemon-enable on
set inactivity-timeout 10
set ipv6-state off
add command api path /bin/api_wrap description "Start, stop, or check status of API server"
add command tecli path /bin/tecli_start description "Threat Emulation Blade shell"
set net-access telnet off
set ntp active on
set ntp server primary pool.ntp.org version 2
set user admin shell /bin/bash
set user admin password-hash $blablabla
set user monitor shell /etc/cli.sh
set user monitor password-hash *
set password-controls min-password-length 6
set password-controls complexity 2
set password-controls palindrome-check true
set password-controls history-checking true
set password-controls history-length 10
set password-controls password-expiration never
set password-controls expiration-warning-days 7
set password-controls expiration-lockout-days never
set password-controls force-change-when no
set password-controls deny-on-nonuse enable false
set password-controls deny-on-nonuse allowed-days 365
set password-controls deny-on-fail enable false
set password-controls deny-on-fail failures-allowed 10
set password-controls deny-on-fail allow-after 1200
set aaa tacacs-servers state off
set aaa radius-servers super-user-uid 96
set max-path-splits 8
set tracefile maxnum 10
set tracefile size 1
set syslog filename /var/log/messages
set syslog cplogs off
set syslog mgmtauditlogs on
set syslog auditlog permanent
set timezone America / New_York
set interface eth0 comments "vSEC01-Ext"
set interface eth0 link-speed 10G/full
set interface eth0 state on
set interface eth0 auto-negotiation on
set interface eth0 mtu 1500
set interface eth0 ipv4-address 10.255.255.22 mask-length 26
set interface eth1 comments "vSEC01-Int"
set interface eth1 link-speed 10G/full
set interface eth1 state on
set interface eth1 auto-negotiation on
set interface eth1 mtu 1500
set interface eth1 ipv4-address 10.255.255.201 mask-length 26
set interface eth2 comments "vSEC01-Proxy"
set interface eth2 link-speed 10G/full
set interface eth2 state on
set interface eth2 auto-negotiation on
set interface eth2 mtu 1500
set interface eth2 ipv4-address 10.255.255.140 mask-length 26
set interface lo state on
set interface lo ipv4-address 127.0.0.1 mask-length 8
add host name Simple01-LogicalServer-Web ipv4-address 10.255.255.23
set inbound-route-filter ospf2 accept-all-ipv4
set inbound-route-filter rip accept-all-ipv4
set management interface eth0
set ospf area backbone on
set rip update-interval default
set rip expire-interval default
set snmp mode default
set snmp agent off
set snmp agent-version v3-Only
set snmp traps trap authorizationError disable
set snmp traps trap biosFailure disable
set snmp traps trap coldStart disable
set snmp traps trap configurationChange disable
set snmp traps trap configurationSave disable
set snmp traps trap fanFailure disable
set snmp traps trap highVoltage disable
set snmp traps trap linkUpLinkDown disable
set snmp traps trap lowDiskSpace disable
set snmp traps trap lowVoltage disable
set snmp traps trap overTemperature disable
set snmp traps trap powerSupplyFailure disable
set snmp traps trap raidVolumeState disable
set snmp traps trap vrrpv2AuthFailure disable
set snmp traps trap vrrpv2NewMaster disable
set snmp traps trap vrrpv3NewMaster disable
set snmp traps trap vrrpv3ProtoError disable
set static-route default comment "To Subnet Router"
set static-route default nexthop gateway address 10.255.255.1 on
set static-route 10.100.100.0/24 comment "To Subnet Router for Peered VPC CIDR"
set static-route 10.100.100.0/24 nexthop gateway address 10.255.255.193 on
set static-route 10.255.255.128/26 comment "To Subnet Router"
set static-route 10.255.255.128/26 nexthop gateway address 10.255.255.129 on
set static-route 10.255.255.192/26 comment "To Subnet Router"
set static-route 10.255.255.192/26 nexthop gateway address 10.255.255.193 on
vSEC01>

Enjoy

Re: Inspection of Inter-Subnet traffic in AWS VPC using CloudGuard

John_Fenoughty — Thu, 15 Feb 2018 21:09:14 GMT

That's interesting and very nicely demonstrated and if I am asked to do this, I'll now have a way to achieve it.

Having said that, some chaps and I are working on the design for an AWS infrastructure with a various 'zones' and Check Point firewalls to provide protection (mainly DLP) between the zones. This is being designed from the ground up with security in mind; security by design as some would call it. We have made the decision that we will have a firewall between the various VPCs while maintaining multiple subnets within a single VPC. In this way, the problem that you have solved will not exist.

When I liken the AWS environment to the traditional network model (understanding that there are important and major differences) I see the multiple subnets in a VPC a bit like the multiple VLANs on a network which are routed within layer 3 switches (usually the core) and then the inter-VPC communications to be much like those VLANs which are tagged to the Check Point(s)' multi VLAN (trunked) ports for firewalling and routing.

Of course, if inheriting an infrastructure that is already in place and having to firewall between subnets in a VPC then your solution is exactly what we'll need - thanks.

I wonder if anyone else has an opinion about which approach they might take when designing this from the ground up?

Re: Inspection of Inter-Subnet traffic in AWS VPC using CloudGuard

Vladimir — Thu, 15 Feb 2018 23:08:38 GMT

Thank you John.

I do prefer separation in VPCs and some of the scenarios are described here:

vSEC deployment scenarios in AWS

It's just in this case, I was presented by the client with the question of "how would you deal with this unsolvable problem", that AWS professional services were not able to find a solution to.

I tend to take umbrage at the notion of unsolvable problems, hence this post

Re: Inspection of Inter-Subnet traffic in AWS VPC using CloudGuard

PhoneBoy — Fri, 16 Feb 2018 04:12:45 GMT

There is nothing preventing someone from changing the routes on a given instance to bypass the firewall entirely.

In other words, the segmentation is merely logical, not physical as it would be in a more traditional network.

So, while technically you can do this, it's not necessarily secure.

It's also something that's not likely to work with auto-provisioned instances since they are usually deployed with DHCP.

Perhaps with a layer of NAT and Network ACLs, you could theoretically set it up so the traffic from one instance can't reach another without going through the firewall.

However, that adds complexity.

In Azure, you can force this sort of segmentation with User-Defined Routes.

Individual instances cannot override them as the routes are basically forced at the network layer.

Re: Inspection of Inter-Subnet traffic in AWS VPC using CloudGuard

Vladimir — Fri, 16 Feb 2018 04:49:30 GMT

You should be able to prevent the bypass by using security groups, specifying that only communication between one that the vSEC belongs to and the ones containing instances is permitted.

Even in autoscaling DHCP scenarios, you can still bootstrap the static route allocation to the launch group making enforcement of the routing mandatory.

The challenge, as it was posited to me, was to make it work explicitly without NAT and I was told it was impossible and asked for best possible alternative of securing instances by means of host-based solutions or any other suitable means.

After making this happen, I've done some searches on this subject, but either my queries were poorly formed, or this is the only way it could presently be accomplished in AWS.

Can you verify this, before I start puffing-up and tooting my own, (and by extension, check points') horn?

Re: Inspection of Inter-Subnet traffic in AWS VPC using CloudGuard

PhoneBoy — Fri, 16 Feb 2018 07:18:38 GMT

When we were looking at this years ago, it was not possible to force the traffic flow the way you describe with security groups.

Perhaps this has changed since then, which would be fantastic news for a lot of people.

I'll have to look at this again.

Re: Inspection of Inter-Subnet traffic in AWS VPC using CloudGuard

Vladimir — Fri, 16 Feb 2018 13:35:19 GMT

Well, unless I am missing something, it looks as it does as advertised.

With this Security group assigned to both hosts in my demo, the 10.255.255.150 and 10.255.255.200:

Where sg-e2264391 is the:

-----

[ec2-user@ip-10-255-255-200 ~]$ date; ssh 10.255.255.150
Fri Feb 16 13:30:04 UTC 2018
Permission denied (publickey).

[ec2-user@ip-10-255-255-200 ~]$ curl http://169.254.169.254/latest/meta-data/security-groups
TempSG1[ec2-user@ip-10-255-255-200 ~]$

----

Hmm... Toot!!!

P.S. Corrected diagram:

Re: Inspection of Inter-Subnet traffic in AWS VPC using CloudGuard

PhoneBoy — Fri, 16 Feb 2018 16:22:18 GMT

If you revert the routing on the instances back to what AWS does by default, what happens?

If this is working "securely" the traffic would not be permitted (i.e. the security groups in AWS should block it).

Re: Inspection of Inter-Subnet traffic in AWS VPC using CloudGuard

Vladimir — Fri, 16 Feb 2018 16:30:10 GMT

About to try it.

If my logic is not faulty, and if I am still thinking straight, shouldn't it be dropped unless specifically permitted in SGs attached to the instances?

Re: Inspection of Inter-Subnet traffic in AWS VPC using CloudGuard

Vladimir — Fri, 16 Feb 2018 16:47:13 GMT

Toot!, Toot! and one more Toot! for a good measure:

----

Re: Inspection of Inter-Subnet traffic in AWS VPC using CloudGuard

Vladimir — Fri, 16 Feb 2018 17:01:51 GMT

And with static routes reinstated:

The system is going down for reboot NOW!
Using username "ec2-user".
Authenticating with public key "imported-openssh-key"
Last login: Fri Feb 16 16:42:28 2018 from xx.xx.xxx.98

__| __|_ )
_| ( / Amazon Linux AMI
___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2017.09-release-notes/
[ec2-user@ip-10-255-255-150 ~]$ ssh ec2-user@10.255.255.200
Permission denied (publickey).
[ec2-user@ip-10-255-255-150 ~]$

-----

[root@ip-10-255-255-200 ec2-user]# nano /etc/sysconfig/network-scripts/route-eth0
[root@ip-10-255-255-200 ec2-user]# reboot
[root@ip-10-255-255-200 ec2-user]#
Broadcast message from ec2-user@ip-10-255-255-200
(/dev/pts/0) at 16:55 ...

The system is going down for reboot NOW!
Using username "ec2-user".
Authenticating with public key "imported-openssh-key"
Last login: Fri Feb 16 16:42:10 2018 from xx.xx.xxx.98

__| __|_ )
_| ( / Amazon Linux AMI
___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2017.09-release-notes/
[ec2-user@ip-10-255-255-200 ~]$ ssh ec2-user@10.255.255.150
Permission denied (publickey).
[ec2-user@ip-10-255-255-200 ~]$

Re: Inspection of Inter-Subnet traffic in AWS VPC using CloudGuard

PhoneBoy — Fri, 16 Feb 2018 17:02:15 GMT

That does look promising

Re: Inspection of Inter-Subnet traffic in AWS VPC using CloudGuard

Vladimir — Fri, 16 Feb 2018 17:05:55 GMT

Do you want me to re-work the document into a more presentable form?

If so, what name would you suggest for it?

Re: Inspection of Inter-Subnet traffic in AWS VPC using CloudGuard

PhoneBoy — Fri, 16 Feb 2018 18:06:54 GMT

It probably wouldn't hurt

The existing name is ok.

Re: Inspection of Inter-Subnet traffic in AWS VPC using CloudGuard

PhoneBoy — Sat, 24 Feb 2018 00:34:33 GMT

While you get points for extreme cleverness, I will add a couple of caveats to this approach beyond what was already discussed:

Not sure this will work with a clustered gateway unless you also include scripts to monitor the "next hop" and adjust in case a cluster member fails. This is because, unlike traditional physical clusters, there is no cluster IP address with clusters in AWS (due to limitations in AWS).
It does require NICs in the relevant subnets (which kinda goes against the SDN philosophy and may also limit you to certain, larger, instance sizes).

Re: Inspection of Inter-Subnet traffic in AWS VPC using CloudGuard

Calvin_Nguyen — Tue, 06 Mar 2018 17:48:27 GMT

I am setting this environment up right now. Does the host need to have public temp IP? I am assuming no, correct?

Re: Inspection of Inter-Subnet traffic in AWS VPC using CloudGuard

Vladimir — Tue, 06 Mar 2018 17:52:53 GMT

If you are talking about EC2 instances, they do not have to have public IPs.

The reason for those in my lab was to gain the remote access to them bypassing vSEC for simplicity and to avoid a lockout should I bork something in the policy.

Re: Inspection of Inter-Subnet traffic in AWS VPC using CloudGuard

Calvin_Nguyen — Tue, 06 Mar 2018 19:38:07 GMT

Got it. What is the purpose of the 169.254.169.254 address in the route table?

Re: Inspection of Inter-Subnet traffic in AWS VPC using CloudGuard

Vladimir — Tue, 06 Mar 2018 19:43:29 GMT

That's the AWS' metadata lookup IP.

Re: Inspection of Inter-Subnet traffic in AWS VPC using CloudGuard

Hugh_McGauran — Tue, 06 Mar 2018 20:32:50 GMT

the big issue that i would have is that the routing is now back in the hands of the sysadmin... if they modify the routing they could bypass the gw no?