Inspection of Inter-Subnet traffic in AWS VPC usin...

Vladimir · ‎2018-02-15

I've been asked an interesting and, seemingly, trivial question: "How would you protect the hosts in AWS VPC located in a different subnets by inspecting traffic between them?"

I was also assured that presently, AWS did not have a solution to this problem, as every routing table you create will contain "local" route, all traffic from all subnets within one VPC will be routed through it.

To work on this puzzle, this lab environment was provisioned:

...and answer to this dilemma is to use static routes in the instances pointing to the interfaces of the vSEC or cluster, as well as security groups as Sources fro the traffic to the Private Subnets:

[root@ip-10-255-255-200 ec2-user]# route

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

default 10.255.255.193 0.0.0.0 UG 0 0 0 eth0

10.255.255.128 10.255.255.201 255.255.255.192 UG 0 0 0 eth0

10.255.255.192 * 255.255.255.192 U 0 0 0 eth0

169.254.169.254 * 255.255.255.255 UH 0 0 0 eth0

[root@ip-10-255-255-200 ec2-user]#

[root@ip-10-255-255-150 ec2-user]# route

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

default 10.255.255.129 0.0.0.0 UG 0 0 0 eth0

10.255.255.128 * 255.255.255.192 U 0 0 0 eth0

10.255.255.192 10.255.255.140 255.255.255.192 UG 0 0 0 eth0

169.254.169.254 * 255.255.255.255 UH 0 0 0 eth0

[root@ip-10-255-255-150 ec2-user]#

With Firewall Access rules set:

With NAT rules set to:

And was able to see the packet traversing firewall (10.255.255.201 and 10.255.255.140 are its interfaces):

[root@ip-10-255-255-150 ec2-user]# ssh ec2-user@10.255.255.200

Permission denied (publickey).

[root@ip-10-255-255-150 ec2-user]#

And here is the tcpdump from the target host:

[root@ip-10-255-255-200 ec2-user]# tcpdump src 10.255.255.150

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

21:03:53.440273 IP 10.255.255.150.60118 > 10.255.255.200.ssh: Flags [S], seq 2098326363, win 26883, options [mss 1460,sackOK,TS val 843716 ecr 0,nop,wscale 7], length 0

...

With this Security group assigned to both hosts in my demo, the 10.255.255.150 and 10.255.255.200:

Where sg-e2264391 is the:

[ec2-user@ip-10-255-255-150 ~]$ date; ssh 10.255.255.200
Fri Feb 16 13:29:42 UTC 2018
Permission denied (publickey).
[ec2-user@ip-10-255-255-150 ~]$ curl http://169.254.169.254/latest/meta-data/security-groups
TempSG1[ec2-user@ip-10-255-255-150 ~]$

---

[ec2-user@ip-10-255-255-200 ~]$ date; ssh 10.255.255.150
Fri Feb 16 13:30:04 UTC 2018
Permission denied (publickey).

[ec2-user@ip-10-255-255-200 ~]$ curl http://169.254.169.254/latest/meta-data/security-groups
TempSG1[ec2-user@ip-10-255-255-200 ~]$

And f you really want to be sure that the traffic in question was traversing the firewall and NOT a default VPC router:

and

[root@ip-10-255-255-200 ec2-user]# ifconfig | grep eth0

eth0 Link encap:Ethernet HWaddr 02:70:96:B0:44:80

[root@ip-10-255-255-200 ec2-user]#

----------------

[root@ip-10-255-255-200 ec2-user]# tcpdump -tttt -ne host 10.255.255.150

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

2018-02-15 16:01:28.245759 02:af:87:e2:04:c6 > 02:70:96:b0:44:80, ethertype IPv4 (0x0800), length 74: 10.255.255.150.39480 > 10.255.255.200.ssh: Flags [S], seq 3739857756, win 26883, options [mss 1460,sackOK,TS val 331468 ecr 0,nop,wscale 7], length 0

2018-02-15 16:01:28.245898 02:70:96:b0:44:80 > 02:af:87:e2:04:c6, ethertype IPv4 (0x0800), length 74: 10.255.255.200.ssh > 10.255.255.150.39480: Flags [S.], seq 3645387522, ack 3739857757, win 26847, options [mss 8961,sackOK,TS val 324911 ecr 331468,nop,wscale 7], length 0

2018-02-15 16:01:28.246290 02:af:87:e2:04:c6 > 02:70:96:b0:44:80, ethertype IPv4 (0x0800), length 66: 10.255.255.150.39480 > 10.255.255.200.ssh: Flags [.], ack 1, win 211, options [nop,nop,TS val 331469 ecr 324911], length 0

2018-02-15 16:01:28.246441 02:af:87:e2:04:c6 > 02:70:96:b0:44:80, ethertype IPv4 (0x0800), length 87: 10.255.255.150.39480 > 10.255.255.200.ssh: Flags [P.], seq 1:22, ack 1, win 211, options [nop,nop,TS val 331469 ecr 324911], length 21

2018-02-15 16:01:28.246450 02:70:96:b0:44:80 > 02:af:87:e2:04:c6, ethertype IPv4 (0x0800), length 66: 10.255.255.200.ssh > 10.255.255.150.39480: Flags [.], ack 22, win 210, options [nop,nop,TS val 324912 ecr 331469], length 0

The addition of the static routes could be either bootstrapped or included in AMIs, depending on your situation.

To verify that the instances residing in different subnets will remain isolated in the absence of the static routes, those were removed and we can see that the SSH connection attempt is timing out:

[ec2-user@ip-10-255-255-150 ~]$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 10.255.255.129 0.0.0.0 UG 0 0 0 eth0
10.255.255.128 * 255.255.255.192 U 0 0 0 eth0
169.254.169.254 * 255.255.255.255 UH 0 0 0 eth0
[ec2-user@ip-10-255-255-150 ~]$ ssh ec2-user@10.255.255.200
ssh: connect to host 10.255.255.200 port 22: Connection timed out
[ec2-user@ip-10-255-255-150 ~]$

----

[ec2-user@ip-10-255-255-200 ~]$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 10.255.255.193 0.0.0.0 UG 0 0 0 eth0
10.255.255.192 * 255.255.255.192 U 0 0 0 eth0
169.254.169.254 * 255.255.255.255 UH 0 0 0 eth0
[ec2-user@ip-10-255-255-200 ~]$ ssh ec2-user@10.255.255.150
ssh: connect to host 10.255.255.150 port 22: Connection timed out
[ec2-user@ip-10-255-255-200 ~]$

And reinstatement of the static routes results in:

[root@ip-10-255-255-150 ec2-user]# nano /etc/sysconfig/network-scripts/route-eth0
[root@ip-10-255-255-150 ec2-user]# reboot
[root@ip-10-255-255-150 ec2-user]#
Broadcast message from ec2-user@ip-10-255-255-150
(/dev/pts/0) at 16:54 ...

The system is going down for reboot NOW!
Using username "ec2-user".
Authenticating with public key "imported-openssh-key"
Last login: Fri Feb 16 16:42:28 2018 from xx.xx.xxx.98

__| __|_ )
_| ( / Amazon Linux AMI
___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2017.09-release-notes/
[ec2-user@ip-10-255-255-150 ~]$ ssh ec2-user@10.255.255.200
Permission denied (publickey).
[ec2-user@ip-10-255-255-150 ~]$

and:

root@ip-10-255-255-200 ec2-user]# nano /etc/sysconfig/network-scripts/route-eth0
[root@ip-10-255-255-200 ec2-user]# reboot
[root@ip-10-255-255-200 ec2-user]#
Broadcast message from ec2-user@ip-10-255-255-200
(/dev/pts/0) at 16:55 ...

The system is going down for reboot NOW!
Using username "ec2-user".
Authenticating with public key "imported-openssh-key"
Last login: Fri Feb 16 16:42:10 2018 from xx.xx.xxx.98

__| __|_ )
_| ( / Amazon Linux AMI
___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2017.09-release-notes/
[ec2-user@ip-10-255-255-200 ~]$ ssh ec2-user@10.255.255.150
Permission denied (publickey).
[ec2-user@ip-10-255-255-200 ~]$

This is the Gaia config for the vSEC used in this lab:

vSEC01> show configuration
#
# Configuration of vSEC01
# Language version: 13.1v1
#
# Exported by admin on Thu Feb 15 13:47:33 2018
#
set installer policy check-for-updates-period 3
set installer policy periodically-self-update on
set installer policy send-cpuse-data off
set installer policy self-test install-policy off
set installer policy self-test network-link-up off
set installer policy self-test start-processes on
set arp table cache-size 4096
set arp table validity-timeout 60
set arp announce 2
set message banner on

set message motd on

set message caption off
set core-dump enable
set core-dump total 1000
set core-dump per_process 2
set clienv debug 0
set clienv echo-cmd off
set clienv output pretty
set clienv prompt "%M"
set clienv rows 24
set clienv syntax-check off
set dns primary 10.255.255.2
set dns secondary 8.8.8.8
set edition 64-bit
set expert-password-hash $blablabla
set format date dd-mmm-yyyy
set format time 24-hour
set format netmask Dotted
set hostname vSEC01
add allowed-client host any-host
set web table-refresh-rate 15
set web session-timeout 30
set web ssl-port 443
set web ssl3-enabled off
set web daemon-enable on
set inactivity-timeout 10
set ipv6-state off
add command api path /bin/api_wrap description "Start, stop, or check status of API server"
add command tecli path /bin/tecli_start description "Threat Emulation Blade shell"
set net-access telnet off
set ntp active on
set ntp server primary pool.ntp.org version 2
set user admin shell /bin/bash
set user admin password-hash $blablabla
set user monitor shell /etc/cli.sh
set user monitor password-hash *
set password-controls min-password-length 6
set password-controls complexity 2
set password-controls palindrome-check true
set password-controls history-checking true
set password-controls history-length 10
set password-controls password-expiration never
set password-controls expiration-warning-days 7
set password-controls expiration-lockout-days never
set password-controls force-change-when no
set password-controls deny-on-nonuse enable false
set password-controls deny-on-nonuse allowed-days 365
set password-controls deny-on-fail enable false
set password-controls deny-on-fail failures-allowed 10
set password-controls deny-on-fail allow-after 1200
set aaa tacacs-servers state off
set aaa radius-servers super-user-uid 96
set max-path-splits 8
set tracefile maxnum 10
set tracefile size 1
set syslog filename /var/log/messages
set syslog cplogs off
set syslog mgmtauditlogs on
set syslog auditlog permanent
set timezone America / New_York
set interface eth0 comments "vSEC01-Ext"
set interface eth0 link-speed 10G/full
set interface eth0 state on
set interface eth0 auto-negotiation on
set interface eth0 mtu 1500
set interface eth0 ipv4-address 10.255.255.22 mask-length 26
set interface eth1 comments "vSEC01-Int"
set interface eth1 link-speed 10G/full
set interface eth1 state on
set interface eth1 auto-negotiation on
set interface eth1 mtu 1500
set interface eth1 ipv4-address 10.255.255.201 mask-length 26
set interface eth2 comments "vSEC01-Proxy"
set interface eth2 link-speed 10G/full
set interface eth2 state on
set interface eth2 auto-negotiation on
set interface eth2 mtu 1500
set interface eth2 ipv4-address 10.255.255.140 mask-length 26
set interface lo state on
set interface lo ipv4-address 127.0.0.1 mask-length 8
add host name Simple01-LogicalServer-Web ipv4-address 10.255.255.23
set inbound-route-filter ospf2 accept-all-ipv4
set inbound-route-filter rip accept-all-ipv4
set management interface eth0
set ospf area backbone on
set rip update-interval default
set rip expire-interval default
set snmp mode default
set snmp agent off
set snmp agent-version v3-Only
set snmp traps trap authorizationError disable
set snmp traps trap biosFailure disable
set snmp traps trap coldStart disable
set snmp traps trap configurationChange disable
set snmp traps trap configurationSave disable
set snmp traps trap fanFailure disable
set snmp traps trap highVoltage disable
set snmp traps trap linkUpLinkDown disable
set snmp traps trap lowDiskSpace disable
set snmp traps trap lowVoltage disable
set snmp traps trap overTemperature disable
set snmp traps trap powerSupplyFailure disable
set snmp traps trap raidVolumeState disable
set snmp traps trap vrrpv2AuthFailure disable
set snmp traps trap vrrpv2NewMaster disable
set snmp traps trap vrrpv3NewMaster disable
set snmp traps trap vrrpv3ProtoError disable
set static-route default comment "To Subnet Router"
set static-route default nexthop gateway address 10.255.255.1 on
set static-route 10.100.100.0/24 comment "To Subnet Router for Peered VPC CIDR"
set static-route 10.100.100.0/24 nexthop gateway address 10.255.255.193 on
set static-route 10.255.255.128/26 comment "To Subnet Router"
set static-route 10.255.255.128/26 nexthop gateway address 10.255.255.129 on
set static-route 10.255.255.192/26 comment "To Subnet Router"
set static-route 10.255.255.192/26 nexthop gateway address 10.255.255.193 on
vSEC01>

Enjoy

John_Fenoughty · ‎2018-02-15

That's interesting and very nicely demonstrated and if I am asked to do this, I'll now have a way to achieve it.

Having said that, some chaps and I are working on the design for an AWS infrastructure with a various 'zones' and Check Point firewalls to provide protection (mainly DLP) between the zones. This is being designed from the ground up with security in mind; security by design as some would call it. We have made the decision that we will have a firewall between the various VPCs while maintaining multiple subnets within a single VPC. In this way, the problem that you have solved will not exist.

When I liken the AWS environment to the traditional network model (understanding that there are important and major differences) I see the multiple subnets in a VPC a bit like the multiple VLANs on a network which are routed within layer 3 switches (usually the core) and then the inter-VPC communications to be much like those VLANs which are tagged to the Check Point(s)' multi VLAN (trunked) ports for firewalling and routing.

Of course, if inheriting an infrastructure that is already in place and having to firewall between subnets in a VPC then your solution is exactly what we'll need - thanks.

I wonder if anyone else has an opinion about which approach they might take when designing this from the ground up?

Vladimir · ‎2018-02-15

Thank you John.

I do prefer separation in VPCs and some of the scenarios are described here:

vSEC deployment scenarios in AWS

It's just in this case, I was presented by the client with the question of "how would you deal with this unsolvable problem", that AWS professional services were not able to find a solution to.

I tend to take umbrage at the notion of unsolvable problems, hence this post

PhoneBoy · ‎2018-02-15

There is nothing preventing someone from changing the routes on a given instance to bypass the firewall entirely.

In other words, the segmentation is merely logical, not physical as it would be in a more traditional network.

So, while technically you can do this, it's not necessarily secure.

It's also something that's not likely to work with auto-provisioned instances since they are usually deployed with DHCP.

Perhaps with a layer of NAT and Network ACLs, you could theoretically set it up so the traffic from one instance can't reach another without going through the firewall.

However, that adds complexity.

In Azure, you can force this sort of segmentation with User-Defined Routes.

Individual instances cannot override them as the routes are basically forced at the network layer.

Vladimir · ‎2018-02-15

You should be able to prevent the bypass by using security groups, specifying that only communication between one that the vSEC belongs to and the ones containing instances is permitted.

Even in autoscaling DHCP scenarios, you can still bootstrap the static route allocation to the launch group making enforcement of the routing mandatory.

The challenge, as it was posited to me, was to make it work explicitly without NAT and I was told it was impossible and asked for best possible alternative of securing instances by means of host-based solutions or any other suitable means.

After making this happen, I've done some searches on this subject, but either my queries were poorly formed, or this is the only way it could presently be accomplished in AWS.

Can you verify this, before I start puffing-up and tooting my own, (and by extension, check points') horn?

PhoneBoy · ‎2018-02-15

When we were looking at this years ago, it was not possible to force the traffic flow the way you describe with security groups.

Perhaps this has changed since then, which would be fantastic news for a lot of people.

I'll have to look at this again.

Vladimir · ‎2018-02-16

Well, unless I am missing something, it looks as it does as advertised.

With this Security group assigned to both hosts in my demo, the 10.255.255.150 and 10.255.255.200:

Where sg-e2264391 is the:

[ec2-user@ip-10-255-255-150 ~]$ date; ssh 10.255.255.200
Fri Feb 16 13:29:42 UTC 2018
Permission denied (publickey).
[ec2-user@ip-10-255-255-150 ~]$ curl http://169.254.169.254/latest/meta-data/security-groups
TempSG1[ec2-user@ip-10-255-255-150 ~]$

-----

[ec2-user@ip-10-255-255-200 ~]$ date; ssh 10.255.255.150
Fri Feb 16 13:30:04 UTC 2018
Permission denied (publickey).

[ec2-user@ip-10-255-255-200 ~]$ curl http://169.254.169.254/latest/meta-data/security-groups
TempSG1[ec2-user@ip-10-255-255-200 ~]$

----

Hmm... Toot!!!

P.S. Corrected diagram:

PhoneBoy · ‎2018-02-16

If you revert the routing on the instances back to what AWS does by default, what happens?

If this is working "securely" the traffic would not be permitted (i.e. the security groups in AWS should block it).

Vladimir · ‎2018-02-16

About to try it.

If my logic is not faulty, and if I am still thinking straight, shouldn't it be dropped unless specifically permitted in SGs attached to the instances?

Vladimir · ‎2018-02-16

Toot!, Toot! and one more Toot! for a good measure:

[ec2-user@ip-10-255-255-150 ~]$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 10.255.255.129 0.0.0.0 UG 0 0 0 eth0
10.255.255.128 * 255.255.255.192 U 0 0 0 eth0
169.254.169.254 * 255.255.255.255 UH 0 0 0 eth0
[ec2-user@ip-10-255-255-150 ~]$ ssh ec2-user@10.255.255.200
ssh: connect to host 10.255.255.200 port 22: Connection timed out
[ec2-user@ip-10-255-255-150 ~]$

----

[ec2-user@ip-10-255-255-200 ~]$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 10.255.255.193 0.0.0.0 UG 0 0 0 eth0
10.255.255.192 * 255.255.255.192 U 0 0 0 eth0
169.254.169.254 * 255.255.255.255 UH 0 0 0 eth0
[ec2-user@ip-10-255-255-200 ~]$ ssh ec2-user@10.255.255.150
ssh: connect to host 10.255.255.150 port 22: Connection timed out
[ec2-user@ip-10-255-255-200 ~]$

Vladimir · ‎2018-02-16

And with static routes reinstated:

[root@ip-10-255-255-150 ec2-user]# nano /etc/sysconfig/network-scripts/route-eth0
[root@ip-10-255-255-150 ec2-user]# reboot
[root@ip-10-255-255-150 ec2-user]#
Broadcast message from ec2-user@ip-10-255-255-150
(/dev/pts/0) at 16:54 ...

The system is going down for reboot NOW!
Using username "ec2-user".
Authenticating with public key "imported-openssh-key"
Last login: Fri Feb 16 16:42:28 2018 from xx.xx.xxx.98

__| __|_ )
_| ( / Amazon Linux AMI
___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2017.09-release-notes/
[ec2-user@ip-10-255-255-150 ~]$ ssh ec2-user@10.255.255.200
Permission denied (publickey).
[ec2-user@ip-10-255-255-150 ~]$

-----

[root@ip-10-255-255-200 ec2-user]# nano /etc/sysconfig/network-scripts/route-eth0
[root@ip-10-255-255-200 ec2-user]# reboot
[root@ip-10-255-255-200 ec2-user]#
Broadcast message from ec2-user@ip-10-255-255-200
(/dev/pts/0) at 16:55 ...

The system is going down for reboot NOW!
Using username "ec2-user".
Authenticating with public key "imported-openssh-key"
Last login: Fri Feb 16 16:42:10 2018 from xx.xx.xxx.98

__| __|_ )
_| ( / Amazon Linux AMI
___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2017.09-release-notes/
[ec2-user@ip-10-255-255-200 ~]$ ssh ec2-user@10.255.255.150
Permission denied (publickey).
[ec2-user@ip-10-255-255-200 ~]$

PhoneBoy · ‎2018-02-16

That does look promising

Vladimir · ‎2018-02-16

Do you want me to re-work the document into a more presentable form?

If so, what name would you suggest for it?

PhoneBoy · ‎2018-02-16

It probably wouldn't hurt

The existing name is ok.

PhoneBoy · ‎2018-02-23

While you get points for extreme cleverness, I will add a couple of caveats to this approach beyond what was already discussed:

Not sure this will work with a clustered gateway unless you also include scripts to monitor the "next hop" and adjust in case a cluster member fails. This is because, unlike traditional physical clusters, there is no cluster IP address with clusters in AWS (due to limitations in AWS).
It does require NICs in the relevant subnets (which kinda goes against the SDN philosophy and may also limit you to certain, larger, instance sizes).

Calvin_Nguyen · ‎2018-03-06

I am setting this environment up right now. Does the host need to have public temp IP? I am assuming no, correct?

Vladimir · ‎2018-03-06

If you are talking about EC2 instances, they do not have to have public IPs.

The reason for those in my lab was to gain the remote access to them bypassing vSEC for simplicity and to avoid a lockout should I bork something in the policy.

Calvin_Nguyen · ‎2018-03-06

Got it. What is the purpose of the 169.254.169.254 address in the route table?

Vladimir · ‎2018-03-06

That's the AWS' metadata lookup IP.

Hugh_McGauran · ‎2018-03-06

the big issue that i would have is that the routing is now back in the hands of the sysadmin... if they modify the routing they could bypass the gw no?

Calvin_Nguyen · ‎2018-03-06

I appreciate the help. I am having trouble setting up this lab config. I don't believe my subnets are even going through the GW at all. Below are screenshots that I have configured. I would love to have this setup to show to our partners so any help is appreciated:

Web Server 1:

Web Server 2:

GW:

Policy:

NAT:

Vladimir · ‎2018-03-06

I'm looking into it.

Can you confirm that you have at least Internet access through the vSEC from the hosts?

Would you mind dropping the vSEC's gaia config as well?

Calvin_Nguyen · ‎2018-03-06

Certainly.

I checked and I cannot go outbound from the host to Internet. I can access the Internet from CloudGuard GW. I suspect that I screwed up somewhere with the routing, but at the moment, I am in need of some fresh eyes.

The configuration of the CloudGuard GW is below:

#
# Configuration of GW10
# Language version: 13.1v1
#
# Exported by admin on Tue Mar 6 21:31:44 2018
#
set installer policy check-for-updates-period 3
set installer policy periodically-self-update on
set installer policy send-cpuse-data off
set installer policy self-test install-policy off
set installer policy self-test network-link-up off
set installer policy self-test start-processes on
set arp table cache-size 4096
set arp table validity-timeout 60
set arp announce 2
set message banner on

set message motd on

set message caption off
set core-dump enable
set core-dump total 1000
set core-dump per_process 2
set clienv debug 0
set clienv echo-cmd off
set clienv output pretty
set clienv prompt "%M"
set clienv rows 24
set clienv syntax-check off
add dhcp client interface eth2
set dhcp client interface eth2 timeout 60
set dhcp client interface eth2 retry 300
set dhcp client interface eth2 reboot 10
set dns primary 10.255.255.2
set edition 64-bit
set expert-password-hash $1$BBRaQSBB$WhNuGpNz5wTGlICO3YYDs/
set format date dd-mmm-yyyy
set format time 24-hour
set format netmask Dotted
set hostname GW10
add allowed-client host any-host
set web table-refresh-rate 15
set web session-timeout 30
set web ssl-port 443
set web ssl3-enabled off
set web daemon-enable on
set inactivity-timeout 10
set ipv6-state off
add command api path /bin/api_wrap description "Start, stop, or check status of API server"
add command tecli path /bin/tecli_start description "Threat Emulation Blade shell"
set net-access telnet off
set ntp active on
set ntp server primary 169.254.169.123 version 4
set ntp server secondary 0.pool.ntp.org version 4
set user admin shell /bin/bash
set user admin password-hash $1$abc$QuOUd9idETLHk/BuzRzYA1
set user monitor shell /etc/cli.sh
set user monitor password-hash *
set password-controls min-password-length 6
set password-controls complexity 2
set password-controls palindrome-check true
set password-controls history-checking true
set password-controls history-length 10
set password-controls password-expiration never
set password-controls expiration-warning-days 7
set password-controls expiration-lockout-days never
set password-controls force-change-when no
set password-controls deny-on-nonuse enable false
set password-controls deny-on-nonuse allowed-days 365
set password-controls deny-on-fail enable false
set password-controls deny-on-fail failures-allowed 10
set password-controls deny-on-fail allow-after 1200
set aaa tacacs-servers state off
set aaa radius-servers super-user-uid 96
set max-path-splits 8
set tracefile maxnum 10
set tracefile size 1
set syslog filename /var/log/messages
set syslog cplogs off
set syslog mgmtauditlogs on
set syslog auditlog permanent
set timezone Etc / UTC
set interface eth0 state on
set interface eth0 ipv4-address 10.255.255.45 mask-length 26
add interface eth0 alias 54.176.229.227/32
set interface eth1 link-speed 10G/full
set interface eth1 state on
set interface eth1 auto-negotiation on
set interface eth1 mtu 1500
set interface eth1 ipv4-address 10.255.255.213 mask-length 26
set interface eth2 state on
set interface eth2 auto-negotiation on
set interface eth2 mtu 1500
set interface lo state on
set interface lo ipv4-address 127.0.0.1 mask-length 8
set inbound-route-filter ospf2 accept-all-ipv4
set inbound-route-filter rip accept-all-ipv4
set management interface eth0
set ospf area backbone on
set rip update-interval default
set rip expire-interval default
set snmp mode default
set snmp agent off
set snmp agent-version v3-Only
set snmp traps trap authorizationError disable
set snmp traps trap biosFailure disable
set snmp traps trap coldStart disable
set snmp traps trap configurationChange disable
set snmp traps trap configurationSave disable
set snmp traps trap fanFailure disable
set snmp traps trap highVoltage disable
set snmp traps trap linkUpLinkDown disable
set snmp traps trap lowDiskSpace disable
set snmp traps trap lowVoltage disable
set snmp traps trap overTemperature disable
set snmp traps trap powerSupplyFailure disable
set snmp traps trap raidVolumeState disable
set snmp traps trap vrrpv2AuthFailure disable
set snmp traps trap vrrpv2NewMaster disable
set snmp traps trap vrrpv3NewMaster disable
set snmp traps trap vrrpv3ProtoError disable
set static-route default comment "To Subnet Router"
set static-route default nexthop gateway address 10.255.255.1 on
set static-route 10.255.255.128/26 comment "To Subnet Router"
set static-route 10.255.255.128/26 nexthop gateway address 10.255.255.129 on
set static-route 10.255.255.192/26 comment "To Subnet Router"
set static-route 10.255.255.192/26 nexthop gateway address 10.255.255.193 on

Vladimir · ‎2018-03-06

Let's try to figure it out.

1. set interface eth0 ipv4-address 10.255.255.45 mask-length 26
add interface eth0 alias 54.176.229.227/32

I suspect that this line is in error, since you are assigning AWS EIP to the 10.255.255.45. I am not sure what having the alias assigned to the interface will do in AWS environment, but rough logic is that the public IP is actually assigned to the IGW and your EIP association created a static NAT entry in it that is pointing it to your vSEC's external interface.

If you have another identical address assigned to the vSEC, it may cause hmm.. some issues.

2. One of the internal legs of your vSEC is acquiring IP from DHCP... :

add dhcp client interface eth2
set dhcp client interface eth2 timeout 60
set dhcp client interface eth2 retry 300
set dhcp client interface eth2 reboot 10

As such, it may cause inheritance of the routing data from AWS' VPC default router.

It is hard for me to decipher all of the implications of this without prolonged meditation session, but I am in favor of not doing it quite yet (at least until you'll get the rest of the things working).

But as this solution is relying on static routes on hosts pointing to correct and, supposedly, fixed IP of the vSEC's interface, unless you've implemented some-kind of additional logic on hosts to adjust the routing dynamically, let's stick with static IPs.

And I cannot refrain from asking the equivalent of "have you plugged it in?" question:

Are you positive that your vSEC and the hosts are in the same region?

Cheers,

Vladimir

Vladimir · ‎2018-03-06

Not unless they are also in control of the security groups and NACLs.

If you'l take a look above, you'll see that the removal of the static route from the hosts does not allow it to bypass Security Group rules that pretty much drops the traffic unless it comes from vSEC.

If the above mentioned "sysadmins" control Security Groups, VPC and host routing as well as NACLs, they can bypass vSEC any way.

Vladimir · ‎2018-03-06

BTW, I've just fired up this lab and it is still working as intended:

You may also want to check the topology of your vSEC:

Calvin_Nguyen · ‎2018-03-06

Hi Vladimir,

Thank you for your help. Unfortunately, even after trying the 3 steps above, my setup still does not work.

However, I believe I know what the problem is. For some reason, I cannot ping from Web Server 2 to the CloudGuard GW interface that leads to Web Server 1 (eth1). Ping succeeds from Web Server 2 to CloudGuard GW interface eth2, but for some reason fails on eth2.

10.255.255.141 is Web Server 2 trying to ping eth1 of CloudGuard GW.

10.255.255.142 is eth2 of CloudGuard GW. As you can see. Pinging within the same subnet range (10.255.255.128/26) works.

Odd part is that Web Server 1 is able to ping to both eth1 and eth2 of CloudGuard GW.

#
# Configuration of GW10
# Language version: 13.1v1
#
# Exported by admin on Tue Mar 6 22:56:11 2018
#
set installer policy check-for-updates-period 3
set installer policy periodically-self-update on
set installer policy send-cpuse-data off
set installer policy self-test install-policy off
set installer policy self-test network-link-up off
set installer policy self-test start-processes on
set arp table cache-size 4096
set arp table validity-timeout 60
set arp announce 2
set message banner on

set message motd on

set message caption off
set core-dump enable
set core-dump total 1000
set core-dump per_process 2
set clienv debug 0
set clienv echo-cmd off
set clienv output pretty
set clienv prompt "%M"
set clienv rows 24
set clienv syntax-check off
set dns primary 10.255.255.2
set edition 64-bit
set expert-password-hash $1$BBRaQSBB$WhNuGpNz5wTGlICO3YYDs/
set format date dd-mmm-yyyy
set format time 24-hour
set format netmask Dotted
set hostname GW10
add allowed-client host any-host
set web table-refresh-rate 15
set web session-timeout 30
set web ssl-port 443
set web ssl3-enabled off
set web daemon-enable on
set inactivity-timeout 10
set ipv6-state off
add command api path /bin/api_wrap description "Start, stop, or check status of API server"
add command tecli path /bin/tecli_start description "Threat Emulation Blade shell"
set net-access telnet off
set ntp active on
set ntp server primary 169.254.169.123 version 4
set ntp server secondary 0.pool.ntp.org version 4
set user admin shell /bin/bash
set user admin password-hash $1$abc$QuOUd9idETLHk/BuzRzYA1
set user monitor shell /etc/cli.sh
set user monitor password-hash *
set password-controls min-password-length 6
set password-controls complexity 2
set password-controls palindrome-check true
set password-controls history-checking true
set password-controls history-length 10
set password-controls password-expiration never
set password-controls expiration-warning-days 7
set password-controls expiration-lockout-days never
set password-controls force-change-when no
set password-controls deny-on-nonuse enable false
set password-controls deny-on-nonuse allowed-days 365
set password-controls deny-on-fail enable false
set password-controls deny-on-fail failures-allowed 10
set password-controls deny-on-fail allow-after 1200
set aaa tacacs-servers state off
set aaa radius-servers super-user-uid 96
set max-path-splits 8
set tracefile maxnum 10
set tracefile size 1
set syslog filename /var/log/messages
set syslog cplogs off
set syslog mgmtauditlogs on
set syslog auditlog permanent
set timezone Etc / UTC
set interface eth0 link-speed 10G/full
set interface eth0 state on
set interface eth0 ipv4-address 10.255.255.45 mask-length 26
set interface eth1 link-speed 10G/full
set interface eth1 state on
set interface eth1 auto-negotiation on
set interface eth1 mtu 1500
set interface eth1 ipv4-address 10.255.255.213 mask-length 26
set interface eth2 comments "Eth2-10.255.255.128"
set interface eth2 link-speed 10G/full
set interface eth2 state on
set interface eth2 auto-negotiation on
set interface eth2 mtu 1500
set interface eth2 ipv4-address 10.255.255.142 mask-length 26
set interface lo state on
set interface lo ipv4-address 127.0.0.1 mask-length 8
set inbound-route-filter ospf2 accept-all-ipv4
set inbound-route-filter rip accept-all-ipv4
set management interface eth0
set ospf area backbone on
set rip update-interval default
set rip expire-interval default
set snmp mode default
set snmp agent off
set snmp agent-version v3-Only
set snmp traps trap authorizationError disable
set snmp traps trap biosFailure disable
set snmp traps trap coldStart disable
set snmp traps trap configurationChange disable
set snmp traps trap configurationSave disable
set snmp traps trap fanFailure disable
set snmp traps trap highVoltage disable
set snmp traps trap linkUpLinkDown disable
set snmp traps trap lowDiskSpace disable
set snmp traps trap lowVoltage disable
set snmp traps trap overTemperature disable
set snmp traps trap powerSupplyFailure disable
set snmp traps trap raidVolumeState disable
set snmp traps trap vrrpv2AuthFailure disable
set snmp traps trap vrrpv2NewMaster disable
set snmp traps trap vrrpv3NewMaster disable
set snmp traps trap vrrpv3ProtoError disable
set static-route default comment "To Subnet Router"
set static-route default nexthop gateway address 10.255.255.1 on
set static-route 10.255.255.128/26 comment "To Subnet Router"
set static-route 10.255.255.128/26 nexthop gateway address 10.255.255.129 on
set static-route 10.255.255.192/26 comment "To Subnet Router"
set static-route 10.255.255.192/26 nexthop gateway address 10.255.255.193 on

Calvin_Nguyen · ‎2018-03-06

I have verified that the above configuration matches mine.

Vladimir · ‎2018-03-06

Can you post the screenshots of your security groups and routing tables as well as verify that the Net_10.255.255.XX objects configured correctly?

Their names look right, but the actual networks assigned to them are in question.

Calvin_Nguyen · ‎2018-03-06

I have verified that the above configuration matches mine.

Are you a member of CheckMates?

Inspection of Inter-Subnet traffic in AWS VPC using CloudGuard