AnsweredAssumed Answered

Inconsistent behavior of vSEC in AWS

Question asked by Vladimir Yakovlev on Sep 27, 2017
Latest reply on Apr 17, 2018 by Iain King

Weirdest thing:

Immediately after policy load, test traffic succeeding.

Few minutes later, no go.

There is NO Dynamic routing involved.

Lab Setup

 

Good traffic immediately after policy load:


Client session:

[vladimir@centos7 ~]$ curl http://34.235.192.92/hello.html

<html><h1>Hello from Amazon EC201</h1></html>

[vladimir@centos7 ~]$


vSEC:

[Expert@vSEC01:0]# fw monitor -e 'accept port(80);'
monitor: getting filter (from command line)
monitor: compiling
monitorfilter:
Compiled OK.
monitor: loading
monitor: monitoring (control-C to stop)
[vs_0][fw_1] eth0:i[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=27946
TCP: 19023 -> 80 .S.... seq=e03cd91b ack=00000000
[vs_0][fw_1] eth0:I[60]: 24.89.129.101 -> 10.255.255.210 (TCP) len=60 id=27946
TCP: 19023 -> 80 .S.... seq=e03cd91b ack=00000000
[vs_0][fw_1] eth1:o[60]: 24.89.129.101 -> 10.255.255.210 (TCP) len=60 id=27946
TCP: 19023 -> 80 .S.... seq=e03cd91b ack=00000000
[vs_0][fw_1] eth1:O[60]: 24.89.129.101 -> 10.255.255.210 (TCP) len=60 id=27946
TCP: 19023 -> 80 .S.... seq=e03cd91b ack=00000000
[vs_0][fw_1] eth1:i[60]: 10.255.255.210 -> 24.89.129.101 (TCP) len=60 id=0
TCP: 80 -> 19023 .S..A. seq=8cd9c86d ack=e03cd91c
[vs_0][fw_1] eth1:I[60]: 10.255.255.210 -> 24.89.129.101 (TCP) len=60 id=0
TCP: 80 -> 19023 .S..A. seq=8cd9c86d ack=e03cd91c
[vs_0][fw_1] eth0:o[60]: 10.255.255.210 -> 24.89.129.101 (TCP) len=60 id=0
TCP: 80 -> 19023 .S..A. seq=8cd9c86d ack=e03cd91c
[vs_0][fw_1] eth0:O[60]: 10.255.255.23 -> 24.89.129.101 (TCP) len=60 id=0
TCP: 80 -> 19023 .S..A. seq=8cd9c86d ack=e03cd91c
[vs_0][fw_1] eth0:i[52]: 24.89.129.101 -> 10.255.255.23 (TCP) len=52 id=27947
TCP: 19023 -> 80 ....A. seq=e03cd91c ack=8cd9c86e
[vs_0][fw_1] eth0:I[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=27947
TCP: 19023 -> 80 ....A. seq=e03cd91c ack=8cd9c86e
[vs_0][fw_1] eth1:o[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=27947
TCP: 19023 -> 80 ....A. seq=e03cd91c ack=8cd9c86e
[vs_0][fw_1] eth1:O[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=27947
TCP: 19023 -> 80 ....A. seq=e03cd91c ack=8cd9c86e
[vs_0][fw_1] eth0:i[139]: 24.89.129.101 -> 10.255.255.23 (TCP) len=139 id=27948
TCP: 19023 -> 80 ...PA. seq=e03cd91c ack=8cd9c86e
[vs_0][fw_1] eth0:I[139]: 24.89.129.101 -> 10.255.255.210 (TCP) len=139 id=27948
TCP: 19023 -> 80 ...PA. seq=e03cd91c ack=8cd9c86e
[vs_0][fw_1] eth1:o[139]: 24.89.129.101 -> 10.255.255.210 (TCP) len=139 id=27948
TCP: 19023 -> 80 ...PA. seq=e03cd91c ack=8cd9c86e
[vs_0][fw_1] eth1:O[139]: 24.89.129.101 -> 10.255.255.210 (TCP) len=139 id=27948
TCP: 19023 -> 80 ...PA. seq=e03cd91c ack=8cd9c86e
[vs_0][fw_1] eth1:i[52]: 10.255.255.210 -> 24.89.129.101 (TCP) len=52 id=36280
TCP: 80 -> 19023 ....A. seq=8cd9c86e ack=e03cd973
[vs_0][fw_1] eth1:I[52]: 10.255.255.210 -> 24.89.129.101 (TCP) len=52 id=36280
TCP: 80 -> 19023 ....A. seq=8cd9c86e ack=e03cd973
[vs_0][fw_1] eth0:o[52]: 10.255.255.210 -> 24.89.129.101 (TCP) len=52 id=36280
TCP: 80 -> 19023 ....A. seq=8cd9c86e ack=e03cd973
[vs_0][fw_1] eth0:O[52]: 10.255.255.23 -> 24.89.129.101 (TCP) len=52 id=36280
TCP: 80 -> 19023 ....A. seq=8cd9c86e ack=e03cd973
[vs_0][fw_1] eth1:i[371]: 10.255.255.210 -> 24.89.129.101 (TCP) len=371 id=36281
TCP: 80 -> 19023 ...PA. seq=8cd9c86e ack=e03cd973
[vs_0][fw_1] eth1:I[371]: 10.255.255.210 -> 24.89.129.101 (TCP) len=371 id=36281
TCP: 80 -> 19023 ...PA. seq=8cd9c86e ack=e03cd973
[vs_0][fw_1] eth0:o[371]: 10.255.255.210 -> 24.89.129.101 (TCP) len=371 id=36281
TCP: 80 -> 19023 ...PA. seq=8cd9c86e ack=e03cd973
[vs_0][fw_1] eth0:O[371]: 10.255.255.23 -> 24.89.129.101 (TCP) len=371 id=36281
TCP: 80 -> 19023 ...PA. seq=8cd9c86e ack=e03cd973
[vs_0][fw_1] eth0:i[52]: 24.89.129.101 -> 10.255.255.23 (TCP) len=52 id=27949
TCP: 19023 -> 80 ....A. seq=e03cd973 ack=8cd9c9ad
[vs_0][fw_1] eth0:I[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=27949
TCP: 19023 -> 80 ....A. seq=e03cd973 ack=8cd9c9ad
[vs_0][fw_1] eth1:o[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=27949
TCP: 19023 -> 80 ....A. seq=e03cd973 ack=8cd9c9ad
[vs_0][fw_1] eth1:O[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=27949
TCP: 19023 -> 80 ....A. seq=e03cd973 ack=8cd9c9ad
[vs_0][fw_1] eth0:i[52]: 24.89.129.101 -> 10.255.255.23 (TCP) len=52 id=27950
TCP: 19023 -> 80 F...A. seq=e03cd973 ack=8cd9c9ad
[vs_0][fw_1] eth0:I[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=27950
TCP: 19023 -> 80 F...A. seq=e03cd973 ack=8cd9c9ad
[vs_0][fw_1] eth1:o[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=27950
TCP: 19023 -> 80 F...A. seq=e03cd973 ack=8cd9c9ad
[vs_0][fw_1] eth1:O[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=27950
TCP: 19023 -> 80 F...A. seq=e03cd973 ack=8cd9c9ad
[vs_0][fw_1] eth1:i[52]: 10.255.255.210 -> 24.89.129.101 (TCP) len=52 id=36282
TCP: 80 -> 19023 F...A. seq=8cd9c9ad ack=e03cd974
[vs_0][fw_1] eth1:I[52]: 10.255.255.210 -> 24.89.129.101 (TCP) len=52 id=36282
TCP: 80 -> 19023 F...A. seq=8cd9c9ad ack=e03cd974
[vs_0][fw_1] eth0:o[52]: 10.255.255.210 -> 24.89.129.101 (TCP) len=52 id=36282
TCP: 80 -> 19023 F...A. seq=8cd9c9ad ack=e03cd974
[vs_0][fw_1] eth0:O[52]: 10.255.255.23 -> 24.89.129.101 (TCP) len=52 id=36282
TCP: 80 -> 19023 F...A. seq=8cd9c9ad ack=e03cd974
[vs_0][fw_1] eth0:i[52]: 24.89.129.101 -> 10.255.255.23 (TCP) len=52 id=27951
TCP: 19023 -> 80 ....A. seq=e03cd974 ack=8cd9c9ae
[vs_0][fw_1] eth0:I[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=27951
TCP: 19023 -> 80 ....A. seq=e03cd974 ack=8cd9c9ae
[vs_0][fw_1] eth1:o[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=27951
TCP: 19023 -> 80 ....A. seq=e03cd974 ack=8cd9c9ae
[vs_0][fw_1] eth1:O[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=27951
TCP: 19023 -> 80 ....A. seq=e03cd974 ack=8cd9c9ae
monitor: caught sig 2
monitor: unloading
[Expert@vSEC01:0]#


Web Server on EC2 behind Logical Server:

[ec2-user@ip-10-255-255-210 ~]$ sudo tcpdump -n host 24.89.129.101
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
21:29:23.086278 IP 24.89.129.101.19023 > 10.255.255.210.http: Flags [S], seq 3762084123, win 29200, options
[mss 1460,sackOK,TS val 169391153 ecr 0,nop,wscale 7], length 0
21:29:23.086307 IP 10.255.255.210.http > 24.89.129.101.19023: Flags [S.], seq 2363082861, ack 3762084124,
win 26847, options [mss 8961,sackOK,TS val 2639313 ecr 169391153,nop,wscale 7], length 0
21:29:23.102856 IP 24.89.129.101.19023 > 10.255.255.210.http: Flags [.], ack 1, win 229, options
[nop,nop,TS val 169391175 ecr 2639313], length 0
21:29:23.106642 IP 24.89.129.101.19023 > 10.255.255.210.http: Flags [P.], seq 1:88, ack 1, win 229, options
[nop,nop,TS val 169391175 ecr 2639313], length 87
21:29:23.106660 IP 10.255.255.210.http > 24.89.129.101.19023: Flags [.], ack 88, win 210, options
[nop,nop,TS val 2639318 ecr 169391175], length 0
21:29:23.106875 IP 10.255.255.210.http > 24.89.129.101.19023: Flags [P.], seq 1:320, ack 88, win 210,
options [nop,nop,TS val 2639318 ecr 169391175], length 319
21:29:23.122734 IP 24.89.129.101.19023 > 10.255.255.210.http: Flags [.], ack 320, win 237, options
[nop,nop,TS val 169391195 ecr 2639318], length 0
21:29:23.124526 IP 24.89.129.101.19023 > 10.255.255.210.http: Flags [F.], seq 88, ack 320, win 237, options
[nop,nop,TS val 169391195 ecr 2639318], length 0
21:29:23.124553 IP 10.255.255.210.http > 24.89.129.101.19023: Flags [F.], seq 320, ack 89, win 210, options
[nop,nop,TS val 2639322 ecr 169391195], length 0
21:29:23.141031 IP 24.89.129.101.19023 > 10.255.255.210.http: Flags [.], ack 321, win 237, options
[nop,nop,TS val 169391213 ecr 2639322], length 0
^C
10 packets captured
10 packets received by filter
0 packets dropped by kernel
[ec2-user@ip-10-255-255-210 ~]$

 


Few minutes later:


Client session:

[vladimir@centos7 ~]$ curl http://34.235.192.92/hello.html

curl: (7) Failed connect to 34.235.192.92:80; Connection timed out
[vladimir@centos7 ~]$


vSEC:

[Expert@vSEC01:0]# fw monitor -e 'accept port(80);'
monitor: getting filter (from command line)
monitor: compiling
monitorfilter:
Compiled OK.
monitor: loading
monitor: monitoring (control-C to stop)
[vs_0][fw_1] eth0:i[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=21286
TCP: 19024 -> 80 .S.... seq=1901a5b5 ack=00000000
[vs_0][fw_0] eth0:i[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=21287
TCP: 19024 -> 80 .S.... seq=1901a5b5 ack=00000000
[vs_0][fw_0] eth0:i[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=21288
TCP: 19024 -> 80 .S.... seq=1901a5b5 ack=00000000
[vs_0][fw_0] eth0:i[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=21289
TCP: 19024 -> 80 .S.... seq=1901a5b5 ack=00000000
[vs_0][fw_0] eth0:i[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=21290
TCP: 19024 -> 80 .S.... seq=1901a5b5 ack=00000000
[vs_0][fw_1] eth0:i[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=21291
TCP: 19024 -> 80 .S.... seq=1901a5b5 ack=00000000
[vs_0][fw_1] eth0:i[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=21292
TCP: 43813 -> 80 .S.... seq=1901a5b5 ack=00000000
^C monitor: caught sig 2
monitor: unloading
[Expert@vSEC01:0]#


Web Server on EC2 behind Logical Server:

[ec2-user@ip-10-255-255-210 ~]$ sudo tcpdump -n host 24.89.129.101
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
[ec2-user@ip-10-255-255-210 ~]$

 

 

Any ideas?

Outcomes