Solved: Re: Inconsistent behavior of vSEC in AWS

Vladimir · ‎2017-09-27

Weirdest thing:

Immediately after policy load, test traffic succeeding.

Few minutes later, no go.

There is NO Dynamic routing involved.

Good traffic immediately after policy load:

Client session:

[vladimir@centos7 ~]$ curl http://34.235.192.92/hello.html

<html><h1>Hello from Amazon EC201</h1></html>

[vladimir@centos7 ~]$

vSEC:

[Expert@vSEC01:0]# fw monitor -e 'accept port(80);'
monitor: getting filter (from command line)
monitor: compiling
monitorfilter:
Compiled OK.
monitor: loading
monitor: monitoring (control-C to stop)
[vs_0][fw_1] eth0:i[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=27946
TCP: 19023 -> 80 .S.... seq=e03cd91b ack=00000000
[vs_0][fw_1] eth0:I[60]: 24.89.129.101 -> 10.255.255.210 (TCP) len=60 id=27946
TCP: 19023 -> 80 .S.... seq=e03cd91b ack=00000000
[vs_0][fw_1] eth1:o[60]: 24.89.129.101 -> 10.255.255.210 (TCP) len=60 id=27946
TCP: 19023 -> 80 .S.... seq=e03cd91b ack=00000000
[vs_0][fw_1] eth1:O[60]: 24.89.129.101 -> 10.255.255.210 (TCP) len=60 id=27946
TCP: 19023 -> 80 .S.... seq=e03cd91b ack=00000000
[vs_0][fw_1] eth1:i[60]: 10.255.255.210 -> 24.89.129.101 (TCP) len=60 id=0
TCP: 80 -> 19023 .S..A. seq=8cd9c86d ack=e03cd91c
[vs_0][fw_1] eth1:I[60]: 10.255.255.210 -> 24.89.129.101 (TCP) len=60 id=0
TCP: 80 -> 19023 .S..A. seq=8cd9c86d ack=e03cd91c
[vs_0][fw_1] eth0:o[60]: 10.255.255.210 -> 24.89.129.101 (TCP) len=60 id=0
TCP: 80 -> 19023 .S..A. seq=8cd9c86d ack=e03cd91c
[vs_0][fw_1] eth0:O[60]: 10.255.255.23 -> 24.89.129.101 (TCP) len=60 id=0
TCP: 80 -> 19023 .S..A. seq=8cd9c86d ack=e03cd91c
[vs_0][fw_1] eth0:i[52]: 24.89.129.101 -> 10.255.255.23 (TCP) len=52 id=27947
TCP: 19023 -> 80 ....A. seq=e03cd91c ack=8cd9c86e
[vs_0][fw_1] eth0:I[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=27947
TCP: 19023 -> 80 ....A. seq=e03cd91c ack=8cd9c86e
[vs_0][fw_1] eth1:o[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=27947
TCP: 19023 -> 80 ....A. seq=e03cd91c ack=8cd9c86e
[vs_0][fw_1] eth1:O[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=27947
TCP: 19023 -> 80 ....A. seq=e03cd91c ack=8cd9c86e
[vs_0][fw_1] eth0:i[139]: 24.89.129.101 -> 10.255.255.23 (TCP) len=139 id=27948
TCP: 19023 -> 80 ...PA. seq=e03cd91c ack=8cd9c86e
[vs_0][fw_1] eth0:I[139]: 24.89.129.101 -> 10.255.255.210 (TCP) len=139 id=27948
TCP: 19023 -> 80 ...PA. seq=e03cd91c ack=8cd9c86e
[vs_0][fw_1] eth1:o[139]: 24.89.129.101 -> 10.255.255.210 (TCP) len=139 id=27948
TCP: 19023 -> 80 ...PA. seq=e03cd91c ack=8cd9c86e
[vs_0][fw_1] eth1:O[139]: 24.89.129.101 -> 10.255.255.210 (TCP) len=139 id=27948
TCP: 19023 -> 80 ...PA. seq=e03cd91c ack=8cd9c86e
[vs_0][fw_1] eth1:i[52]: 10.255.255.210 -> 24.89.129.101 (TCP) len=52 id=36280
TCP: 80 -> 19023 ....A. seq=8cd9c86e ack=e03cd973
[vs_0][fw_1] eth1:I[52]: 10.255.255.210 -> 24.89.129.101 (TCP) len=52 id=36280
TCP: 80 -> 19023 ....A. seq=8cd9c86e ack=e03cd973
[vs_0][fw_1] eth0:o[52]: 10.255.255.210 -> 24.89.129.101 (TCP) len=52 id=36280
TCP: 80 -> 19023 ....A. seq=8cd9c86e ack=e03cd973
[vs_0][fw_1] eth0:O[52]: 10.255.255.23 -> 24.89.129.101 (TCP) len=52 id=36280
TCP: 80 -> 19023 ....A. seq=8cd9c86e ack=e03cd973
[vs_0][fw_1] eth1:i[371]: 10.255.255.210 -> 24.89.129.101 (TCP) len=371 id=36281
TCP: 80 -> 19023 ...PA. seq=8cd9c86e ack=e03cd973
[vs_0][fw_1] eth1:I[371]: 10.255.255.210 -> 24.89.129.101 (TCP) len=371 id=36281
TCP: 80 -> 19023 ...PA. seq=8cd9c86e ack=e03cd973
[vs_0][fw_1] eth0:o[371]: 10.255.255.210 -> 24.89.129.101 (TCP) len=371 id=36281
TCP: 80 -> 19023 ...PA. seq=8cd9c86e ack=e03cd973
[vs_0][fw_1] eth0:O[371]: 10.255.255.23 -> 24.89.129.101 (TCP) len=371 id=36281
TCP: 80 -> 19023 ...PA. seq=8cd9c86e ack=e03cd973
[vs_0][fw_1] eth0:i[52]: 24.89.129.101 -> 10.255.255.23 (TCP) len=52 id=27949
TCP: 19023 -> 80 ....A. seq=e03cd973 ack=8cd9c9ad
[vs_0][fw_1] eth0:I[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=27949
TCP: 19023 -> 80 ....A. seq=e03cd973 ack=8cd9c9ad
[vs_0][fw_1] eth1:o[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=27949
TCP: 19023 -> 80 ....A. seq=e03cd973 ack=8cd9c9ad
[vs_0][fw_1] eth1:O[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=27949
TCP: 19023 -> 80 ....A. seq=e03cd973 ack=8cd9c9ad
[vs_0][fw_1] eth0:i[52]: 24.89.129.101 -> 10.255.255.23 (TCP) len=52 id=27950
TCP: 19023 -> 80 F...A. seq=e03cd973 ack=8cd9c9ad
[vs_0][fw_1] eth0:I[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=27950
TCP: 19023 -> 80 F...A. seq=e03cd973 ack=8cd9c9ad
[vs_0][fw_1] eth1:o[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=27950
TCP: 19023 -> 80 F...A. seq=e03cd973 ack=8cd9c9ad
[vs_0][fw_1] eth1:O[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=27950
TCP: 19023 -> 80 F...A. seq=e03cd973 ack=8cd9c9ad
[vs_0][fw_1] eth1:i[52]: 10.255.255.210 -> 24.89.129.101 (TCP) len=52 id=36282
TCP: 80 -> 19023 F...A. seq=8cd9c9ad ack=e03cd974
[vs_0][fw_1] eth1:I[52]: 10.255.255.210 -> 24.89.129.101 (TCP) len=52 id=36282
TCP: 80 -> 19023 F...A. seq=8cd9c9ad ack=e03cd974
[vs_0][fw_1] eth0:o[52]: 10.255.255.210 -> 24.89.129.101 (TCP) len=52 id=36282
TCP: 80 -> 19023 F...A. seq=8cd9c9ad ack=e03cd974
[vs_0][fw_1] eth0:O[52]: 10.255.255.23 -> 24.89.129.101 (TCP) len=52 id=36282
TCP: 80 -> 19023 F...A. seq=8cd9c9ad ack=e03cd974
[vs_0][fw_1] eth0:i[52]: 24.89.129.101 -> 10.255.255.23 (TCP) len=52 id=27951
TCP: 19023 -> 80 ....A. seq=e03cd974 ack=8cd9c9ae
[vs_0][fw_1] eth0:I[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=27951
TCP: 19023 -> 80 ....A. seq=e03cd974 ack=8cd9c9ae
[vs_0][fw_1] eth1:o[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=27951
TCP: 19023 -> 80 ....A. seq=e03cd974 ack=8cd9c9ae
[vs_0][fw_1] eth1:O[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=27951
TCP: 19023 -> 80 ....A. seq=e03cd974 ack=8cd9c9ae
monitor: caught sig 2
monitor: unloading
[Expert@vSEC01:0]#

Web Server on EC2 behind Logical Server:

[ec2-user@ip-10-255-255-210 ~]$ sudo tcpdump -n host 24.89.129.101
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
21:29:23.086278 IP 24.89.129.101.19023 > 10.255.255.210.http: Flags [S], seq 3762084123, win 29200, options
[mss 1460,sackOK,TS val 169391153 ecr 0,nop,wscale 7], length 0
21:29:23.086307 IP 10.255.255.210.http > 24.89.129.101.19023: Flags [S.], seq 2363082861, ack 3762084124,
win 26847, options [mss 8961,sackOK,TS val 2639313 ecr 169391153,nop,wscale 7], length 0
21:29:23.102856 IP 24.89.129.101.19023 > 10.255.255.210.http: Flags [.], ack 1, win 229, options
[nop,nop,TS val 169391175 ecr 2639313], length 0
21:29:23.106642 IP 24.89.129.101.19023 > 10.255.255.210.http: Flags [P.], seq 1:88, ack 1, win 229, options
[nop,nop,TS val 169391175 ecr 2639313], length 87
21:29:23.106660 IP 10.255.255.210.http > 24.89.129.101.19023: Flags [.], ack 88, win 210, options
[nop,nop,TS val 2639318 ecr 169391175], length 0
21:29:23.106875 IP 10.255.255.210.http > 24.89.129.101.19023: Flags [P.], seq 1:320, ack 88, win 210,
options [nop,nop,TS val 2639318 ecr 169391175], length 319
21:29:23.122734 IP 24.89.129.101.19023 > 10.255.255.210.http: Flags [.], ack 320, win 237, options
[nop,nop,TS val 169391195 ecr 2639318], length 0
21:29:23.124526 IP 24.89.129.101.19023 > 10.255.255.210.http: Flags [F.], seq 88, ack 320, win 237, options
[nop,nop,TS val 169391195 ecr 2639318], length 0
21:29:23.124553 IP 10.255.255.210.http > 24.89.129.101.19023: Flags [F.], seq 320, ack 89, win 210, options
[nop,nop,TS val 2639322 ecr 169391195], length 0
21:29:23.141031 IP 24.89.129.101.19023 > 10.255.255.210.http: Flags [.], ack 321, win 237, options
[nop,nop,TS val 169391213 ecr 2639322], length 0
^C
10 packets captured
10 packets received by filter
0 packets dropped by kernel
[ec2-user@ip-10-255-255-210 ~]$

Few minutes later:

Client session:

[vladimir@centos7 ~]$ curl http://34.235.192.92/hello.html

curl: (7) Failed connect to 34.235.192.92:80; Connection timed out
[vladimir@centos7 ~]$

vSEC:

[Expert@vSEC01:0]# fw monitor -e 'accept port(80);'
monitor: getting filter (from command line)
monitor: compiling
monitorfilter:
Compiled OK.
monitor: loading
monitor: monitoring (control-C to stop)
[vs_0][fw_1] eth0:i[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=21286
TCP: 19024 -> 80 .S.... seq=1901a5b5 ack=00000000
[vs_0][fw_0] eth0:i[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=21287
TCP: 19024 -> 80 .S.... seq=1901a5b5 ack=00000000
[vs_0][fw_0] eth0:i[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=21288
TCP: 19024 -> 80 .S.... seq=1901a5b5 ack=00000000
[vs_0][fw_0] eth0:i[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=21289
TCP: 19024 -> 80 .S.... seq=1901a5b5 ack=00000000
[vs_0][fw_0] eth0:i[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=21290
TCP: 19024 -> 80 .S.... seq=1901a5b5 ack=00000000
[vs_0][fw_1] eth0:i[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=21291
TCP: 19024 -> 80 .S.... seq=1901a5b5 ack=00000000
[vs_0][fw_1] eth0:i[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=21292
TCP: 43813 -> 80 .S.... seq=1901a5b5 ack=00000000
^C monitor: caught sig 2
monitor: unloading
[Expert@vSEC01:0]#

Web Server on EC2 behind Logical Server:

[ec2-user@ip-10-255-255-210 ~]$ sudo tcpdump -n host 24.89.129.101
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
[ec2-user@ip-10-255-255-210 ~]$

Any ideas?

Vladimir · ‎2017-12-21

Update: The reason for the timeout appears to be related to the ICMP health probe the gateway is running on the target servers. In the absence of ICMP echo request in the Security Group the servers belong to, after one minute Logical Server object deems server as not reachable. If ICMP echo request is permitted, it works without timeout.

I am still waiting for the response from R&D in regards to supported modes of load balancing in AWS using Logical Servers and will update this thread once I receive it.

For now, it was successfully tested using:

Service Type: Other

Server Group: [Simple Group with target servers]

Use persistent server mode: True

Persistency by server

Balance method: Round trip

View solution in original post

Danny · ‎2017-09-27

After you load the policy fw monitor shows:

eth0:i: 24.89.129.101 -> 10.255.255.23   TCP: 19023 -> 80 SYN inbound
eth0:I: 24.89.129.101 -> 10.255.255.210  TCP: 19023 -> 80 SYN Inbound
eth1:o: 24.89.129.101 -> 10.255.255.210  TCP: 19023 -> 80 SYN outbound
eth1:O: 24.89.129.101 -> 10.255.255.210  TCP: 19023 -> 80 SYN Outbound

and a couple minutes later only:

eth0:i: 24.89.129.101 -> 10.255.255.23   TCP: 19024 -> 80 SYN
eth0:I: missing

In order to troubleshoot where the traffic is lost on the Inbound interface eth0:I I suggest running:

fwaccel stat (check if it's active)
fw ctl set int fw_antispoofing_enabled 0; sim feature anti_spoofing off; fwaccel off
fw ctl zdebug drop | grep 24.89.129.101
fw ctl set int fw_antispoofing_enabled 1; sim feature anti_spoofing on
fwaccel on (if it was active initially)‍‍‍‍‍

Also check your SmartLog for src:24.89.129.101 AND http and reply with the results.

Vladimir · ‎2017-09-27

Danny, thank you for trying to puzzle it out with me.

The acceleration was off, but nonetheless:

[Expert@vSEC01:0]# fwaccel stat
Accelerator Status : off

Accelerator Features : Accounting, NAT, Cryptography, Routing,
HasClock, Templates, Synchronous, IdleDetection,
Sequencing, TcpStateDetect, AutoExpire,
DelayedNotif, TcpStateDetectV2, CPLS, McastRouting,
WireMode, DropTemplates, NatTemplates,
Streaming, MultiFW, AntiSpoofing, Nac,
ViolationStats, AsychronicNotif, ERDOS,
McastRoutingV2, NMR, NMT, NAT64, GTPAcceleration,
SCTPAcceleration
Cryptography Features Mask : not available
[Expert@vSEC01:0]#

[Expert@vSEC01:0]# fw ctl zdebug drop | grep 24.89.129.101
;[cpu_1];[fw4_0];fw_log_drop_ex: Packet proto=6 24.89.129.101:43817 -> 10.255.255.23:80 dropped by fw_send_log_drop Reason: Rulebase drop - on layer "Network" rule 1;
;[cpu_0];[fw4_1];fw_log_drop_ex: Packet proto=6 24.89.129.101:43817 -> 10.255.255.23:80 dropped by fw_send_log_drop Reason: Rulebase drop - on layer "Network" rule 1;
;[cpu_0];[fw4_1];fw_log_drop_ex: Packet proto=6 24.89.129.101:43817 -> 10.255.255.23:80 dropped by fw_send_log_drop Reason: Rulebase drop - on layer "Network" rule 1;
;[cpu_1];[fw4_0];fw_log_drop_ex: Packet proto=6 24.89.129.101:43817 -> 10.255.255.23:80 dropped by fw_send_log_drop Reason: Rulebase drop - on layer "Network" rule 1;
;[cpu_0];[fw4_1];fw_log_drop_ex: Packet proto=6 24.89.129.101:43817 -> 10.255.255.23:80 dropped by fw_send_log_drop Reason: Rulebase drop - on layer "Network" rule 1;

> show access-rule name "Web-Server-Access" layer "Network"

uid: "e6065103-64ed-468f-aff8-db4cb3cdd860"
name: "Web-Server-Access"
type: "access-rule"
domain:
uid: "41e821a0-3720-11e3-aa6e-0800200c9fde"
name: "SMC User"
domain-type: "domain"
track:
type:
uid: "598ead32-aa42-4615-90ed-f51a5928d41d"
name: "Log"
type: "Track"
domain:
uid: "a0bbbc99-adef-4ef8-bb6d-defdefdefdef"
name: "Check Point Data"
domain-type: "data domain"
per-session: false
per-connection: true
accounting: false
alert: "none"
layer: "94b3713a-be33-4cfc-adb4-b25b58fe23c2"
source:
- uid: "97aeb369-9aea-11d5-bd16-0090272ccb30"
name: "Any"
type: "CpmiAnyObject"
domain:
uid: "a0bbbc99-adef-4ef8-bb6d-defdefdefdef"
name: "Check Point Data"
domain-type: "data domain"
source-negate: false
destination:
- uid: "adcc18e6-3220-4b6a-a48f-0027de013bca"
name: "Simple01-LogicalServer-Web"
type: "CpmiLogicalServer"
domain:
uid: "41e821a0-3720-11e3-aa6e-0800200c9fde"
name: "SMC User"
domain-type: "domain"
destination-negate: false
service:
- uid: "97aeb3d4-9aea-11d5-bd16-0090272ccb30"
name: "http"
type: "service-tcp"
domain:
uid: "a0bbbc99-adef-4ef8-bb6d-defdefdefdef"
name: "Check Point Data"
domain-type: "data domain"
port: "80"
- uid: "97aeb443-9aea-11d5-bd16-0090272ccb30"
name: "https"
type: "service-tcp"
domain:
uid: "a0bbbc99-adef-4ef8-bb6d-defdefdefdef"
name: "Check Point Data"
domain-type: "data domain"
port: "443"
service-negate: false
vpn:
- uid: "97aeb369-9aea-11d5-bd16-0090272ccb30"
name: "Any"
type: "CpmiAnyObject"
domain:
uid: "a0bbbc99-adef-4ef8-bb6d-defdefdefdef"
name: "Check Point Data"
domain-type: "data domain"
action:
uid: "6c488338-8eec-4103-ad21-cd461ac2c472"
name: "Accept"
type: "RulebaseAction"
domain:
uid: "a0bbbc99-adef-4ef8-bb6d-defdefdefdef"
name: "Check Point Data"
domain-type: "data domain"
action-settings:
enable-identity-captive-portal: false
content:
- uid: "97aeb369-9aea-11d5-bd16-0090272ccb30"
name: "Any"
type: "CpmiAnyObject"
domain:
uid: "a0bbbc99-adef-4ef8-bb6d-defdefdefdef"
name: "Check Point Data"
domain-type: "data domain"
content-negate: false
content-direction: "any"
time:
- uid: "97aeb369-9aea-11d5-bd16-0090272ccb30"
name: "Any"
type: "CpmiAnyObject"
domain:
uid: "a0bbbc99-adef-4ef8-bb6d-defdefdefdef"
name: "Check Point Data"
domain-type: "data domain"
custom-fields:
field-1: ""
field-2: ""
field-3: ""
meta-info:
lock: "unlocked"
validation-state: "ok"
last-modify-time:
posix: 1506546973191
iso-8601: "2017-09-27T17:16-0400"
last-modifier: "admin"
creation-time:
posix: 1506355676576
iso-8601: "2017-09-25T12:07-0400"
creator: "admin"
comments: ""
enabled: true
install-on:
- uid: "11893d30-4594-4281-be49-8e50cf40244d"
name: "vSEC01"
type: "simple-gateway"
domain:
uid: "41e821a0-3720-11e3-aa6e-0800200c9fde"
name: "SMC User"
domain-type: "domain"

Still no idea...

Vladimir · ‎2017-09-27

P.S. The SmartLog does not log these attempts at all, even with the Any--Any--Drop--Log.

Danny · ‎2017-09-27

Ok, since this is a rulebase drop, this relates to either the destination object 'Simple01-LogicalServer-Web' or the http/https services.

Related SKs (old): sk100636, sk97876

Let's try to replace the destination object 'Simple01-LogicalServer-Web' with 'Any' and test again.

As you are missing log entries, please make sure your Global Properties > Implied Rules are set to be logged.

Vladimir · ‎2017-09-27

Well, with "Any" it looks much the same:

[Expert@vSEC01:0]# fwaccel off
SecureXL device is not active.
[Expert@vSEC01:0]# fw ctl zdebug drop | grep 24.89.129.101
;[cpu_0];[fw4_1];fw_log_drop_ex: Packet proto=6 24.89.129.101:43819 -> 10.255.255.23:80 dropped by fw_send_log_drop Reason: Rulebase drop - on layer "Network" rule 1;
;[cpu_0];[fw4_1];fw_log_drop_ex: Packet proto=6 24.89.129.101:43819 -> 10.255.255.23:80 dropped by fw_send_log_drop Reason: Rulebase drop - on layer "Network" rule 1;
;[cpu_1];[fw4_0];fw_log_drop_ex: Packet proto=6 24.89.129.101:43819 -> 10.255.255.23:80 dropped by fw_send_log_drop Reason: Rulebase drop - on layer "Network" rule 1;
;[cpu_0];[fw4_1];fw_log_drop_ex: Packet proto=6 24.89.129.101:43819 -> 10.255.255.23:80 dropped by fw_send_log_drop Reason: Rulebase drop - on layer "Network" rule 1;
;[cpu_1];[fw4_0];fw_log_drop_ex: Packet proto=6 24.89.129.101:43819 -> 10.255.255.23:80 dropped by fw_send_log_drop Reason: Rulebase drop - on layer "Network" rule 1;
[Expert@vSEC01:0]#

As if the Logical Server object doesn't matter.

I didn't expect this test to succeed, as there is no Static Nat present for the Web Server nested in the Simple Group defined in the Logical Server.

Danny · ‎2017-09-27

Let's put a new rule above this one allowing only echo-request to 'Simple01-LogicalServer-Web'.

Create a new TCP service called http_80 and https_443 with their specific ports and don't touch the advanced settings. Put 'Simple01-LogicalServer-Web' back into the destination of rule #2 and replace http/https with the new services http_80/https_443.

Then try reaching the web server via http and icmp ping again.

Vladimir · ‎2017-09-27

Policy installed with warnings, which is not surprising:

[Expert@vSEC01:0]# fwaccel stat
Accelerator Status : off

Accelerator Features : Accounting, NAT, Cryptography, Routing,
HasClock, Templates, Synchronous, IdleDetection,
Sequencing, TcpStateDetect, AutoExpire,
DelayedNotif, TcpStateDetectV2, CPLS, McastRouting,
WireMode, DropTemplates, NatTemplates,
Streaming, MultiFW, AntiSpoofing, Nac,
ViolationStats, AsychronicNotif, ERDOS,
McastRoutingV2, NMR, NMT, NAT64, GTPAcceleration,
SCTPAcceleration
Cryptography Features Mask : not available
[Expert@vSEC01:0]# fw ctl zdebug drop | grep 24.89.129.101
;[cpu_1];[fw4_0];fw_log_drop_ex: Packet proto=6 24.89.129.101:19033 -> 10.255.255.23:80 dropped by fw_send_log_drop Reason: Rulebase drop - on layer "Network" rule 2;
;[cpu_1];[fw4_0];fw_log_drop_ex: Packet proto=6 24.89.129.101:19033 -> 10.255.255.23:80 dropped by fw_send_log_drop Reason: Rulebase drop - on layer "Network" rule 2;
;[cpu_0];[fw4_1];fw_log_drop_ex: Packet proto=6 24.89.129.101:19033 -> 10.255.255.23:80 dropped by fw_send_log_drop Reason: Rulebase drop - on layer "Network" rule 2;
;[cpu_1];[fw4_0];fw_log_drop_ex: Packet proto=6 24.89.129.101:19033 -> 10.255.255.23:80 dropped by fw_send_log_drop Reason: Rulebase drop - on layer "Network" rule 2;
;[cpu_0];[fw4_1];fw_log_drop_ex: Packet proto=6 24.89.129.101:19033 -> 10.255.255.23:80 dropped by fw_send_log_drop Reason: Rulebase drop - on layer "Network" rule 2;
;[cpu_1];[fw4_0];fw_log_drop_ex: Packet proto=1 24.89.129.101:2048 -> 10.255.255.23:9902 dropped by fw_send_log_drop Reason: Rulebase drop - on layer "Network" rule 1;
;[cpu_1];[fw4_0];fw_log_drop_ex: Packet proto=1 24.89.129.101:2048 -> 10.255.255.23:17583 dropped by fw_send_log_drop Reason: Rulebase drop - on layer "Network" rule 1;
;[cpu_1];[fw4_0];fw_log_drop_ex: Packet proto=1 24.89.129.101:2048 -> 10.255.255.23:9902 dropped by fw_send_log_drop Reason: Rulebase drop - on layer "Network" rule 1;
;[cpu_1];[fw4_0];fw_log_drop_ex: Packet proto=1 24.89.129.101:2048 -> 10.255.255.23:18349 dropped by fw_send_log_drop Reason: Rulebase drop - on layer "Network" rule 1;
;[cpu_0];[fw4_1];fw_log_drop_ex: Packet proto=1 24.89.129.101:2048 -> 10.255.255.23:17836 dropped by fw_send_log_drop Reason: Rulebase drop - on layer "Network" rule 1;
;[cpu_0];[fw4_1];fw_log_drop_ex: Packet proto=1 24.89.129.101:2048 -> 10.255.255.23:19627 dropped by fw_send_log_drop Reason: Rulebase drop - on layer "Network" rule 1;

Vladimir · ‎2017-09-27

P.S. I haven't changed the Advanced parameters, but in the protocols' properties I did specify the http and https:

PhoneBoy · ‎2017-09-27

Curious if the output of fw stat changes when it's working and when it's not.

Vladimir · ‎2017-09-28

You may be onto something, although I am not sure what exactly.

There was a change of the "fw stat" output, but it did not correlate exactly with the failure of the traffic passing through the firewall:

After policy load, the status shows the eth0, but not the [<eth1] briefly:

[Expert@vSEC01:0]# fw stat
HOST POLICY DATE
localhost Standard 28Sep2017 8:22:23 : [>eth0] [<eth0] [<eth1]
[Expert@vSEC01:0]# fw stat
HOST POLICY DATE
localhost Standard 28Sep2017 8:26:52 : [>eth0] [<eth0]
[Expert@vSEC01:0]# fw stat
HOST POLICY DATE
localhost Standard 28Sep2017 8:26:52 : [>eth0] [<eth0] [<eth1]
[Expert@vSEC01:0]# fw stat
HOST POLICY DATE
localhost Standard 28Sep2017 8:26:52 : [>eth0] [<eth0] [<eth1]
[Expert@vSEC01:0]# fw stat
HOST POLICY DATE
localhost Standard 28Sep2017 8:26:52 : [>eth0] [<eth0] [<eth1]
[Expert@vSEC01:0]#

Running curls from the client, the failure was observed later on, when the [<eth1] re-appeared in the fw stat output:

[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
curl: (7) Failed connect to 34.235.192.92:80; Connection timed out
[root@centos7 ~]#

Just to make sure all are on the same page, the environment was re-set back to the original with stock HTTP and HTTPS services in the Rule 1:

Vladimir · ‎2017-09-28

Just re-run the test and the fw stat output stays the same during traffic traversal failure:

-----------------------------------------------------------------------------------------

Policy Installation (completed)

-----------------------------------------------------------------------------------------

[Expert@vSEC01:0]# fw stat
HOST POLICY DATE
localhost Standard 28Sep2017 8:40:21 : [>eth0] [<eth0] [<eth1]
[Expert@vSEC01:0]# fw stat
HOST POLICY DATE
localhost Standard 28Sep2017 8:40:21 : [>eth0] [<eth0] [<eth1]
[Expert@vSEC01:0]# fw stat
HOST POLICY DATE
localhost Standard 28Sep2017 8:40:21 : [>eth0] [<eth0] [<eth1]
[Expert@vSEC01:0]# fw stat
HOST POLICY DATE
localhost Standard 28Sep2017 8:40:21 : [>eth0] [<eth0] [<eth1]
[Expert@vSEC01:0]# fw stat
HOST POLICY DATE
localhost Standard 28Sep2017 8:40:21 : [>eth0] [<eth0] [<eth1]
[Expert@vSEC01:0]#

curl: (7) Failed connect to 34.235.192.92:80; Connection timed out

-----------------------------------------------------------------------------------------

Policy Installation (completed)

-----------------------------------------------------------------------------------------
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html
<html><h1>Hello from Amazon EC201</h1></html>
[root@centos7 ~]# curl http://34.235.192.92/hello.html

-----------------------------------------------------------------------------------------

Timeout

-----------------------------------------------------------------------------------------

^C
[root@centos7 ~]#

Vladimir · ‎2017-09-28

Dameon,

there is obviously a change of state happening on the vSEC shortly after policy load. How can we log the event for further analysis?

PhoneBoy · ‎2017-09-28

fw stat shows when the last policy was installed.

If the date changes, it suggests the policy was reloaded.

The observed behavior suggests the policy is changing (to what and why I’m not sure).

As troubleshooting steps, you might try (test connectivity after each step)

fw unloadlocal
fw fetch localhost
fw unloadlocal
Push policy from manager again

Note the “fw unloadlocal” will unload the policy from the firewall.

Beyond that, I suspect we’ll need to have someone take a deeper look.

For that, a TAC case is needed: Contact Support | Check Point Software

Please message me the TAC SR and I can have someone from R&D take a look.

Vladimir · ‎2017-09-28

I suppose that unloading the policy will not allow for connectivity, as it relies on the Logical Server object being present, unless the simple fact that it is created is sufficient for routing to persist in the absence of policy.

Please let me know if this is the case.

PhoneBoy · ‎2017-09-28

You should still be able to test connectivity to/from the gateway by unloading the policy.

It will break the logical server functionality, of course.

The steps I am proposing is to see if we can "reset" the policy that is stored on the vSEC instance.

Vladimir · ‎2017-09-28

Well, these are the results, all "login as: admin" is indicative of interruptions of the SSH session:

login as: admin
This system is for authorized use only.
Authenticating with public key "imported-openssh-key"
vSEC01> fw unloadlocal

Uninstalling Security Policy from all.all@vSEC01
Done.
vSEC01>
vSEC01>
vSEC01> fw fetch localhost

Installing Security Policy Standard on all.all@vSEC01
login as: admin
This system is for authorized use only.
Authenticating with public key "imported-openssh-key"
vSEC01> fw stat
HOST POLICY DATE
localhost Standard 28Sep2017 14:06:02 : [>eth0] [<eth0] [>eth1] [<eth1]
vSEC01> fw unloadlocal

Uninstalling Security Policy from all.all@vSEC01
Done.
vSEC01> fw stat
HOST POLICY DATE
localhost - - - : >eth0 <eth0 >eth1 <eth1
vSEC01>
login as: admin
This system is for authorized use only.
Authenticating with public key "imported-openssh-key"
CLINFR0771 Config lock is owned by admin. Use the command 'lock database override' to acquire the lock.
vSEC01> fw stat
HOST POLICY DATE
localhost Standard 28Sep2017 14:18:27 : [>eth0] [<eth0] [>eth1] [<eth1]
vSEC01>

What are the interfaces and directional arrows represent in fw stat output?

[>eth0] [<eth0] [>eth1] [<eth1] are shown, but there is also an eth2 active interface present in topology, but is not listed here.

PhoneBoy · ‎2017-09-28

Reloading the policy will kill active connections, so SSH terminating is not unusual.

The interfaces and arrows indicate the interfaces that have received traffic since policy install and the direction (inbound or outbound).

The fact you don't see eth2 means it's seen no traffic since the last policy install.

What I meant in terms of testing connectivity was do the test you had shown earlier in the thread.

Vladimir · ‎2017-09-28

Policy Loaded from SMS:

[Expert@vSEC01:0]# fw stat
HOST POLICY DATE
localhost Standard 28Sep2017 15:16:51 : [>eth0] [<eth0] [<eth1]
[Expert@vSEC01:0]# fw monitor -e 'accept port(80);'
monitor: getting filter (from command line)
monitor: compiling
monitorfilter:
Compiled OK.
monitor: loading
monitor: monitoring (control-C to stop)
[vs_0][fw_0] eth0:i[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=41686
TCP: 43996 -> 80 .S.... seq=49d1ba26 ack=00000000
[vs_0][fw_0] eth0:I[60]: 24.89.129.101 -> 10.255.255.210 (TCP) len=60 id=41686
TCP: 43996 -> 80 .S.... seq=49d1ba26 ack=00000000
[vs_0][fw_0] eth1:o[60]: 24.89.129.101 -> 10.255.255.210 (TCP) len=60 id=41686
TCP: 43996 -> 80 .S.... seq=49d1ba26 ack=00000000
[vs_0][fw_0] eth1:O[60]: 24.89.129.101 -> 10.255.255.210 (TCP) len=60 id=41686
TCP: 43996 -> 80 .S.... seq=49d1ba26 ack=00000000
[vs_0][fw_0] eth1:i[60]: 10.255.255.210 -> 24.89.129.101 (TCP) len=60 id=0
TCP: 80 -> 43996 .S..A. seq=f2771eb3 ack=49d1ba27
[vs_0][fw_0] eth1:I[60]: 10.255.255.210 -> 24.89.129.101 (TCP) len=60 id=0
TCP: 80 -> 43996 .S..A. seq=f2771eb3 ack=49d1ba27
[vs_0][fw_0] eth0:o[60]: 10.255.255.210 -> 24.89.129.101 (TCP) len=60 id=0
TCP: 80 -> 43996 .S..A. seq=f2771eb3 ack=49d1ba27
[vs_0][fw_0] eth0:O[60]: 10.255.255.23 -> 24.89.129.101 (TCP) len=60 id=0
TCP: 80 -> 43996 .S..A. seq=f2771eb3 ack=49d1ba27
[vs_0][fw_0] eth0:i[52]: 24.89.129.101 -> 10.255.255.23 (TCP) len=52 id=41687
TCP: 43996 -> 80 ....A. seq=49d1ba27 ack=f2771eb4
[vs_0][fw_0] eth0:I[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=41687
TCP: 43996 -> 80 ....A. seq=49d1ba27 ack=f2771eb4
[vs_0][fw_0] eth1:o[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=41687
TCP: 43996 -> 80 ....A. seq=49d1ba27 ack=f2771eb4
[vs_0][fw_0] eth1:O[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=41687
TCP: 43996 -> 80 ....A. seq=49d1ba27 ack=f2771eb4
[vs_0][fw_0] eth0:i[139]: 24.89.129.101 -> 10.255.255.23 (TCP) len=139 id=41688
TCP: 43996 -> 80 ...PA. seq=49d1ba27 ack=f2771eb4
[vs_0][fw_0] eth0:I[139]: 24.89.129.101 -> 10.255.255.210 (TCP) len=139 id=41688
TCP: 43996 -> 80 ...PA. seq=49d1ba27 ack=f2771eb4
[vs_0][fw_0] eth1:o[139]: 24.89.129.101 -> 10.255.255.210 (TCP) len=139 id=41688
TCP: 43996 -> 80 ...PA. seq=49d1ba27 ack=f2771eb4
[vs_0][fw_0] eth1:O[139]: 24.89.129.101 -> 10.255.255.210 (TCP) len=139 id=41688
TCP: 43996 -> 80 ...PA. seq=49d1ba27 ack=f2771eb4
[vs_0][fw_0] eth1:i[52]: 10.255.255.210 -> 24.89.129.101 (TCP) len=52 id=13346
TCP: 80 -> 43996 ....A. seq=f2771eb4 ack=49d1ba7e
[vs_0][fw_0] eth1:I[52]: 10.255.255.210 -> 24.89.129.101 (TCP) len=52 id=13346
TCP: 80 -> 43996 ....A. seq=f2771eb4 ack=49d1ba7e
[vs_0][fw_0] eth0:o[52]: 10.255.255.210 -> 24.89.129.101 (TCP) len=52 id=13346
TCP: 80 -> 43996 ....A. seq=f2771eb4 ack=49d1ba7e
[vs_0][fw_0] eth0:O[52]: 10.255.255.23 -> 24.89.129.101 (TCP) len=52 id=13346
TCP: 80 -> 43996 ....A. seq=f2771eb4 ack=49d1ba7e
[vs_0][fw_0] eth1:i[371]: 10.255.255.210 -> 24.89.129.101 (TCP) len=371 id=13347
TCP: 80 -> 43996 ...PA. seq=f2771eb4 ack=49d1ba7e
[vs_0][fw_0] eth1:I[371]: 10.255.255.210 -> 24.89.129.101 (TCP) len=371 id=13347
TCP: 80 -> 43996 ...PA. seq=f2771eb4 ack=49d1ba7e
[vs_0][fw_0] eth0:o[371]: 10.255.255.210 -> 24.89.129.101 (TCP) len=371 id=13347
TCP: 80 -> 43996 ...PA. seq=f2771eb4 ack=49d1ba7e
[vs_0][fw_0] eth0:O[371]: 10.255.255.23 -> 24.89.129.101 (TCP) len=371 id=13347
TCP: 80 -> 43996 ...PA. seq=f2771eb4 ack=49d1ba7e
[vs_0][fw_0] eth0:i[52]: 24.89.129.101 -> 10.255.255.23 (TCP) len=52 id=41689
TCP: 43996 -> 80 ....A. seq=49d1ba7e ack=f2771ff3
[vs_0][fw_0] eth0:I[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=41689
TCP: 43996 -> 80 ....A. seq=49d1ba7e ack=f2771ff3
[vs_0][fw_0] eth1:o[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=41689
TCP: 43996 -> 80 ....A. seq=49d1ba7e ack=f2771ff3
[vs_0][fw_0] eth1:O[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=41689
TCP: 43996 -> 80 ....A. seq=49d1ba7e ack=f2771ff3
[vs_0][fw_0] eth0:i[52]: 24.89.129.101 -> 10.255.255.23 (TCP) len=52 id=41690
TCP: 43996 -> 80 F...A. seq=49d1ba7e ack=f2771ff3
[vs_0][fw_0] eth0:I[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=41690
TCP: 43996 -> 80 F...A. seq=49d1ba7e ack=f2771ff3
[vs_0][fw_0] eth1:o[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=41690
TCP: 43996 -> 80 F...A. seq=49d1ba7e ack=f2771ff3
[vs_0][fw_0] eth1:O[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=41690
TCP: 43996 -> 80 F...A. seq=49d1ba7e ack=f2771ff3
[vs_0][fw_0] eth1:i[52]: 10.255.255.210 -> 24.89.129.101 (TCP) len=52 id=13348
TCP: 80 -> 43996 F...A. seq=f2771ff3 ack=49d1ba7f
[vs_0][fw_0] eth1:I[52]: 10.255.255.210 -> 24.89.129.101 (TCP) len=52 id=13348
TCP: 80 -> 43996 F...A. seq=f2771ff3 ack=49d1ba7f
[vs_0][fw_0] eth0:o[52]: 10.255.255.210 -> 24.89.129.101 (TCP) len=52 id=13348
TCP: 80 -> 43996 F...A. seq=f2771ff3 ack=49d1ba7f
[vs_0][fw_0] eth0:O[52]: 10.255.255.23 -> 24.89.129.101 (TCP) len=52 id=13348
TCP: 80 -> 43996 F...A. seq=f2771ff3 ack=49d1ba7f
[vs_0][fw_0] eth0:i[52]: 24.89.129.101 -> 10.255.255.23 (TCP) len=52 id=41691
TCP: 43996 -> 80 ....A. seq=49d1ba7f ack=f2771ff4
[vs_0][fw_0] eth0:I[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=41691
TCP: 43996 -> 80 ....A. seq=49d1ba7f ack=f2771ff4
[vs_0][fw_0] eth1:o[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=41691
TCP: 43996 -> 80 ....A. seq=49d1ba7f ack=f2771ff4
[vs_0][fw_0] eth1:O[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=41691
TCP: 43996 -> 80 ....A. seq=49d1ba7f ack=f2771ff4
monitor: caught sig 2
monitor: unloading
[Expert@vSEC01:0]#

Few minutes later:

[Expert@vSEC01:0]# fw monitor -e 'accept port(80);'
monitor: getting filter (from command line)
monitor: compiling
monitorfilter:
Compiled OK.
monitor: loading
monitor: monitoring (control-C to stop)
[vs_0][fw_1] eth0:i[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=52050
TCP: 43997 -> 80 .S.... seq=5dcfde53 ack=00000000
[vs_0][fw_1] eth0:i[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=52051
TCP: 43997 -> 80 .S.... seq=5dcfde53 ack=00000000
[vs_0][fw_0] eth0:i[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=52052
TCP: 43997 -> 80 .S.... seq=5dcfde53 ack=00000000
[vs_0][fw_0] eth0:i[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=52053
TCP: 43997 -> 80 .S.... seq=5dcfde53 ack=00000000
monitor: caught sig 2
monitor: unloading
[Expert@vSEC01:0]#

Policy unloaded:

[Expert@vSEC01:0]# fw unloadlocal

Uninstalling Security Policy from all.all@vSEC01
Done.
[Expert@vSEC01:0]# fw monitor -e 'accept port(80);'
monitor: getting filter (from command line)
monitor: compiling
monitorfilter:
Compiled OK.
monitor: loading
monitor: monitoring (control-C to stop)
[vs_0][fw_0] eth0:i[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=22893
TCP: 43998 -> 80 .S.... seq=e23068c7 ack=00000000
[vs_0][fw_0] eth0:I[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=22893
TCP: 43998 -> 80 .S.... seq=e23068c7 ack=00000000
[vs_0][fw_0] eth0:i[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=22894
TCP: 43998 -> 80 .S.... seq=e23068c7 ack=00000000
[vs_0][fw_0] eth0:I[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=22894
TCP: 43998 -> 80 .S.... seq=e23068c7 ack=00000000
[vs_0][fw_0] eth0:i[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=22895
TCP: 43998 -> 80 .S.... seq=e23068c7 ack=00000000
[vs_0][fw_0] eth0:I[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=22895
TCP: 43998 -> 80 .S.... seq=e23068c7 ack=00000000
monitor: caught sig 2
monitor: unloading
[Expert@vSEC01:0]#

Policy fetched from localhost (note that it started working when acks had value other than 00000000 and than abruptly stopped:

[Expert@vSEC01:0]# fw fetch localhost

Installing Security Policy Standard on all.all@vSEC01
login as: admin
This system is for authorized use only.
Authenticating with public key "imported-openssh-key"
vSEC01> expert
Enter expert password:

Warning! All configurations should be done through clish
You are in expert mode now.

[Expert@vSEC01:0]# fw monitor -e 'accept port(80);'
monitor: getting filter (from command line)
monitor: compiling
monitorfilter:
Compiled OK.
monitor: loading
monitor: monitoring (control-C to stop)
[vs_0][fw_0] eth0:i[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=6614
TCP: 43999 -> 80 .S.... seq=2d39bde1 ack=00000000
[vs_0][fw_0] eth0:I[60]: 24.89.129.101 -> 10.255.255.210 (TCP) len=60 id=6614
TCP: 43999 -> 80 .S.... seq=2d39bde1 ack=00000000
[vs_0][fw_0] eth1:o[60]: 24.89.129.101 -> 10.255.255.210 (TCP) len=60 id=6614
TCP: 43999 -> 80 .S.... seq=2d39bde1 ack=00000000
[vs_0][fw_0] eth1:O[60]: 24.89.129.101 -> 10.255.255.210 (TCP) len=60 id=6614
TCP: 43999 -> 80 .S.... seq=2d39bde1 ack=00000000
[vs_0][fw_0] eth1:i[60]: 10.255.255.210 -> 24.89.129.101 (TCP) len=60 id=0
TCP: 80 -> 43999 .S..A. seq=d5802251 ack=2d39bde2
[vs_0][fw_0] eth1:I[60]: 10.255.255.210 -> 24.89.129.101 (TCP) len=60 id=0
TCP: 80 -> 43999 .S..A. seq=d5802251 ack=2d39bde2
[vs_0][fw_0] eth0:o[60]: 10.255.255.210 -> 24.89.129.101 (TCP) len=60 id=0
TCP: 80 -> 43999 .S..A. seq=d5802251 ack=2d39bde2
[vs_0][fw_0] eth0:O[60]: 10.255.255.23 -> 24.89.129.101 (TCP) len=60 id=0
TCP: 80 -> 43999 .S..A. seq=d5802251 ack=2d39bde2
[vs_0][fw_0] eth0:i[52]: 24.89.129.101 -> 10.255.255.23 (TCP) len=52 id=6615
TCP: 43999 -> 80 ....A. seq=2d39bde2 ack=d5802252
[vs_0][fw_0] eth0:I[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=6615
TCP: 43999 -> 80 ....A. seq=2d39bde2 ack=d5802252
[vs_0][fw_0] eth1:o[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=6615
TCP: 43999 -> 80 ....A. seq=2d39bde2 ack=d5802252
[vs_0][fw_0] eth1:O[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=6615
TCP: 43999 -> 80 ....A. seq=2d39bde2 ack=d5802252
[vs_0][fw_0] eth0:i[139]: 24.89.129.101 -> 10.255.255.23 (TCP) len=139 id=6616
TCP: 43999 -> 80 ...PA. seq=2d39bde2 ack=d5802252
[vs_0][fw_0] eth0:I[139]: 24.89.129.101 -> 10.255.255.210 (TCP) len=139 id=6616
TCP: 43999 -> 80 ...PA. seq=2d39bde2 ack=d5802252
[vs_0][fw_0] eth1:o[139]: 24.89.129.101 -> 10.255.255.210 (TCP) len=139 id=6616
TCP: 43999 -> 80 ...PA. seq=2d39bde2 ack=d5802252
[vs_0][fw_0] eth1:O[139]: 24.89.129.101 -> 10.255.255.210 (TCP) len=139 id=6616
TCP: 43999 -> 80 ...PA. seq=2d39bde2 ack=d5802252
[vs_0][fw_0] eth1:i[52]: 10.255.255.210 -> 24.89.129.101 (TCP) len=52 id=28466
TCP: 80 -> 43999 ....A. seq=d5802252 ack=2d39be39
[vs_0][fw_0] eth1:I[52]: 10.255.255.210 -> 24.89.129.101 (TCP) len=52 id=28466
TCP: 80 -> 43999 ....A. seq=d5802252 ack=2d39be39
[vs_0][fw_0] eth0:o[52]: 10.255.255.210 -> 24.89.129.101 (TCP) len=52 id=28466
TCP: 80 -> 43999 ....A. seq=d5802252 ack=2d39be39
[vs_0][fw_0] eth0:O[52]: 10.255.255.23 -> 24.89.129.101 (TCP) len=52 id=28466
TCP: 80 -> 43999 ....A. seq=d5802252 ack=2d39be39
[vs_0][fw_0] eth1:i[371]: 10.255.255.210 -> 24.89.129.101 (TCP) len=371 id=28467
TCP: 80 -> 43999 ...PA. seq=d5802252 ack=2d39be39
[vs_0][fw_0] eth1:I[371]: 10.255.255.210 -> 24.89.129.101 (TCP) len=371 id=28467
TCP: 80 -> 43999 ...PA. seq=d5802252 ack=2d39be39
[vs_0][fw_0] eth0:o[371]: 10.255.255.210 -> 24.89.129.101 (TCP) len=371 id=28467
TCP: 80 -> 43999 ...PA. seq=d5802252 ack=2d39be39
[vs_0][fw_0] eth0:O[371]: 10.255.255.23 -> 24.89.129.101 (TCP) len=371 id=28467
TCP: 80 -> 43999 ...PA. seq=d5802252 ack=2d39be39
[vs_0][fw_0] eth0:i[52]: 24.89.129.101 -> 10.255.255.23 (TCP) len=52 id=6617
TCP: 43999 -> 80 ....A. seq=2d39be39 ack=d5802391
[vs_0][fw_0] eth0:I[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=6617
TCP: 43999 -> 80 ....A. seq=2d39be39 ack=d5802391
[vs_0][fw_0] eth1:o[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=6617
TCP: 43999 -> 80 ....A. seq=2d39be39 ack=d5802391
[vs_0][fw_0] eth1:O[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=6617
TCP: 43999 -> 80 ....A. seq=2d39be39 ack=d5802391
[vs_0][fw_0] eth0:i[52]: 24.89.129.101 -> 10.255.255.23 (TCP) len=52 id=6618
TCP: 43999 -> 80 F...A. seq=2d39be39 ack=d5802391
[vs_0][fw_0] eth0:I[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=6618
TCP: 43999 -> 80 F...A. seq=2d39be39 ack=d5802391
[vs_0][fw_0] eth1:o[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=6618
TCP: 43999 -> 80 F...A. seq=2d39be39 ack=d5802391
[vs_0][fw_0] eth1:O[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=6618
TCP: 43999 -> 80 F...A. seq=2d39be39 ack=d5802391
[vs_0][fw_0] eth1:i[52]: 10.255.255.210 -> 24.89.129.101 (TCP) len=52 id=28468
TCP: 80 -> 43999 F...A. seq=d5802391 ack=2d39be3a
[vs_0][fw_0] eth1:I[52]: 10.255.255.210 -> 24.89.129.101 (TCP) len=52 id=28468
TCP: 80 -> 43999 F...A. seq=d5802391 ack=2d39be3a
[vs_0][fw_0] eth0:o[52]: 10.255.255.210 -> 24.89.129.101 (TCP) len=52 id=28468
TCP: 80 -> 43999 F...A. seq=d5802391 ack=2d39be3a
[vs_0][fw_0] eth0:O[52]: 10.255.255.23 -> 24.89.129.101 (TCP) len=52 id=28468
TCP: 80 -> 43999 F...A. seq=d5802391 ack=2d39be3a
[vs_0][fw_0] eth0:i[52]: 24.89.129.101 -> 10.255.255.23 (TCP) len=52 id=6619
TCP: 43999 -> 80 ....A. seq=2d39be3a ack=d5802392
[vs_0][fw_0] eth0:I[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=6619
TCP: 43999 -> 80 ....A. seq=2d39be3a ack=d5802392
[vs_0][fw_0] eth1:o[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=6619
TCP: 43999 -> 80 ....A. seq=2d39be3a ack=d5802392
[vs_0][fw_0] eth1:O[52]: 24.89.129.101 -> 10.255.255.210 (TCP) len=52 id=6619
TCP: 43999 -> 80 ....A. seq=2d39be3a ack=d5802392
[vs_0][fw_1] eth0:i[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=54867
TCP: 44000 -> 80 .S.... seq=5618676f ack=00000000
[vs_0][fw_1] eth0:i[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=54868
TCP: 44000 -> 80 .S.... seq=5618676f ack=00000000
[vs_0][fw_1] eth0:i[60]: 24.89.129.101 -> 10.255.255.23 (TCP) len=60 id=54869
TCP: 44000 -> 80 .S.... seq=5618676f ack=00000000
monitor: caught sig 2
monitor: unloading
[Expert@vSEC01:0]#

PhoneBoy · ‎2017-09-28

So the same, basically.

Have you opened a TAC case like I suggested earlier?

Vladimir · ‎2017-09-28

Tried with no success: these licenses are NFR and trials from the VAR I am working with, but their support contract apparently expired.

PhoneBoy · ‎2017-09-28

Hm... that could be an issue.

Try setting the protocol to be "blank" in the HTTP and HTTPS services you created.

Vladimir · ‎2017-09-28

OK, got to create SR regardless: 1-9784890381

Will try the empty protocols next.

Vladimir · ‎2017-09-28

Blank protocols behave the same way: Policy install-->Working--Working--Dead

Vladimir · ‎2017-09-29

The case was moved to R&D.

I am attaching a most detailed debug output focusing on NAT, since it is pretty large, just in case anyone is interested to see how granular it could be.

It was produced by:

fw ctl debug 0
fw ctl debug -buf 32000
fw ctl debug -m fw + conn drop nat xlate
fw ctl debug -m UP + all
fw ctl kdebug -T -f > debug.txt

replicate behavior

Ctrl + c = to stop
fw ctl debug 0 = reset to default

Vladimir · ‎2017-10-01

Update:

Tried following to pin down the issue:

Created single web server, statically NATed and it is working fine.
Implemented Hide NAT on dummy object representing WebServer-facing interface of the vSEC as per sk104249, timeout after 60 seconds following policy installation.
Uninstalled Check_Point_R80_10_JUMBO_HF_Bundle_T42_sk116380_FULL.tgz to see if this release was causing the issue and installed the policy, same timeout after 60 seconds following policy installation.

This is getting extremely frustrating. I cannot possibly be the only user of vSEC R80.10 in AWS, yet this behavior is so consistent that the only possible causes that I can see are:

1. Some file in vSEC is corrupt

2. It is a systemic problem across multiple HFAs

PhoneBoy · ‎2017-10-02

You should have gotten a note from TAC on progressing the troubleshooting of this issue through your SR.

The specific suggestions came from R&D.

What DNS servers are configured in your vSEC instances?

Can they resolve the Logical Server name (which, in turn, are the DNS names of the Elastic Load Balancers)?

Vladimir · ‎2017-10-02

Dameon,

This Logical Server does not represent ELB, it is a Check Point object only

and, as such, will not be resolved by DNS.

It contains a simple group that, in turn, contains host objects representing

web servers, (also hosts defined in Check Point, not the EC2 named instances).

see "vSEC Gateway for Amazon Web Services" document, page 15, "Protecting a

Web Server".

The references to ELB and the name resolution are applicable if we are using

load balancing by "Domain" , not per "Round Trip" , as it is in my case.

Given that the Simple Group in question contains a single host, at present,

this should not be an issue that I am seeing.

Vladimir Yakovlev

973.558.2738

<mailto:vlad@eversecgroup.com> vlad@eversecgroup.com

Vladimir · ‎2017-10-26

Update for those following this thread:

The issue was replicated and forwarded to R&D.

Service Request # 1-9784890381

It was determined that the health check that the Logical Server performing on target hosts erroneously labels them "Dead" after about 60 seconds.

I'll let you know when I receive anything from R&D.

Vladimir · ‎2017-11-15

Looks like the source document (5 April 2017 Getting Started Guide vSEC Gateway for Amazon Web Services) contains erroneous information.

From CP R&D:

"After some discussion with R&D and a little research, there is a SK that states this configuration is not supported.

So, I will need to follow up with the vSEC team to understand why we publisheed a document for an unsuppported configuration.

Please refer to sk31162:

Domain and Round Trip load balancing is not supported for Logical Servers."

This statement, however, begs a question: why are these properties available in the object that does not support them?

Are you a member of CheckMates?

Inconsistent behavior of vSEC in AWS

Service Request # 1-9784890381