topic Re: Zendesk voice application 3 second delay in Firewall and Security Management

Zendesk voice application 3 second delay

dehaasm — Tue, 23 Jul 2024 10:31:16 GMT

Not sure if this is a Check Point related issue but the issue only appears when connected behind a Check Point firewall, when connected to direct internet the issue does not occur.

The issue is that whenever the Zendesk servicedesk application makes a Voice call (via Cloud STUN server) after answering a call there is a 3 second delay before the voice is heared, after 3 seconds the call works fine. While performing the same test without a firewal in between this issue does not occur.

We made packet captures and also tried to disable coreXL (fwaccell off) but it didnt resolve the issue.

In the packet capture the only thing i see is that there is a DTLS handhakes with a 3 second delay, meaning that it takes 3 seconds for the external server to reply. This does not show any delay on the Check Point firewall but perhaps i am overlooking something as we cannot find the root cause.

Dit someone already experience such issue?

Re: Zendesk voice application 3 second delay

the_rock — Tue, 23 Jul 2024 12:59:42 GMT

Do you have simple network diagram?

Andy

Re: Zendesk voice application 3 second delay

bacim — Tue, 23 Jul 2024 14:19:11 GMT

Re: Zendesk voice application 3 second delay

bacim — Tue, 23 Jul 2024 14:24:37 GMT

Traffic is initiated by the client (user)

FW is a cluster of 2 (no difference noticed in which node is active)

Re: Zendesk voice application 3 second delay

the_rock — Tue, 23 Jul 2024 14:23:38 GMT

Will respond in a bit with some ideas/suggeestions.

Andy

Re: Zendesk voice application 3 second delay

the_rock — Tue, 23 Jul 2024 16:11:32 GMT

K, so here is how I would approach troubleshooting this. So, lets start with whats logical, or what we know...so, we know 100% that if all this works without CP fw in the "pitcure", there is something on the fw side causing the problem. What can it be? Well, usually, for things like this, I would first look at the service(s) used.

In the old days of CP, what people would do is edit the service, select protocol as 'NONE, which in simple terms, would essentially bypass IPS inspectrion, if you will.

Thats one thing to try and install policy, test. If that fails, I would generate fw monitor as per below.

Lets pretend user's IP is 1.1.1.1, zendesk is 2.2.2.2 and port is 4434

idea is srcip,srcport,dstip,dstport, protocol

so it would be like this (just use right IPs and ports, of course, though for the context, ONLY dst port matters)

fw monitor -F "1.1.1.1,4434,2.2.2.2,443,0" -F 2.2.2.2,443,1.1.1.1,443,0"

0 is for any protocol

Once you have that, send, so we can analyze.

Best,

Andy

Re: Zendesk voice application 3 second delay

bacim — Wed, 24 Jul 2024 13:53:39 GMT

Hi Rock,

Thanks for your feedback.

I believe an FW monitor would not work as the public IP of zendesk is randomly within the 168.86.128.0/18 range?
Furthermore, traffic is sent over UDP ranging from ports 10000-60000.

Our rulebase cell settings prohibit the use of "none" for services.

Could I interest you in pcaps? The delays are visible there in the DTLSv1.2 handshake prior to the establishment of the connection.

Re: Zendesk voice application 3 second delay

bacim — Wed, 24 Jul 2024 08:43:31 GMT

I thought you meant "none" in the policy in like this:

In case you meant this, the port object is already using protocol none in the policy:

no IPS logs can be found for this public range

Re: Zendesk voice application 3 second delay

the_rock — Wed, 24 Jul 2024 11:32:59 GMT

Is there "none" listed under protocol menu of the service?

If so, please try that.

Andy

Re: Zendesk voice application 3 second delay

bacim — Wed, 24 Jul 2024 11:42:53 GMT

There is, but clicking on it results in the same "no item selected"

Re: Zendesk voice application 3 second delay

the_rock — Wed, 24 Jul 2024 12:25:06 GMT

Yes, message me directly, we can connect offline, I will review the pcaps.

Andy

Re: Zendesk voice application 3 second delay

the_rock — Wed, 24 Jul 2024 12:34:07 GMT

This is bugging me now, as I see none is there, BUT, it always shows no item selected if you click on it, non never stays. Im trying to see if thats actually expected or if it can be changed.

I will keep you posted. Sorry, have to work on some large Fortinet project, so will work on this when free at times.

Andy

Re: Zendesk voice application 3 second delay

bacim — Wed, 24 Jul 2024 12:37:10 GMT

Of course, no rush, meanwhile I sent you the pcaps via message.

Thank you!

Re: Zendesk voice application 3 second delay

the_rock — Wed, 24 Jul 2024 12:48:58 GMT

Since Im waiting on the dude on the other side to do stuff, I had quick look and I have a gut feeling I know why this happens.

Are you using ssl inspection on CP fw? If so, please add bypass rule for this traffic.

If not, run cipher_util command from expert and see whats used there.

I think based on below messages it gives clear indication...

Andy

Re: Zendesk voice application 3 second delay

Reeve60 — Wed, 24 Jul 2024 12:59:05 GMT

Hi @bacim @the_rock

I have been experiencing this same issue for the past few months, and unfortunately, it is still ongoing. The problem was first noticed around October 2023.

I raised a support case with TAC, and the latest update is that I need to collect packet capture logs from the firewall while fwaccell is turned off. However, after seeing your post, it seems this step may not yield any results.

I have attached an example from my side with the same 3 second delay...

Thanks,

Stefan

Re: Zendesk voice application 3 second delay

the_rock — Wed, 24 Jul 2024 13:01:27 GMT

Put it this way...IF turning off sxl does not fix the issue, generating debug while sxl is off, in my opinion, is not going to do anything.

Andy

Re: Zendesk voice application 3 second delay

bacim — Wed, 24 Jul 2024 13:02:57 GMT

Hi all, SSL inspection is being bypassed now for this specific connection. Will test again with the client and take a new pcap, keep you posted.

Re: Zendesk voice application 3 second delay

the_rock — Wed, 24 Jul 2024 13:04:21 GMT

Sounds good! Hope that works...if NOT, dont bother turning off ssl inspection, since bypass is there now, I would examine cipher_util.

Andy

Re: Zendesk voice application 3 second delay

the_rock — Wed, 24 Jul 2024 13:06:22 GMT

I did quick search and I see some posts online where this could happen due to NAT. Can you confirm how (if nat is used) its configured?

Andy

Re: Zendesk voice application 3 second delay

the_rock — Wed, 24 Jul 2024 13:14:45 GMT

@bacim Though, thinking about it, in case issue is still there, it might be worth, just as a test, to turn off ssl inspection and push policy, it would take less than 5 mins.

Andy