Quantum Spark - how to monitor traffic inside a switch interface ?

[marcyn]

Hi Checkmates,

Today I faced ... a problem ... that suprized me.
I wanted to monitor traffic inside a switch interface on Spark ... and to my surprise I was not able to do it.

I have a LAN1_Switch interface that contains LAN1-LAN8 interfaces - this is 1570 model, but I believe it will work the same on other models as well.

Spark side:

Spark1570> show interfaces
name: LAN1_Switch
ipv4-address: 10.99.99.254

(mask /24)

Client1 connected to LAN1 port:
10.99.99.1

Client2 connected to LAN2 port:
10.99.99.2

So both clients are of course inside the same network, but traffic from one to another has to go via Spark's switch interface.
Because of that I expected that I will see this traffic for example in tcpdump or fw monitor.
To my surprise there is nothing - only arp who-has messages.

Let's see an easy example:

On client1:

root@black:/mnt/c/Users/marcyn# ping 10.99.99.2
PING 10.99.99.2 (10.99.99.2) 56(84) bytes of data.
64 bytes from 10.99.99.2: icmp_seq=1 ttl=64 time=0.691 ms
64 bytes from 10.99.99.2: icmp_seq=2 ttl=64 time=0.427 ms
^C
--- 10.99.99.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.427/0.559/0.691/0.132 ms

root@black:/mnt/c/Users/marcyn# telnet 10.99.99.2 80
Trying 10.99.99.2...
Connected to 10.99.99.2.
Escape character is '^]'.
^]

telnet> quit
Connection closed.
root@black:/mnt/c/Users/marcyn#

And how does it look like on Spark ? - tcpdump in this example:

[Expert@Spark1570]# tcpdump -nnei any host \(10.99.99.1 or 10.99.99.2\) and \(icmp or port 80\)
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes


(nothing ...
in case there would be no arp entries for 10.99.99.1 or 10.99.99.2 yet ... I would see here arp who-has messages)

So ... how to monitor traffic inside a switch ?
There has to be some way ... 🙂

Have you faced this "issue" before and know the solution ?

BTW
Of course there is absolutely no problem at all with monitoring traffic from one interface to another if they are not inside a switch.

--
Best
m.

[PhoneBoy]

There is a hardware-level switch involved on the Quantum Spark appliances.
That traffic isn't typically inspected, but you can enable this function (with a performance impact):

[marcyn]

Hi @PhoneBoy,

Thank your for your reply.
I admit that I completely forgot to take a look at Advanced Settings ... 🙂

Unfortunately this option that you mentioned, and also some other that I checked (ex. "OS advanced settings - Enable flow-control for network switch") ... doesn't change the situation.
I still see no packets that go from one interface to another inside a switch.

But as you mentioned as it is a hardware switch ... it seems that I will not achieve this goal.
Why I want that, you may ask ... there can be a lot of reasons, for example to just see traffic flow in logs (not neccessary to inspect this traffic, but just to have better visibility of this traffic).

--
Best
m.

[PhoneBoy]

I completely understand the need/desire for this 🙂
Did you try using fw monitor to check this traffic after enabling this option?

[marcyn]

Hi @PhoneBoy,

I checked this only with tcpdump before - there was nothing.
But why not checking it with fw monitor as well (I expect nothing more 🙂 ).

So here you have it:

C:\Users\marcyn>ping 10.99.99.2

Pinging 10.99.99.2 with 32 bytes of data:
Reply from 10.99.99.2: bytes=32 time<1ms TTL=64
Reply from 10.99.99.2: bytes=32 time<1ms TTL=64
Reply from 10.99.99.2: bytes=32 time<1ms TTL=64
Reply from 10.99.99.2: bytes=32 time<1ms TTL=64

[Expert@Spark1570]# fw monitor -F "10.99.99.1,0,10.99.99.2,0,1" -F "10.99.99.2,0,10.99.99.1,0,1"
(...)
 fw: monitoring (control-C to stop)
PPAK 0: Get before set operation succeeded of fwmonitormaxpacket
PPAK 0: Get before set operation succeeded of fwmonitormask
PPAK 0: Get before set operation succeeded of fwmonitorallocbufs
PPAK 0: Get before set operation succeeded of printuuid
PPAK 0: Get before set operation succeeded of fwmonitor_kiss_enable

(nothing ... as expected)


The same test between different networks:

[Expert@Spark1570]# fw monitor -F "10.98.98.3,0,10.99.99.2,0,1" -F "10.99.99.2,0,10.98.98.3,0,1"
(...)
 fw: monitoring (control-C to stop)
PPAK 0: Get before set operation succeeded of fwmonitormaxpacket
PPAK 0: Get before set operation succeeded of fwmonitormask
PPAK 0: Get before set operation succeeded of fwmonitorallocbufs
PPAK 0: Get before set operation succeeded of printuuid
PPAK 0: Get before set operation succeeded of fwmonitor_kiss_enable
[vs_0][ppak_0] wlan0:i[44]: 10.98.98.3 -> 10.99.99.2 (ICMP) len=84 id=54008
ICMP: type=8 code=0 echo request id=17 seq=1
[vs_0][fw_1] wlan0:i[44]: 10.98.98.3 -> 10.99.99.2 (ICMP) len=84 id=54008
ICMP: type=8 code=0 echo request id=17 seq=1
[vs_0][fw_1] wlan0:I[44]: 10.98.98.3 -> 10.99.99.2 (ICMP) len=84 id=54008
ICMP: type=8 code=0 echo request id=17 seq=1
[vs_0][ppak_0] br0:i[44]: 10.98.98.3 -> 10.99.99.2 (ICMP) len=84 id=54008
ICMP: type=8 code=0 echo request id=17 seq=1
[vs_0][fw_1] LAN1:o[44]: 10.98.98.3 -> 10.99.99.2 (ICMP) len=84 id=54008
ICMP: type=8 code=0 echo request id=17 seq=1
[vs_0][fw_1] LAN1:O[44]: 10.98.98.3 -> 10.99.99.2 (ICMP) len=84 id=54008
ICMP: type=8 code=0 echo request id=17 seq=1
[vs_0][ppak_0] LAN1:i[44]: 10.99.99.2 -> 10.98.98.3 (ICMP) len=84 id=10860
ICMP: type=0 code=0 echo reply id=17 seq=1
[vs_0][fw_1] LAN1:i[44]: 10.99.99.2 -> 10.98.98.3 (ICMP) len=84 id=10860
ICMP: type=0 code=0 echo reply id=17 seq=1
[vs_0][fw_1] LAN1:I[44]: 10.99.99.2 -> 10.98.98.3 (ICMP) len=84 id=10860
ICMP: type=0 code=0 echo reply id=17 seq=1
[vs_0][fw_1] wlan0:o[44]: 10.99.99.2 -> 10.98.98.3 (ICMP) len=84 id=10860
ICMP: type=0 code=0 echo reply id=17 seq=1
[vs_0][fw_1] wlan0:O[44]: 10.99.99.2 -> 10.98.98.3 (ICMP) len=84 id=10860
ICMP: type=0 code=0 echo reply id=17 seq=1

As you can see above - with different networks I have perfect request and reply 🙂

--
br,
m.

[PhoneBoy]

It was worth a try.
However, it does make me wonder how that Advanced option works...or if it still does.

[marcyn]

To be honest ... the same from my side 🙂
I was thinking that maybe it works as the name suggests .... so if enabled it will inspect traffic from LAN 2 LAN ... but no, it's not working like that.
Simple example gave this answer - rule on top of incoming rules (local management) where source and destination is 10.99.99.0/255.255.255.0, service is icmp and action is block.
With such a rule if we will have inspection between LAN and LAN ... it should block ping.
Of course ping is working.

Sure, this example had no sense at all ... because if there is no visible traffic in tcpdump/fw monitor between host in this network ... such a rule will just be nonsense .... but as you wrote "It was worth a try" 🙂

To to summarize this discussion - it looks like there is no way ... and I have to accept this that this is hardware switch and period 🙂

But still .... too bad ...

--
BR,
m.