Short answer: You need to download and fire up @HeikoAnkenbrand's awesome econn tool (Easy Tool - R81.20 Real time connection table analysis v5.0 ) while the overflow is happening, it will slice and dice the connections table multiple ways and allow you to identify the "top talkers" or whatever traffic is causing the connections table overflow. Never tried it on VSX but as long as you are in the proper VS I assume it would work.
Long Answer: This exact scenario covered extensively in my Gateway Performance Optimization R81.20 class and actually experienced/corrected in a lab exercise as well. Here are the relevant pages:
Gateway Performance Optimization R81.20 Course
now available at maxpowerfirewalls.com
