The fw monitor -F option is not really the typical fw monitor chain module insertion mechanism employed by the -e option, -F is merely setting a kernel debug in the sim driver which is why the filtering syntax is so limited with -F. It sounds like some prior -F debugs were not properly terminated and you were allowed to start more. In R80.40 (and possibly earlier versions, not sure) if any prior currently-running fw monitor captures (-e and/or -F) are detected the new capture attempt is rejected, and it tells you to run fw monitor -U first to kill the old one.
The other thing that may have happened to you is that there does not appear to be any syntax checking on the srcIP,srcPort,dstIP,dstPort,IPProto filter used with -F, and if you make a mistake it simply gives you the equivalent of 0,0,0,0,0 with no warning which is absolutely all traffic. Concerning and quite dangerous in my opinion since a runaway debug inside SecureXL/sim is an easy way to thoroughly overload or at least severely impact the firewall.
New 2-day Live "Max Power" Series Course Now Available:
"Gateway Performance Optimization R81.20" at maxpowerfirewalls.com