There is not out of state entries on the connection table. the security gateway does not storage them, for example if you are under a DDoS attack millions of ACK will arrive to the gateway but non of them will be save on the connection table. for obvious reasons.
You can delete an specific entry with the command bellow however is not recommended on production environments :
# fw tab -t connections -x -e <5 touple on HEX >
Example :
# fw tab -t connections -x -e 0000020,ad1e2f98,0000cb08,ab1aa870,0000470c,00000006
to see your connection table :
#fw tab -t 8158
after the connection is delete you will have some out of state drops until the connection is established once again.