There is a reason that is needed and this is what it is.
The VERY first time you connect to a VPN Gateway with a Client it asks you to trust the VPN Certificate as being from the ICA then is not a Trusted CA.
That connection is made over HTTPS not IPSEC protocols
You will see subsequently when you connect that before the IPSEC tunnel is initiated then the Client makes a HTTPS connection to the Gateway.
The Visitor Mode allows this HTTPS connection to be made.
No response from the HTTPS request and the IPSEC tunnel doesn't attempt instead it says is unreachable etc.
264 is the fw1_topo port that used for downloading the topology.
You don't know in advance where they are coming from so you have to have open everywhere.
Same as port 500 and proto 50/51 to allow the IPSec Tunnel to build, you don't know the source so has to be open, of course it doesn't stop them being reported by scanners as vulnerabilties but won't work without them being open