The underlying mechanics of IKE and IPSec work exactly the same regardless of whether domain-based or route-based VPNs are used; the only difference is how traffic is determined to be "interesting" and in need of encryption. With domain-based VPNs if the source IP is in your firewall's VPN domain AND the destination IP address is in a peer's VPN domain, the traffic is interesting and needs to be encrypted. If only one or neither are true, the traffic goes in the clear.
For a route-based VPN, if IP routing determines that the egress interface is a VTI the traffic is interesting and will be encrypted, then forwarded out the appropriate physical interface. If IP routing determines that the egress interface is a physical one (i.e. eth0), the traffic simply goes in the clear. Note that the routes making this determination can be statically created, or dynamically learned through a routing protocol such as OSPF.
The main thing you have to watch out for when mixing domain and route-based setups on the same tunnel is how the Phase 2 Proxy-IDs are negotiated. As the initiator, domain-based VPN setups will negotiate subnets (i.e. 192.168.1.0/24) because the VPN domains are fixed and known ahead of time. However with a route-based VPN setup, the firewall does not necessarily know ahead of time which IP addresses will be used in the tunnel because routes can be dynamically received through OSPF. For that reason when the route-based VPN side initiates a Phase 2 negotiation, it will request a universal tunnel (i.e. 0.0.0.0/0, 0.0.0.0/0) unless told otherwise which will most definitely honk off the domain-based VPN side.
Almost all existing Check VPN implementations are domain-based due to the CoreXL limitations with route-based VPNs in R77.30 and earlier. Route-based VPNs will definitely become more common now that this restriction has been lifted in R80.10+.
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com
New 2-day Live "Max Power" Series Course Now Available:
"Gateway Performance Optimization R81.20" at maxpowerfirewalls.com