Let me guess, if you use HTTPS/TLS as the VPN transport instead of IPSec, performance is just great.
You have a low MTU in your network path somewhere, or somehow the VPN client is affecting the MTU when it is active. The symptom of this is terrible performance due to packet loss because of the inability to fragment IPSec traffic due to the DF bit being set.
To verify, run netstat -sv in Windows and note the counters associated with IP frags and TCP segment retransmissions. Initialize the VPN tunnel with IPSec and start a big TCP-based upload. Which frag/retransmit counters in the netstat -sv output jump? This should give you some idea of where to look.
Gateway Performance Optimization R81.20 Course
now available at maxpowerfirewalls.com