Asymmetric routing with AWS Transit VPC and BGP

[Dave_Hollis]

I'm setting up a lab environment to get familiar with setting up a Transit VPC architecture at AWS and I think that I have all of the bits working, except I'm getting asymmetric routing on return traffic, thus keeping things from working.  If I shutdown the vpnt interfaces on one of the transit gateways, everything becomes symmetric and traffic flows freely.  I simply can't find what little thing I may have missed that's causing this.

Here's my (fairly typical I think) setup for this implementation:

On-prem gateway VM on VMware - BGP AS 65001

Transit VPC @ AWS (R80.20, built using the cloud formation template w/ new VPC) - BGP AS 65000

Spoke VPC (same tenant, added via tags for autoprovisioning to do it's magic, which it did) - BGP AS 64512

I stood up a Amazon Linux EC2 host in the spoke VPC as a test destination.  If I ping or SSH to it from an on-prem host, it fails and watching the traffic shows that it goes out the on-prem gateway, hits 'GW1' of the Transit VPC, hits the EC2 host, the return traffic winds up going through 'GW2' of the Transit VPC and gets dropped with "ICMP reply does not match a previous request" or 'First packet isn't SYN' for a TCP connection.

If the Transit VPC hosts were ClusterXL, I suspect it would work due to state-sync, but these guys are independent, thus the asymmetric issue.

Any quick tips on how I solve this?

Thanks

[Dave_Hollis]

Unfortunately, I wasn't able to figure out what was going wrong with this deployment so I nuked the Cloudformation and rebuilt it - always easier the second time around because you have all of the variables figured out - and it worked right out of the gate.

Between the CF templates and the autoprovisioning stuff automating a lot of the tedious stuff, that makes it a lot easier to do iterative build/tear-down steps to really get this stuff worked out and well documented.  That's a big plus!