R77.30 is still supported and will be at least until September 2019. So there is plenty of time preparing and planning the migration.
You are right, as you are running a Security Management, SmartEvent Server, Correlation Unit and Enforcement Gateway all on the same appliance (standalone HA) the performance hit by upgrading to R80.x will be something you'll need to put into consideration. Therefore I suggest to split the management. Either you buy two Smart-1 Appliances for Management-HA or buy a management container and run your new management within a redundant VM environment.
Wether choosing between R80.10 or R80.20 I'd like to forward you to the vendor's recommendation. Check Point writes in sk95746: "R80.20 is initially recommended for customers who are interested in implementing the new features. We will make it the default version after significant adoption. It will then be available in the 'Showing Recommended Packages' section in the CPUSE tab in Gaia portal."
So you could test R80.20 in the first place (remember, you still have many months of time) and if it's working great for you it's perfect. Otherwise use Option B and migrate to R80.10.
Regarding your BGP-4 question, refer to the GAiA Advanced Routing Admin Guide. It's included starting from R80.10.