You can install VSX into ESXI and do for LAB purposes.
The main thing that need to know is that if you use a Virtual Switch in VSX then the corresponding vmware vSwitch needs to be put into promiscuous mode in vmware.
If put in direct interface then nothing special in vmware to be done.
The reason that you won't find documentation is that pretty sure it isn't supported to run VSX in vmware. vmware products are the Cloudguard and there is no VSX in Cloudguard.
However if you treat as if a normal gateway so simply set as
RHEL 5 - 64bit
vmware 7 ( so can see the proper boot process in the vmware console )
then will actually work.
For LAB then is fine, however not for Production.
If you need to virtualize multiple firewalls in vmware for production the just use multiple Cloudguard deployment. Going to be simpler to manage, effetively instead of multiple VS then can simply run multiple gateways.
Not seeing any benefits for VSX in vmware, don't think going to be cheaper either to be honest.