In a nutshell:
Multi-domain is management virtualisation. Each domain is independent of the other so you can manage clients from the same multi-domain management server without mixing their data, each one has their own domain.
VSX is gateway virtualisation: you can use a dedicated appliance or server to run virtual firewalls which have their own IP space and each can use a different set of blades. Unlike virtual machines, the OS and patches are common since they run at the appliance/server level. So a single cluster could run for example 10 VS which do each their own routing and have their own policy.
You can use VSX with a multi-domain manager or an SMS. With an MDS, you could have each VS or multiple VS in a dedicated domain and some others in other ones. Without MDS, with SMS, you have one base domain and all VS share all objects but still have their own IP/blades configuration.
Please note this is really a basic explanation, the CCVS course for instance goes into much details of what VSX is.