Well, if the issue appears only after the client has connected once, I'd strongly point into the firewall's direction.
The issue I had in the past was, that the WiFi Direct Connection could not be created due to a bug in CP, but that should be solved. If you have the Windows Firewall enabled I often noticed some "strange" behavior in the past. Most important thing to consider is, that from an NLA / Firewall point of view (network location awareness) the WiFi Direct connection to the Remote-Screen is considered a Private / Public connection. There is a good article from a competioner (not sure if I'm allowed to link that) that describes IP ranges and ports. Just google for: wireless display ports firewall should be in the top 5 results.
A good start troubleshooting would be, to have the feature (temporarily) enabled so a user can turn off the firewall policy of Endpoint Security, so you can check if it works without it. Also check the Windows Firewall Log (if enabled) if you can see any drops.