EU and UK will mitigate losses. The UK joined the EU without joining and the UK will leave the EU without leaving.
The UK is in a unique situation in Europe due to its geographic separation it cannot benefit from borderless travel anyway. The UK never gave up the Pound so no benefit from a common currency. I suppose the British isles have lost control of migration to their shores on occasion throughout history, but they were never happy about it. Trade will be maintained.