True, but you've gotta consider:
Not ALL High Elves are assholes. The Thalmor are the ones ruining Skyrim and the Empire. Elves like Calcelmo and his nephew probably don't care a fig for the Thalmor, and wouldn't care if they left (it would also be a damn shame to see them go, as Calcelmo is bloody awesome). The same applies for Niranye in Windhelm; she gives no fudgeos whatsoever, she's just there to turn a profit.
You also have to consider other unnamed High Elven citizens, and all that jazz.
(But if by "elves" you mean the Thalmor exclusively then I'm blowing this all out of proportion MY BAD >.<)