If they aggressively defend them, it annoys me. But that's for any kind of fan-shipping, not just yaoi and yuri.
People have gotten upset with me for shipping one of my OCs with an official character. Other people have told me it's fine as long as I don't get overly defensive about it.
Personally, I have better things to do than worry about the relationship status of a cartoon character, know what I mean?