Why all the fuss about Gay Marriage?

I have to admit that I just don't get why the opposition to gay marriage is so passionate. I can understand that people think homosexuality is sinful and that it would be better if everyone were neatly heterosexual. But I also accept as crucial to a principled Democracy a difference between private and public morality.

Here's a thought experiment of sorts:
Two men live next door to me.
I have no reason at all to think they are gay, and in fact I have some evidence that they are not.
But suppose they are. Given that I do not know this, is there any reason at all to think that society is being harmed by their private behaviors? Clearly not.
But perhaps it isn't their private behavior that is at issue, but public behaviors. Maybe their private "gayness" bleeds over into the public sphere - perhaps even in insidious ways.
There is no reason to believe this, either. Certainly no more than there is reason to believe that my other neighbor's private fondness for reading celebrity gossip magazines has insidious public consequences. There is certainly nothing that would support legal opposition to gossip magazines.