I don't believe all of the Bible to be literal either. There is plenty of content that is clearly meant to be poetic and/or symbolic. However, just because something can be interpreted in multiple ways it doesn't mean it should be. It's like that annoying English teacher that reads in the book, "The curtains were blue." The author was only describing the color of the curtains, but the English teacher babbles on and on about the possible implications and hidden meanings in the sentence while there aren't any there. Anyone can find any interpretation of anything they want, and people use that excuse to say we shouldn't listen to any of the Bible. Let's not forget how even secular music can have multiple interpretations of their own, yet I don't see anyone discarding them for that reason, do you?
I don't know about you, but if everyone in the world suddenly believed that the moon was made of cheese I wouldn't compromise my knowledge to go with their idea for the sake of unity. It's been proven that it's not cheese, so why would I be ignorant enough to throw away all the facts to accept a new belief based on faulty assumptions?
Discarding truth for the sake of unity is edging towards the Utilitarian point of view, which according to multiple sociologists over the decades, does not work in a functioning society.
Consider this scenario:
Me: I'm tired of Kansas; let's move to California! We'll head east.
You: Umm, east? That's the opposite direction of where we need to go.
Me: It doesn't matter what you say, I believe if I go east I will end up in California!
You: But just look at this map! If we go east we'll arrive in Ne---
Me: I DON'T CARE what the map says! My parents said California is east, my grandparents said California is east, so it doesn't matter what the map tells me. I believe that California is east, so that's where I will go.
If I go by that logic will I end up in California, going east from Kansas? No, it doesn't matter how hard I believe it, if it's the wrong direction the country's geography won't change to suit my beliefs. Same goes with Christian doctrines. It doesn't matter what a Baptist, Catholic, or Lutheran believes. If it's true, it's true, and if it's wrong, it's wrong. Just because you don't know what truth is, is that motivation for you to not pursue it?
It doesn't matter how much people disagree about doctrine. Would you rather be happy with a lie or sad with the truth? I personally wouldn't be able to stand the fact that someone told me a lie that I believed for years. Going along with a lie for the sake of getting along is herd mentality, and even the apostle Paul praises individuals that seek the truth, "...they received the word with all readiness, and searched the Scriptures daily to find out whether these things were so" (Acts 17:11).