I think I get what you were referring to. What's worse, I installed a mod that takes away quest markers and the whole HUD when I'm not fighting, so I got lost so many times in that damn fort.
The worst part was when Isran asked me to join him upstairs. I must have spent like 30 minutes trying to find out how to get up there XD