Are you using a smudge brush? It sounds like you have "overlay mode" activated for the Smudge Brush.
To turn it off, go to the brush settings drop-down, and uncheck Overlay mode in the column where you have size, spacing etc.
If that doesn't work, or if you're using another brush, please tell me what brush you're using exactly etc.