They can be annoying at times, but it's something you get used to rather quickly. They're still more of a help than a hindrance. You can lower the rate at which your own pawn speaks, but unfortunately, they will never be completely mute-- and hiring other people's pawns won't be affected by the speech restrictions you set to your main pawn.
If you set them to speak less and play it solo, you really notice how big of a change it is.

Their AI is exceptional, and honestly, I still think Navi rates at a much higher annoyance level when it comes to needless, sidekick banter.

It can be quite funny though!