"Every part of the body is somewhat sexual...." Because any other part of the body that is not genitals, that is sexual, is made so by us. In point, breasts are sexual... by men. But for you, they are not.
"Breasts are sexual." --- Dudes.
"Breasts are for breastfeeding." --- Chicks.
Breasts are for breastfeeding, something that is necessary in a woman for the successful survival of offspring. But since they are not necessary in a man for the successful survival of offspring, they lack, making them opposite in this way than woman. Anything that physically determines the opposite gender, makes them attractive to the other gender.
Breasts are by far one of the biggest differentiating features in woman as adults, so they are of a course a very large target for attraction, so much that it becomes sexual.
In short, breasts being for breastfeeding, make them sexual for men.