I've heard stories of groups of men harassing scantily clad women who passed near by--sometimes to the point of assault.

Men will be men sometimes means a**holes will be a**holes and predators will be predators. Deprive them of the excuse of a veil, they'll find another excuse.