Actually you are correct. There is no legal basis for denying a polygamist marriage if all parties are consenting adults. From what I can tell, the primary legal reasoning against it is that a polygamist marriage introduces a nightmare of legalities. IE - If wife A and husband B disagree on which medical procedures to approve for husband A, which takes precedence? If husband C dies, without a will, who gets which share of the left overs?
Other than these problems, morally there is no problem. Most of the populace gets hung up on the horror stories about those polygamist cults that pop up now and then that introduce force and coercion to the mix. In a group of normal functioning humans, it is a severe gap of logic to think that a person can only love one other person. Love is not like water with a finite amount, it is more like laughter of children, an unending supply that can be spread among many.
(Personally, I can barely stand one person in my life. I am not a candidate for that type of marriage. I just don't feel like standing in the way if it works for other people.)