I think you have the issue exactly right. It is one thing to allow individuals to agree to an alternative judicial system where it enforces the same principals and laws of our society. For example, parties to a dispute may agree to have an arbitrator resolve their dispute instead of a court, but the arbitrator will still apply our society's laws and principals to resolve that dispute.
It is something else, however, to allow people to agree to a system that would enforce rules opposed to the principals and laws of our society. For example, a religious court determining that a husband properly beat his wife.
Although consenting adults should be given broad discretion in what they can agree to do, agreements that contravene (or lead to the contravention of) our society's laws and principals should not be enforced. (and generally are not)
Further, there would be too many problems in determining whether consent to a religious court was truly "voluntary." We already have far to many examples of closed religous communities using their cloistered and secretive nature to abuse followers.
In short, there is simply too much mischief that would be permissible if our society allowed religious courts to issue binding and final decisions.