Basically you are right about the individual chooses what is private. So Elizabeth Edwards may remain private and not hounded on the subject.

But, in public life when John Edwards denies having an affair or when he says that if a man has affairs you can use that to measure his fitness to hold public office, then it is fair to look at that aspect in him.

Note how it reveals poor judgment in Edwards in regards to Rielle who evidently notified the Enquirer and must have taken the pictures of Edwards in the hotel room (that or the agents of the Enquirer have commited a serious crime). If Edwards couldn't deny himself Rielle what else can't he deny himself when choosing between himself and what is good for the public? Money, power, fame.