The caveat to the "raw evidence that you are sexually desireable" idea is that when two people come together who are not in a primary relationship (either one or both parties are "straying" from their primary relationship) there is so much stuff going on that it is FAR more complicated than (simply) evidence of either party's sexual desireability.
It's an ironic and occasionally troubling aspect of human nature that we can be partnered with the most intelligent, wonderful, polite, sexy, educated, good cooking-good sexing-well dressed person in the world and yet STILL desire validation from outside.
Perhaps if our culture were more open to validation, which does not necessarily have to be sexual but can come in many guises, we could begin to understand that the human need (which expresses itself very differently in people, with some having none to little desire for another partner and others struggling with it daily) does not have to mean the end of a relationship or marriage. It doesn't even need to mean SEX. But whatever it is that is driving people to find this validation needs to be explored.
Frankly, I think it has a lot to do with how needy Americans are in general, that instant gratification has become nearly a cultural right in the USA, and how little people understand about how hard it is to find peace and satisfaction. Most people look to the outside but true peace can only come from within (sorry for the cliche, but so it goes...)