Or, perhaps she mentioned it, and he wasn't listening? I've had that happen numerous times. From what it sounds like to me, the man is solely trying to make excuses for himself should anything "happen". He says that she's much younger, and beautiful, and even mentions that she seems uninterested, which would lead any savvy reader to get the point that he may have already tried to get her attention in an unwarrented manner. I think Prudie totally missed on this one. She needs to go back and re-read. this man is obviously trying to have an affair that he can blame his wife for. It's like he's saying she put a tasty treat in front of him, and like a dog he feels he should have every right to blame her for his misconduct.
You are wrong, the biggest issue is that this man is trying to validate himself for cheating, or his thoughts on cheating. If not with this neighbor, it will happen some time down the line.
As for the other post above, you are listening too much to Howard Stern. Try reading a few books on human behavior.