I, too, am surprised that Prudie would go there, especially on the thin evidence presented here.
1. "Several times, he's called me by her name."
I don't know of anyone that doesn't call someone the wrong name occasionally -- especially if both are people that they spend a lot of time with. I occasionally call one of my sons by his brother's name and it certainly doesn't mean I conflate the two. Unless he's calling her by his sister's name while they're having sex, I don't see this as proof of anything.
2. "At the last family dinner, he was playing footsies with her under the table."
My mind didn't go to the "sensuously rubbing up and down each other's legs with their feet" place, either. Maybe since I have a 23-year old (married with a kid) and a 22-year old that end up picking at each other like little kids whenever they're together, I instantly pictured the siblings stepping on each other's toes and kicking each other and giggling. Yeah, it can be very annoying, but an indication of an untoward relationship? Not so much. Especially since it was a family dinner. Are we really supposed to believe that they were engaging in some sort of sexual flirtation with the parents, the husband, the fiance, and possibly the inlaws and/or grandparents present?
3. "My boyfriend knows very private things about her, the kind I would never discuss with my brother."
Define "private". Maybe the LW would never discuss much more than weather with her brother. We certainly don't know. Does he know his sister is trying to get pregnant? Or does he know her favorite sexual position? There's certainly a wealth of difference between the two.
"I do distance myself from her because I don't like being with the two of them due to their behavior...Am I being unreasonable?"
If she really believes that her fiance is acting inappropriately with his sister, why is she even still in this relationship? If it turned out to be true, would she really want to continue to be with him, even if he stopped? I think that she doesn't really believe there is anything going on, she just wants permission to tell him that he has to put distance between him and his sister because, "see, even the advice columnist thinks you're creepy! Now pay more attention to me, dammit!" Maybe she's jealous because she wishes her relationship with her own brother was closer. Maybe she just doesn't like him acting so goofy whenever his sister is around. Maybe she's just an insecure, jealous cow that can't stand to see him paying attention to anyone else, especially when she's sitting right there. Whatever the problem is, these two need to talk. However, if she opens with "so just what are you doing with your sister, anyway!", I don't think the conversation (or the engagement) will last too long.