I wouldn't go so far as to say there was an affair between the speaker and the "beautiful mild woman" in the poem, but there does seem to be at least an implied comparison between her and the woman who is the "you." If one is a "beautiful mild woman," the other (presumably Maud Gonne) is perhaps a beautiful fierce one. Perhaps there is a mild rebuke to the beloved that if only she had been a bit milder herself, things might have worked out better, although if she had been milder, perhaps she would not have been so loved. The speaker is able to say things ("There have been lovers ...") to the mild woman that are an indirect commentary on his love for the other woman, things he cannot or would not say to her directly because of the criticism of her they imply (that if she had been a bit "milder" like her friend, she might have appreciated his "high courtesy" more).

Yeats is perhaps my favorite poet, but ultimately I have to say I just don't think this is one of his best poems, at least not for me. I may have the chronology wrong, but I think it's correct that this was written after Yeats' earliest poems like "The Lake Isle of Innisfree," but well before his later poems like "No Second Troy" or "Leda and the Swan," poems that show a much less mild view of love. For me "Adam's Curse" is a transitional poem, one whose awkwardness shows why Yeats had to leave behind the "sweet sounds" of his early poems in order to write his great later poems. The bitterness of "No Second Troy" ("Was there another Troy for her to burn?") makes an interesting contrast with "Adam's Curse" in its attitude toward presumably the same love.